1 min
Wren Haldane
Where the 340 milliseconds go
A single agent step, taken apart: network, queue, model, tools. Most of the time is not where people expect.
When someone says a run took 340 milliseconds, they usually picture the model thinking for 340 milliseconds. It is rarely that simple.
Four stops
A single agent step passes through the network, the queue, the model and, most of the time, a tool. In our traffic the model takes a little more than half. The rest is waiting: for a connection, for a free worker, for a tool to answer.
That is good news. Waiting can be removed. Thinking can only be made faster.
What we did about it
Warm pools removed most of the cold starts. Streaming tool results removed most of the idle time after a tool call. The queue is next.