Integrably Sorry

Anthropic API SSE and Eager Tool Dispatch

Today I was looking at how Eager Tool Dispatch might be implemented. That's when you execute tool calls while they are streaming. It might sound finicky at first, perhaps you need to parse partial jsons. Luckily that's not the case.

Anthropic API responses are streamed using SSE (server-sent events). Let's start with an example. Imagine you send "what's the weather in SF and NYC?" to an agent with a get_weather function.

The streaming response might look something like this.

+   0ms  message_start        usage.input_tokens=620, output_tokens=5
+   0ms  ping
+   0ms  [idx 0] content_block_start   <text>
+   0ms  [idx 0] content_block_delta   text="I'll check the current"
+ 433ms  [idx 0] content_block_delta   text=" weather in both cities" ...
+ 435ms  [idx 0] content_block_stop
+ 435ms  [idx 1] content_block_start   <tool_use id=toolu_vrtx_01ANbu… name=get_weather>
+ 435ms  [idx 1] content_block_delta   partial_json=''
+ 777ms  [idx 1] content_block_delta   partial_json='{"city":'
+ 778ms  [idx 1] content_block_delta   partial_json=' "San Francis'
+ 778ms  [idx 1] content_block_delta   partial_json='co"'
+ 786ms  [idx 1] content_block_delta   partial_json=', "unit": "celsius"}'
+ 789ms  [idx 1] content_block_stop        ◄── ETD fires: dispatch get_weather(SF) now
+ 789ms  [idx 2] content_block_start   <tool_use id=toolu_vrtx_02XyZq… name=get_weather>
+ 790ms  [idx 2] content_block_delta   partial_json=''
+1002ms  [idx 2] content_block_delta   partial_json='{"city":'
+1003ms  [idx 2] content_block_delta   partial_json=' "New Yor'
+1004ms  [idx 2] content_block_delta   partial_json='k"'
+1010ms  [idx 2] content_block_delta   partial_json=', "unit": "celsius"}'
+1011ms  [idx 2] content_block_stop        ◄── ETD fires: dispatch get_weather(NYC) now
+1013ms  message_delta        stop_reason=tool_use, output_tokens=142
+1013ms  message_stop

Each event has a type (message_start/delta/stop, content_block_start/delta/stop) and content. The response is a sequence of content blocks, each is either text or a tool call, the content is a partial "delta" which you can concatenate, and it explicitly tells you when a block ends, at which point you know you have the full tool call. This let's you dispatch the get_weather call to SF while you wait for the tool call to NY, saving some latency.

social