Streaming failures agents can detect

Streaming is how most interactive model applications deliver output, and increasingly how agents consume it too, because it lets them start acting on a response before it finishes. Streaming also changes the rules for reporting errors in a way that is easy to overlook until an agent acts on half an answer.

The status code is already gone

A streamed response is sent with server-sent events (SSE) or a similar chunked format. The sequence is:

1. The server sends the status line and headers, normally 200 OK.
2. The server flushes those to the client so it can begin reading.
3. Content chunks follow as the model produces them.
4. The stream ends.

After step 2, the status code has been delivered. HTTP provides no way to change it. If the upstream model fails at step 3, after the first hundred tokens, the server cannot turn the response into a 502 or 503. Whatever it does next has to be expressed inside the body.

This is not specific to any provider or gateway. It is how HTTP works. What varies is what servers do about it.

A stream that just stops

The simplest thing a server can do when an upstream fails mid-stream is close the connection, or end the stream as if it had finished. From the client's side, this is very hard to distinguish from success.

Consider an agent asking a model to produce a JSON plan with five steps. The upstream fails after step two. If the stream simply ends:

• the HTTP status was 200,
• the client received well-formed chunks,
• the content ends at a plausible boundary, perhaps the end of a sentence,
• no error was raised by the SDK.

The agent now holds a two-step plan and has no reason to think it is incomplete. If it executes the plan, the failure has become silent and permanent. A human reading a chat window might notice the answer stopped abruptly. An agent will not.

Some signals can help. Protocols usually send a final event with a stop reason, such as a message_delta carrying a stop_reason followed by message_stop in the Anthropic Messages API, or a chunk with a finish_reason and a [DONE] sentinel on OpenAI-compatible streams. A careful client can treat a stream that ends without those as suspect. But that relies on every client implementing the check, and on the server never emitting a terminal event on the failure path. Both assumptions break in practice.

Emit an explicit error event

The reliable fix is on the server: when an upstream fails after the stream has started, send an error event in the stream before closing it.

On the Anthropic Messages API, the stream format already defines one:

event: error
data: {"type": "error", "error": {"type": "api_error", "message": "Upstream stream failed"}}

On OpenAI-compatible streams, the common convention is a final data chunk carrying an error object instead of choices.

An explicit event gives the client something unambiguous. Official SDKs raise an exception when they see an error event, which means the agent's normal error handling runs, instead of the agent receiving a truncated result as a return value.

Gatewayz ends a stream that fails upstream after starting with an explicit error event. This was fixed on 2026-09-09; before that, a mid-stream upstream failure could end the stream without one, which is exactly the silent truncation described above.

Retryable or not

A mid-stream failure caused by the upstream is transient in nature: the same request might succeed on a new attempt. The error event should say so, in the same way a 5xx would have if the failure had happened before the headers were sent. See Who reads the error? for why that classification matters.

Retrying a partially streamed request has a cost the client must weigh. The tokens already produced are usually billed, and if the agent already acted on partial output, such as calling a tool, a retry may duplicate that action. Agents that act on streamed content should make those actions idempotent or defer them until the stream completes successfully.

Do not turn client disconnects into server errors

The reverse case needs equal care. Clients disconnect for ordinary reasons: a user closes a tab, an agent cancels a request it no longer needs, a timeout fires on the client. When that happens mid-stream, the server sees a broken connection.

A server should not record that as a server error. If it does:

• error rate dashboards show failures that were never failures,
• alerting fires on normal cancellation behavior,
• a circuit breaker that counts these may conclude the upstream is unhealthy and start rejecting traffic that would succeed.

The server should stop reading from the upstream, record the request as cancelled by the client, bill the tokens that were actually produced, and not emit anything further. There is nobody left to send an error event to.

The distinction is who ended the stream. An upstream failure is the server's problem and must be reported to the client. A client disconnect is the client's decision and must not be reported as a server fault.

Client checklist

For agents consuming streams from any model API:

• Let the SDK raise on error events. Do not catch and discard stream exceptions in the iteration loop.
• Require a terminal event. Treat a stream that ends without a stop reason as a failure, as a second line of defense.
• Validate structured output. If you asked for JSON with five steps, check that it parses and has five steps before acting on it.
• Defer side effects until the stream completes, or make them idempotent.
• Cancel deliberately. When an agent abandons a request, close the stream explicitly, so the server can tell cancellation from failure.

Server checklist

For anyone serving streams:

• Validate everything you can before sending headers, so predictable errors still get a real status code.
• On upstream failure after headers, send a protocol-appropriate error event, then close.
• Never emit a normal stop event on a failure path.
• Classify client disconnects separately from server and upstream errors in logs and metrics.
• Test the failure path by killing an upstream mid-stream, and assert the client raises.