What is an inference layer?

Most teams that call more than one model provider end up putting something in the middle. It starts as a convenience: one base URL, one key, one bill. Over time that middle piece takes on more responsibility than anyone planned, and the vocabulary for it stays vague. People say "proxy", "gateway" and "router" interchangeably.

This article separates three things that often get the same name, and explains why software agents that run without a human in the loop push you from the first two toward the third.

Three things that sit in the middle

A proxy

A proxy forwards bytes. It may terminate TLS, add a header, or log a request, but it does not interpret the request. If the upstream returns a 503, the proxy returns a 503. If the upstream hangs, the proxy hangs.

A proxy is the right tool when you already trust the upstream contract and only need a network boundary. It has no opinion about models, prices, or errors.

A gateway

A gateway understands the shape of the traffic. For language models, that usually means it can:

• accept one request format and translate it to several providers,
• hold provider credentials so your application holds one key,
• meter usage and attach a price to it,
• apply per-key limits.

This is where most products in the category stop. A gateway is a translation and accounting layer. It is useful, and it removes a real amount of glue code.

An inference layer

An inference layer is a gateway that takes responsibility for the contract your software depends on, not only the transport. The questions it answers are different:

• When a request names a model, which exact model runs, and is that decision visible and reproducible?
• When something fails, does the status code tell the caller whether retrying can help?
• When a key has spent what it was allowed to spend, does the caller get a signal it can act on without a human reading a log?
• When a stream fails halfway, can the client tell a truncated answer from a complete one?
• Which version of the service answered this request?

None of these are about moving bytes. They are about making the behavior of the middle layer something a program can reason about.

Why unattended agents change the requirements

A person using a chat interface is an excellent error handler. They notice when an answer looks cut off, they re-read an error message, they decide whether to try again in a minute or give up. Most API design in this space quietly relies on that person.

An agent running overnight has none of that judgment available. It has whatever the SDK does by default, plus whatever logic its author wrote for the cases they anticipated. That shifts the burden onto the layer serving the requests in several concrete ways.

Status codes become control flow

For a human, the difference between a 400 and a 503 is a detail. For an SDK, it is a branch. Most official client libraries retry 429 and 5xx responses automatically with backoff, and treat 4xx responses (other than 429) as final. If a misspelled model id comes back as a 503, the SDK will retry a request that can never succeed, the agent will wait, and a circuit breaker further up may conclude the provider is down.

An inference layer has to map each failure to the class that matches reality: terminal errors as 4xx, transient ones as 429 or 5xx. See Who reads the error? for the full reasoning.

Model names must resolve, not drift

An unattended job that ran correctly last week should run the same model this week unless someone changed it. That rules out silently substituting a "close enough" model when the requested one is unavailable, and it rules out resolving an ambiguous alias to "whatever is newest". The layer should resolve an alias to one exact dated snapshot, or refuse. Resolution, not substitution covers this in detail.

Spending needs a hard, machine-readable stop

A loop with a bug can make a very large number of calls. Rate limits slow that down, but they reset. What an operator actually wants is a ceiling per key, and a distinct signal when the ceiling is reached so the agent stops rather than backing off and trying again. Spend ceilings for unattended agents explains why that signal should be a 402, not a 429.

Partial output must be detectable

Streaming responses begin with a 200 before the model has finished. If the upstream fails after that point, the HTTP status cannot change. A stream that simply ends looks identical to a short, successful answer. The layer must terminate a failed stream with an explicit error event. Streaming failures agents can detect walks through the cases.

The running version should be observable

When behavior changes, you want to know whether the service changed or your code did. Exposing the commit that is currently serving traffic turns "did something get deployed?" into a single request.

What this looks like in Gatewayz

Gatewayz is built as an inference layer in the sense above. Concretely:

• Two request surfaces: an OpenAI-compatible endpoint at https://api.gatewayz.ai/v1/chat/completions, and the native Anthropic Messages API at https://api.gatewayz.ai/v1/messages. Tool use and prompt caching have been verified end to end through Gatewayz.
• The live catalog is published at https://api.gatewayz.ai/v1/models and currently covers models from OpenAI, Anthropic, xAI, Moonshot and Meta. The catalog is the source of truth; we do not maintain a hand-written count.
• An unknown model returns 400 model_not_found. An undated alias resolves to its dated snapshot; if two snapshots could match, the request is refused rather than guessed.
• A key that has used up its request cap returns 402 request_cap_exhausted. An account with no credits returns 402 insufficient_credits. Rate limiting returns 429, and upstream failures return 5xx; those are the retryable ones.
• A stream that fails upstream after it has started ends with an explicit error event.
• GET /health/quick reports the commit currently serving requests.
• Pricing is per token at the provider's list price plus a routing fee.
• Plain API calls do not store prompt or completion content. Billing metadata is kept.

A test you can apply to any provider

If you are evaluating something to put in the middle of your agent traffic, a short checklist separates a gateway from an inference layer:

1. Send a request with a misspelled model id. Is the status 4xx, and does the body name the problem?
2. Send an undated alias. Does the response tell you which exact model ran?
3. Exhaust a spending limit on a test key. Is the status distinct from a rate limit?
4. Kill an upstream mid-stream, or find a way to simulate it. Does the client receive an error event, or just an end of stream?
5. Ask the service what version is running.

Each "no" is a case your agent code has to handle on its own, usually after the first incident that reveals it.