How to benchmark a gateway honestly

Any layer between an application and a model provider adds some work: a network hop, authentication, a catalog lookup, metering. The honest question is how much, under what conditions, and whether it matters for your workload. The usual way that question gets answered in this category, a single chart with one bar per product, rarely answers it.

This article describes a method for measuring a gateway that holds up to scrutiny, and the rules we apply before any number is published. It also states plainly where Gatewayz is: we have not yet published a production benchmark run. When we do, it will follow these rules.

What you are actually measuring

A request through a gateway has two parts: the time the model takes, which the gateway does not control, and the time the gateway adds. Model time is large and highly variable. It depends on the provider's load, the prompt length, the output length and the model itself. Gateway overhead is usually smaller and more stable.

Because model variance is so large, comparing a gateway run from Monday with a direct run from Tuesday measures the difference between Monday and Tuesday. The only way to isolate the gateway's contribution is to compare it with the direct route under the same conditions.

Method

Run back to back

For each sample, send the same request to the direct provider endpoint and through the gateway, one immediately after the other. Alternate which goes first, so that any warm-up or ordering effect is spread evenly. Keep:

• the same model id, resolved to the same dated snapshot on both routes,
• the same prompt, byte for byte,
• the same max_tokens and sampling parameters, with temperature at zero where the model allows it,
• the same client machine and network, in the same region.

If output length differs between paired responses, total latency is not comparable for that pair. Either constrain output tightly or compare time to first token only.

Measure the right moments

Record timestamps at:

• request sent,
• response headers received,
• first content token received (time to first token, TTFT, for streaming),
• last byte received.

Measure TTFT from the first chunk that contains model content, not from the first byte of the stream. Some servers flush headers or an empty event early, which makes TTFT look better than what a user experiences.

Report percentiles, not averages

Latency distributions for model APIs have long tails. A few slow requests pull an average far away from what most requests experience, and a few fast ones can hide a regression. Report at least:

• p50, the median: what a typical request experiences,
• p95: what the slowest one in twenty experiences.

Report the sample count next to every percentile. A p95 from forty samples rests on the two slowest requests and should be labelled as such.

Sample enough, and across time

Run enough paired samples that p95 is stable when you add more, and spread them across at least several different times of day. Provider load varies, and a run taken entirely during a quiet hour describes that hour.

Rules for publishing a number

Measurement is the easy part. The discipline is in deciding what the data allows you to say. These are the rules Gatewayz applies to its own claims.

No direct baseline, no latency claim. A latency figure for the gateway alone, with no direct-route figure measured in the same run, says nothing about the gateway. It is mostly a measurement of the model.

Publish latency only when the gateway median is within 15% of the best direct route. If the gateway's p50 is more than 15% above the best direct route measured in the same run, the honest summary is that the gateway adds meaningful latency for that workload, and a chart would obscure that. Publish the finding, not a flattering figure.

Never fabricate TTFT. If a route was not streamed, or the harness could not observe the first content token, TTFT for that route is unknown. Leave the cell empty. Do not estimate it from total time, and do not copy it from another run.

No cost-advantage claim from a run with zero cache reads. If caching is part of the cost story, the run has to demonstrate it: cache_read_input_tokens > 0 on the requests that should hit the cache. A run where every request reports zero or absent cache reads measured uncached traffic, and a cost comparison from it cannot claim savings from caching. Prompt caching through a gateway explains why small test prompts routinely produce exactly this result.

State what the gateway does that the direct route does not. Metering, cap checks and model resolution have a cost. A comparison that omits them from the gateway, or adds none of the equivalent work to the direct client, is not like for like.

Cost comparisons

Cost is easier to reason about than latency because it is arithmetic, but it has its own traps.

• Compare billed amounts, not list prices. Pull the actual charge for each request from each route, including cache write and read rates.
• Include fees. Gatewayz bills per token at the provider's list price plus a routing fee. A comparison with the direct provider should show that fee, not hide it.
• Match token counts. If input or output token counts differ between paired requests, compare per-token cost, or discard the pair.

Publish the harness

A result that cannot be reproduced is an assertion. A publishable benchmark includes:

• the harness code,
• the exact model ids and request parameters,
• the client region and the dates and times of the run,
• the raw per-request timings, so others can compute their own percentiles,
• the sample counts,
• every route measured, including the ones that did not look good.

Where Gatewayz stands

We have not published a production benchmark run, and we do not publish latency figures today. The site makes no speed claims for that reason. When we publish a run, it will include a direct baseline, p50 and p95 with sample counts, the raw data, and the harness, and it will follow the publication rules above even when the result is unflattering.

In the meantime, the most reliable measurement for your use case is your own: the same method, your prompts, your region, back to back against the direct provider.