Measured against going direct

1. The same set of tasks is sent through Gatewayz and, where we hold that provider's key, straight to the provider.
2. Each task runs several times per route; we take the median for each task, then across tasks.
3. We record end-to-end latency and, where the stream reports it, time to first token.

These rules are enforced by the benchmark harness itself, not applied by hand afterwards.

• No baseline, no number. A latency figure is published only next to a direct-to-provider measurement from the same run.
• Within 15% or not at all. Gateway latency is published only when its median is within 15% of the fastest direct route. If we're slower than that, the page says so rather than showing a flattering subset.
• No cost claim without cache reads. A cost comparison is published only when the run actually read from the prompt cache and has a direct baseline. If a provider doesn't report cache usage, it counts as unmeasured, not as zero.
• Nothing fabricated. If a stream doesn't report time to first token, that column stays empty.

The reasoning behind these rules: how to benchmark a gateway honestly.