Prompt caching through a gateway

Prompt caching lets a model provider reuse the processed form of a long, repeated prefix, such as a system prompt, tool definitions or a large document, across requests. Cached input is billed at a lower rate than fresh input and is processed faster. For agents that resend the same large context on every turn, it is often the single largest cost lever available.

When requests go through a gateway, caching adds a question: does it still happen? This article covers what has to be true for caching to work through a middle layer, and how to write a test that proves it rather than one that passes for the wrong reason.

How Anthropic prompt caching works

On the Anthropic Messages API, caching is explicit. You mark the end of a cacheable prefix with a cache_control block:

{
  "model": "anthropic/claude-sonnet-4-6",
  "max_tokens": 256,
  "system": [
    {
      "type": "text",
      "text": "<long, stable instructions>",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "messages": [{ "role": "user", "content": "First question" }]
}

The response's usage object then reports how the input was handled:

• cache_creation_input_tokens: tokens written to the cache on this request.
• cache_read_input_tokens: tokens served from the cache on this request.
• input_tokens: tokens processed normally.

On the first request with a given prefix you expect a cache write. On a later request with the identical prefix, within the cache lifetime, you expect a cache read.

The authoritative description, including cache lifetimes, pricing multipliers and which content can be cached, is in Anthropic's prompt caching documentation.

What a gateway must do

For caching to work through a middle layer, three things have to hold:

1. cache_control is passed through unchanged. A layer that rebuilds the request from a normalized internal format can easily drop fields it does not model.
2. The prefix bytes are stable. If the layer injects anything into the system prompt, reorders tools, or rewrites content between requests, the prefix changes and the cache misses every time.
3. The usage fields are returned. If the layer maps the provider's usage into a smaller schema, cache tokens may disappear from the response even when caching happened, and billing can no longer reflect cache rates.

Gatewayz passes cache_control through on the native Messages endpoint at https://api.gatewayz.ai/v1/messages and returns the cache usage fields. This has been verified end to end: a cache write followed by a cache read through the gateway.

The trap: a minimum cacheable length

Anthropic only caches prompts above a minimum length. At the time of writing, Anthropic's documentation lists a minimum of 1,024 tokens for Sonnet and Opus models and 2,048 tokens for Haiku models, with per-model details in its table. These minimums are set by Anthropic and can change with new models, so read the current values in the documentation rather than hardcoding them into a test.

The important behavior is what happens below the minimum. The request succeeds, there is no error and no warning, and the cache usage fields are absent from the response. Not zero: absent.

That produces a test that is easy to write and wrong:

# A test that proves nothing
resp = client.messages.create(
    model="anthropic/claude-sonnet-4-6",
    max_tokens=16,
    system=[{"type": "text", "text": "You are terse.",
             "cache_control": {"type": "ephemeral"}}],
    messages=[{"role": "user", "content": "hi"}],
)
assert resp.usage.cache_read_input_tokens is None or resp.usage.cache_read_input_tokens >= 0

The system prompt is a handful of tokens, far below the minimum. Caching never engages. The assertion accepts both "missing" and "zero", so it passes, and it would pass identically through a layer that strips cache_control entirely. A green result here tells you nothing about whether caching works.

A test that proves caching

A meaningful test has to be able to fail. That requires four things.

A prefix above the minimum for the model under test. Build a system prompt comfortably larger than the documented minimum. Use the provider's token counting endpoint if you want to be exact.

A salt. Caches persist across requests for a period. If your test reuses the same prefix as an earlier run, the "first" request may already be a cache read, and you never observe a write. Put a unique value near the start of the cached block on every run.

An assertion on the write. The first request must report cache_creation_input_tokens > 0. If the field is missing, fail the test with a message saying caching did not engage.

An assertion on the read. A second request with the identical prefix must report cache_read_input_tokens > 0.

import uuid
from anthropic import Anthropic

client = Anthropic(
    base_url="https://api.gatewayz.ai",   # no /v1: the SDK appends it
    auth_token=GATEWAYZ_KEY,
)

MODEL = "anthropic/claude-sonnet-4-6"
salt = uuid.uuid4().hex
# Well above the documented minimum for this model; check the current table.
long_prefix = f"run {salt}\n" + ("Reference material paragraph. " * 900)

def ask(question):
    return client.messages.create(
        model=MODEL,
        max_tokens=16,
        system=[{"type": "text", "text": long_prefix,
                 "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": question}],
    )

first = ask("One word: first.")
write = getattr(first.usage, "cache_creation_input_tokens", None)
assert write and write > 0, f"no cache write (field={write!r}); prefix below minimum or cache_control dropped"

second = ask("One word: second.")
read = getattr(second.usage, "cache_read_input_tokens", None)
assert read and read > 0, f"no cache read (field={read!r}); prefix changed between requests or cache not honored"

Every failure mode of the plumbing makes one of those assertions fail: a stripped cache_control, a mutated prefix, a dropped usage field, or a prefix that is too short.

Caching on the OpenAI-compatible path

On OpenAI's own API, caching is automatic above a threshold and is reported in a different usage field. Behavior on an OpenAI-compatible endpoint depends on the provider serving the model. If you rely on caching, test on the exact endpoint and model you will use in production, with the same write-then-read structure, rather than assuming parity between surfaces.

Using cache results in cost claims

If you are comparing costs between routes, or reporting savings from caching, the run has to contain cache reads. A benchmark in which every request reports zero or missing cache_read_input_tokens has not measured caching at all, and any cost comparison drawn from it describes uncached traffic only. How to benchmark a gateway honestly covers this and the related rules for latency.

Checklist

• Mark stable prefixes with cache_control on the Messages API.
• Keep the prefix byte-for-byte identical between requests.
• Size test prefixes above the current documented minimum for the model.
• Salt test prefixes per run.
• Assert cache_creation_input_tokens > 0, then cache_read_input_tokens > 0.
• Treat a missing field as a failure, not as zero.