What is an AI gateway? Why model routing matters

An AI gateway is a service that sits between an application and several model providers. The application sends every request to one endpoint with one API key, and the gateway forwards it to a provider that serves the requested model and returns the response in a consistent format. Model routing is the rule that decides where each request goes.

What does an AI gateway actually do?

A gateway accepts a request that names a model, sends it to a provider that serves that model, and returns the result. Around that core it handles the work every multi-provider application would otherwise build itself.

The jobs usually include:

1. Authentication. You hold one API key for the gateway instead of one per model provider.
2. Format translation. Providers expose different request and response shapes. A gateway can accept one shape, such as the OpenAI-compatible chat completions format, and translate for each provider behind it.
3. Model resolution. It turns the string you sent into one exact catalog entry, or refuses.
4. Routing and failover. It picks a route to a provider that serves that model, and can move to another route for the same model when one is failing.
5. Metering and billing. It counts tokens per request and charges one balance.
6. Consistent errors. It reports failures with the same status codes and error codes regardless of which provider was involved.

Gatewayz is a gateway of this kind. It has no model of its own. It exposes an OpenAI-compatible endpoint at https://api.gatewayz.ai/v1/chat/completions, a native Anthropic Messages API endpoint at https://api.gatewayz.ai/v1/messages, and a public catalog at https://api.gatewayz.ai/v1/models. The providers in the catalog today are OpenAI, Anthropic, xAI, Moonshot and Meta. For the broader concept, see What is an inference layer?.

Why use one integration instead of many?

Because every direct integration is its own ongoing project: its own SDK, credentials, error types, rate limits, billing account and changelog. The cost is not writing the first request. It is maintaining several of them.

Providers do converge in places. Anthropic, for example, documents compatibility with the OpenAI SDK for testing. But native features, error formats and billing still differ per provider, and each one needs a signed-up account and a funded balance.

The last three rows are real trade-offs. A gateway is another service your requests pass through, it may lag a provider in exposing a brand-new feature, and it charges for the work it does. For a team using a single model from a single provider, a direct integration can be the simpler choice. The case for a gateway grows with each additional provider you depend on.

How do you switch models through a gateway?

You change the model string in the request. The endpoint, key, SDK and response handling stay the same.

Before, calling an OpenAI model:

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.gatewayz.ai/v1",
    api_key=os.environ["GATEWAYZ_API_KEY"],
)

response = client.chat.completions.create(
    model="openai/gpt-5",
    messages=[{"role": "user", "content": "Summarize this incident report."}],
)
print(response.choices[0].message.content)

After, calling an Anthropic model instead:

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Summarize this incident report."}],
)

Model ids on Gatewayz are namespaced by provider, such as openai/gpt-5, anthropic/claude-haiku-4-5-20251001 or moonshot/kimi-k2.6, so the string says both what you want and who serves it. Without a gateway, the same switch would mean installing a second SDK, creating a second account, storing a second key, and adapting to a second response and error format.

A one-string change is easy to make, which is exactly why it deserves the same care as any other production change. Different models behave differently on the same prompt. Run your evaluations on the new model before you switch the string in production.

What is failover, and what does it not do?

Failover means sending a request for a model to another route serving that same model when the first route is degraded. It keeps the model fixed and changes only the path to it.

A single model can sometimes be reached through more than one provider route. If one route starts returning server errors or timing out, a gateway can stop sending traffic to it for a while, a pattern commonly implemented as a circuit breaker (Martin Fowler's description of the pattern is a good introduction), and use another route for the same model.

On Gatewayz, the capability to fail over to another provider route serving the same model is built. It is not yet something we describe as proven at scale, and a route for a given model may not always have an alternative. When no route can serve the request, you get a retryable 5xx error, which your code should handle with bounded backoff.

What failover does not do is equally important. It does not switch you to a different model because your model is unavailable. That would be substitution, not failover.

Why does Gatewayz never substitute a different model?

Because the model you name is part of your contract with your own product. Cost, quality and behavior all follow from it, and a layer that silently changes it breaks all three.

• Cost reports. Models differ in price per token. If a request for one model runs on another, your spend no longer matches what you budgeted, and the discrepancy is hard to trace.
• Evaluations. Teams test prompts against a specific model and approve it for a use case. A substituted model has passed none of those tests.
• Behavior. Tool-call formatting, refusals, output length and tone vary between models. An unattended agent will not notice the change; it will act on the different output.

So Gatewayz resolves names by fixed rules and refuses when it cannot. An alias, an undated name like claude-sonnet-4-5, resolves to one exact dated snapshot, in this case anthropic/claude-sonnet-4-5-20250929. Providers publish models this way too; Anthropic's page on model ids and versions explains how its aliases point to snapshots. An id that matches nothing returns an error:

{
  "error": {
    "message": "Model 'anthropic/claude-sonet-4-6' does not exist. See GET /v1/models for available model ids.",
    "type": "invalid_request_error",
    "code": "model_not_found"
  }
}

That response comes with status 400, a terminal client error in the sense of RFC 9110, so SDKs do not retry it and a typo shows up as a typo instead of an apparent outage. The full argument is in Resolution, not substitution.

If you want a fallback to a different model, write it in your own code, where it is explicit, logged and reviewable.

What are common misconceptions about model routing?

Most confusion comes from treating routing as a model-selection service. On Gatewayz, it is not.

A router chooses the right model for each request

Some products try to do this by classifying prompts and sending them to different models. Gatewayz does not. It runs the model you name. Choosing a model involves your quality bar, your budget and your evaluations, and those are decisions a gateway cannot make on your behalf without breaking the guarantees above.

Failover means I always get an answer

Failover changes the route, not the model, and only when another route for that model exists. Requests can still fail, and your code still needs to handle 429 and 5xx responses with bounded retries.

An alias is as stable as a snapshot

An alias is a convenience that a provider may move to a newer snapshot over time. For production, pin the dated snapshot when the catalog offers one, and check the model field in each response.

A gateway sees and keeps everything I send

A gateway does have to pass your prompt to the provider that runs the model. Gatewayz does not store prompt or completion content on ordinary API calls; it keeps billing metadata such as token counts, cost, model, status and timing. The provider serving the model still receives the prompt.

A gateway makes models cost less

On Gatewayz, you pay the provider's list price plus a routing fee. What the gateway reduces is integration and operational work, not the price of a token.

Frequently asked questions

Is an AI gateway the same as a model router?

The terms overlap. A router is the part that decides where a request goes, and a gateway is the whole service around it, including authentication, format translation, metering and errors. Some products use router to mean a service that picks a model for you, which Gatewayz does not do.

Does Gatewayz pick a model for me?

No. You name the model in every request, and that is the model that runs. An alias resolves to one exact dated snapshot, and an unknown id returns 400 model_not_found. Gatewayz never replaces your model with a different one, including when a route is degraded.

What happens if the model I request is unavailable?

If another provider route serving the same model is available, the request can fail over to it. If not, you receive a retryable 5xx error. You never receive a response from a different model, so your code should retry with bounded backoff or apply a fallback you wrote yourself.

Can I still use the Anthropic SDK or Claude Code?

Yes. Gatewayz exposes the native Anthropic Messages format at https://api.gatewayz.ai/v1/messages, so the Anthropic SDK can point its base URL at https://api.gatewayz.ai. For Claude Code, set ANTHROPIC_BASE_URL=https://api.gatewayz.ai and ANTHROPIC_AUTH_TOKEN to your key, as described in Run Claude Code through Gatewayz.

Where can I see which models are available?

The catalog is public at https://api.gatewayz.ai/v1/models and needs no key. It lists the exact ids you can send, including dated snapshots. Checking your configured model ids against it at deploy time turns a runtime 400 into a failed deploy you can fix before anything runs.

Related

• Resolution, not substitution. Why an alias resolves to one snapshot or is refused.
• What is an inference layer?. The layer a gateway belongs to.
• What is an AI agent?. Why unattended callers need the model they asked for.
• AI models and providers. Snapshots, aliases and catalogs explained.
• Who reads the error?. Why a typo must return a 4xx.