Glossary

Model

A trained set of numerical parameters plus the code that runs them. A language model takes text (and sometimes images or audio) as input and produces text one token at a time. Read more

Model provider

The company that trains or hosts a model and sells access to it through an API, such as OpenAI or Anthropic. The provider publishes the model's list price and decides when versions are released and retired. Read more

Training

The one-time, compute-heavy process of adjusting a model's parameters on large amounts of data. It happens before a model is released. Using the finished model is inference. Read more

Inference

Running a trained model on new input to produce an output. Every API call to a language model is an inference request, and each one consumes compute, which is why inference is billed per use. Read more

Compute

The processing capacity, measured in hardware and time, needed to run software. For AI it usually means accelerator time on GPUs or similar chips, along with the memory, power and networking around them. Read more

CPU

Central processing unit. A general-purpose processor with a small number of powerful cores, good at varied sequential work. It can run small models but is slow for large ones. Read more

GPU

Graphics processing unit. A processor with thousands of smaller cores built for parallel arithmetic, which suits the matrix multiplication at the heart of neural networks. Most large-model inference runs on GPUs or other accelerators. Read more

Token

The unit of text a language model reads and writes: a word, part of a word, or a punctuation mark, depending on the model's tokenizer. Usage is counted and priced in tokens. Here it always means a text token. Read more

Input and output tokens

Input tokens are everything sent to the model in a request, including instructions and conversation history. Output tokens are what the model generates. Providers price them separately, and output tokens usually cost more. Read more

Context window

The maximum number of tokens a model can consider in one request, counting both input and output. Longer conversations must be trimmed or summarized to fit. Read more

API

Application programming interface. A defined way for one program to ask another for something, with an agreed request format, response format and set of errors. Model providers expose their models through HTTP APIs. Read more

Endpoint

A specific URL of an API that performs one job, such as https://api.gatewayz.ai/v1/chat/completions for generating a chat response. Read more

API key

A secret string sent with each request, usually in the Authorization header, that identifies the account and authorizes use. Anyone holding the key can spend from that account, so treat it like a password. Read more

AI agent

A program that uses a model to decide on and carry out steps toward a goal, often calling tools and the model many times in a loop without a person reviewing each call. Read more

Router

The part of a gateway that decides where a request goes: which provider route serves the requested model. A router that respects the request sends it to the model named, never a different one. Read more

Inference layer

A layer between applications and model providers that owns the contract software depends on: which exact model runs, what each error status means, how spending stops, and how a failed stream is reported. A superset of a gateway. Read more

Gateway

A service that accepts model requests in one format, forwards them to one or more providers, holds provider credentials and meters usage. Unlike a plain proxy, it understands the shape of the traffic. Read more

Alias

A short, undated model name such as claude-sonnet-4-5 that points to a dated snapshot. The provider may move an alias to a newer snapshot over time, so aliases are convenient for exploration and risky for production. Read more

Dated snapshot

A model id that includes a release date, such as anthropic/claude-haiku-4-5-20251001, and identifies one fixed model version for as long as it is offered. Pin these in production. Read more

Model resolution

The deterministic mapping from a requested model name to exactly one catalog entry. In Gatewayz, an exact id runs that model, an undated alias resolves to its dated snapshot, two candidate snapshots are refused, and an unknown id returns 400 model_not_found. Read more

Substitution

Running a different model than the one requested, for example the nearest name, an available fallback or the newest snapshot, without the caller deciding to. It breaks cost reports and evaluations. Gatewayz does not substitute models. Read more

Retryable status

A status that tells the client the identical request may succeed later: 429 for rate limits and 5xx for server or upstream failures. Most official SDKs retry these automatically with backoff. Read more

Terminal status

A status that tells the client retrying the identical request cannot help, such as 400 or 402. Something about the request, the key or the account has to change first. Reporting a terminal condition with a retryable status causes retry loops and false outage signals. Read more

402 Payment Required

The status Gatewayz returns when a request cannot proceed until a limit or balance changes. The code field separates request_cap_exhausted (this key's cap is spent) from insufficient_credits (the account has no credits). Neither is retryable. Read more

Request cap

A per-key ceiling on usage. Once spent, requests with that key return 402 request_cap_exhausted until the cap is raised. Rate limiting is evaluated first, so a 429 does not consume cap. Read more

Prompt caching

Provider-side reuse of the processed form of a repeated prompt prefix. Cached input is billed at a different rate than fresh input. On the Anthropic Messages API it is requested explicitly and only engages above a minimum prompt length set by Anthropic. Read more

cache_control

The field on an Anthropic Messages API content block that marks the end of a cacheable prefix. A gateway must pass it through unchanged. Success is visible in cache_creation_input_tokens and cache_read_input_tokens; below the minimum length those fields are absent, not zero. Read more

TTFT (time to first token)

The time from sending a streaming request to receiving the first chunk that contains model content. Measure it from content, not from the first byte, since headers or empty events may arrive earlier. If it was not observed, report it as unknown. Read more

p50 / p95

The 50th and 95th percentiles of a latency distribution: what a typical request experiences, and what the slowest one in twenty experiences. More informative than an average for long-tailed distributions. Always report the sample count alongside. Read more

Failover

Sending a request to an alternative route after the first one fails with a retryable error. Failover to the same model on another route preserves behavior; switching to a different model is substitution and should be an explicit decision in the caller's code.

Circuit breaker

A client-side guard that stops sending requests to a dependency after repeated failures, then probes before resuming. It should count timeouts and 5xx responses, not client errors, or it will mistake a typo for an outage. Read more

List price

The per-token price a model provider publishes for its model. Gatewayz bills per token at the provider's list price plus a routing fee.

Routing fee

The amount Gatewayz adds on top of the provider's list price for serving a request. Any honest cost comparison with a direct provider should include it. Read more

OpenAI-compatible

An API that accepts the request and response format of OpenAI's Chat Completions API, so existing OpenAI SDKs work by changing the base URL. The Gatewayz endpoint is https://api.gatewayz.ai/v1/chat/completions, with SDK base URL https://api.gatewayz.ai/v1.

Anthropic Messages API

Anthropic's native request format, used by Claude Code and the Anthropic SDKs. Gatewayz serves it at https://api.gatewayz.ai/v1/messages; clients take the host https://api.gatewayz.ai as their base URL and append the path themselves. Read more

← All articles