What are AI tokens? How LLM usage is priced

AI tokens are the small units of text that language models read and generate, such as a common word, part of a longer word, a space or a punctuation mark. Model providers count the tokens in each request and response and charge per token, usually quoting separate prices per million input tokens and per million output tokens.

How does text become tokens?

A tokenizer is a fixed piece of software, trained alongside a model family, that splits text into pieces from a known vocabulary and maps each piece to a number. The model only ever works with those numbers.

Tokenizers are built so that frequent sequences get their own single token and rare sequences are assembled from smaller pieces. In practice, that means:

• A short, common English word is often one token, frequently including the space in front of it.
• A long or unusual word is split into several pieces. A word like "tokenization" might become something like "token" plus "ization", depending on the tokenizer.
• Numbers, code, URLs and names can break into many small tokens.
• Text in languages that were less represented when the tokenizer was built often needs more tokens for the same meaning.

There is no universal conversion from words or characters to tokens. The count depends on the tokenizer, the language and the kind of content. Anthropic's documentation, for example, notes that its newer models use a newer tokenizer that produces more tokens for the same input text than earlier ones, and recommends recounting prompts against the model you plan to use.

To measure rather than guess, use the provider's own tools. OpenAI publishes its tokenizer library as tiktoken, and Anthropic offers a token counting endpoint that estimates the input tokens for a request before you send it. The authoritative number for any real request is the usage object in the response.

What is the difference between input and output tokens?

Input and output tokens are the two sides of every request. Input tokens are everything the model reads. Output tokens are everything it writes back.

Input is more than the question you typed. It includes:

• the system prompt or instructions,
• the full conversation history the application resends on each turn,
• tool or function definitions, which are described to the model as text,
• documents, retrieved passages and tool results you include.

Output includes the visible answer and, on models that reason before answering, any reasoning tokens the provider counts as output.

Output is usually priced higher than input because of how the hardware works. Input tokens can be processed together in one parallel pass. Output tokens are generated one at a time, and each step reads through the model's weights again, so each output token occupies the hardware longer. The compute guide explains why.

What is a context window?

The context window is the maximum number of tokens a model can work with in a single request, counting both the input and the output it generates. Anything that does not fit is rejected or has to be removed before sending.

Context windows differ by model. Anthropic's context window documentation describes the window as holding the conversation history plus the new output, and lists sizes per model. Check the provider's documentation for the model you use.

A large window is a capacity limit, not a target. The model has no memory between API calls. A chat application creates the feeling of memory by resending the whole conversation each time, so the input grows with every turn, and you pay for the full history again on each request.

How is LLM usage priced?

Most language model APIs charge per token, with an input price and an output price, each quoted per million tokens. Both OpenAI's pricing page and Anthropic's pricing page use this format, along with separate rates for cached input.

The formula for one request is:

cost = (input_tokens / 1,000,000) * input_price
     + (output_tokens / 1,000,000) * output_price

Some requests have additional line items, such as cache writes, images, audio or batch discounts. The core model is the same: count tokens, multiply by rates.

A worked example with illustrative prices

The prices below are hypothetical round numbers chosen to make the arithmetic easy. They are not the price of any real model.

A single support answer. The request carries a system prompt, a short history and the user's question, 3,000 input tokens in total. The model replies with 500 output tokens.

• Input: 3,000 / 1,000,000 * $2.00 = $0.006
• Output: 500 / 1,000,000 * $8.00 = $0.004
• Total: $0.010 per request

At 10,000 such requests a day, that is $100 a day. Notice that output was a sixth of the tokens but two fifths of the cost.

The same prompt with a cached prefix. Suppose 2,500 of those input tokens are a stable system prompt that the provider serves from its cache.

• Cached input: 2,500 / 1,000,000 * $0.20 = $0.0005
• Fresh input: 500 / 1,000,000 * $2.00 = $0.001
• Output: $0.004, unchanged
• Total: $0.0055 per request

An agent loop. An agent runs 20 steps. Each step resends the growing transcript, starting at 3,000 input tokens and adding about 1,000 tokens per step as tool results pile up, with 300 output tokens per step. Input across the run is 3,000 + 4,000 + ... + 22,000, which is 250,000 tokens. Output is 6,000 tokens.

• Input: 250,000 / 1,000,000 * $2.00 = $0.50
• Output: 6,000 / 1,000,000 * $8.00 = $0.048
• Total: about $0.55 for one task

The per-step cost looks small, but the resent history dominates. That pattern is why agent workloads often cost far more than their individual prompts suggest.

How does Gatewayz bill for tokens?

Gatewayz bills per token at the provider's list price for the model you called, plus a routing fee, deducted from one prepaid balance. You use one API key across providers instead of holding a separate account and balance with each.

A few properties follow from that:

• The model you name is the model you are billed for. Aliases resolve to one exact dated snapshot, and an unknown id returns 400 model_not_found. Gatewayz never runs a different model at a different price. See Resolution, not substitution.
• Billing records hold metadata, not content. On ordinary API calls, Gatewayz keeps token counts, cost, model, status and timing for billing, not the prompt or completion content.
• An empty balance stops requests cleanly. When the balance runs out, requests return 402 insufficient_credits, which is not retryable until you top up.

The public catalog at https://api.gatewayz.ai/v1/models lists the available models. Current pricing details are on the pricing page.

What is cached input pricing?

Cached input pricing is a lower rate for input tokens the provider has already processed recently, such as a long system prompt or document repeated at the start of many requests. Providers list it as its own column on their pricing pages.

On the Anthropic Messages API, caching is explicit: you mark the cacheable prefix with a cache_control block, and the response's usage reports cache_creation_input_tokens and cache_read_input_tokens separately. OpenAI describes its own approach in its prompt caching guide. Writing to a cache can be priced above normal input, so caching pays off when the same prefix is read again within the cache lifetime.

Caching has to survive the trip through any middle layer, and small test prompts can mislead because providers set a minimum cacheable length. Prompt caching through a gateway covers how to verify it.

How can you control LLM costs?

Control cost by limiting what the model generates, limiting what you send, reusing what you can, and putting hard limits on how many requests can run. Each lever maps to a term in the pricing formula.

Cap output with max_tokens

Set max_tokens on every request. It sets the maximum number of output tokens the model may generate, so it bounds the most expensive part of the bill. Pick a value that fits the task: a classification needs a handful of tokens, a summary a few hundred.

curl https://api.gatewayz.ai/v1/messages \
  -H "Authorization: Bearer $GATEWAYZ_API_KEY" \
  -H "content-type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "anthropic/claude-haiku-4-5-20251001",
    "max_tokens": 200,
    "messages": [
      {"role": "user", "content": "Summarize this ticket in two sentences: ..."}
    ]
  }'

The response's usage object reports input_tokens and output_tokens. Log them per request so cost is measured, not estimated.

Trim the context you send

• Drop or summarize old turns instead of resending an entire conversation.
• Send only the retrieved passages that matter, not whole documents.
• Include only the tool definitions a step can actually use.
• Truncate long tool results before feeding them back to an agent.

Reuse and right-size

• Put stable content, such as instructions and reference material, at the start of the prompt so it can be cached.
• Use a smaller model, such as openai/gpt-5-mini or anthropic/claude-haiku-4-5-20251001, for routing, extraction and classification steps, and reserve larger models for steps that need them. Test quality on your own tasks before switching.

Set request caps

A per-key request cap limits how many requests a key can make. On Gatewayz, you set max_requests when creating or updating a key. When it is spent, the key returns 402 request_cap_exhausted, which is not retryable until the cap is raised. Give each agent or environment its own key with its own cap, so a runaway loop stops instead of draining the shared balance. Spend ceilings for unattended agents covers the pattern in detail.

Frequently asked questions

How many words are in a token?

There is no fixed ratio. Common short English words are often a single token, while long words, code, numbers and many non-English languages split into several tokens. The count also changes between tokenizers, so the reliable approach is to measure with the provider's tokenizer or token counting endpoint and to read the usage object returned with each response.

Why are output tokens more expensive than input tokens?

Input tokens can be processed together in one parallel pass, while output tokens are generated one at a time, with each new token requiring another pass through the model's weights. Each output token therefore occupies the hardware for longer, and providers generally price output higher to reflect that.

Do I pay for the whole conversation on every message?

Usually yes. Models keep no memory between API calls, so chat applications resend the conversation history as input on each turn, and every resent token is billed again. Summarizing old turns, trimming history and caching stable prefixes are the main ways to keep that growth in check.

Does Gatewayz charge more than going to the provider directly?

Gatewayz bills the provider's list price for the model plus a routing fee, from one prepaid balance. In exchange you use one key and one endpoint across providers. Whether that suits you depends on how much you value consolidating keys, balances and billing, and the pricing page has the current details.

Does max_tokens reduce my input cost?

No. max_tokens only caps how many output tokens the model may generate. Input cost depends on what you send, so reducing it means trimming the prompt, history, documents and tool definitions, or using cached input for repeated prefixes.

Related

• What is AI compute?. Why tokens cost what they do.
• Prompt caching through a gateway. How to confirm cached input pricing actually applies.
• Spend ceilings for unattended agents. Request caps as a hard stop for loops.
• Resolution, not substitution. Why the model you are billed for is the one you asked for.