What is AI inference? Training vs inference explained

Inference is the act of running a trained AI model on new input to produce an output. When you send a prompt to a language model and get an answer back, that exchange is one inference request. Training creates the model once. Inference uses it, again and again, and every use consumes computing resources.

What is the difference between training and inference?

Training is how a model is made. Inference is how a model is used. The two differ in who does them, how often they happen and what they cost.

During training, a model is shown very large amounts of data and its internal parameters, often called weights, are adjusted step by step until its predictions become useful. This work runs for weeks or months on large clusters of accelerators. It is done by the organizations that build models, and it happens once per model version, with occasional further rounds of fine-tuning.

During inference, the weights are frozen. The model does not learn from your request. It takes your input, runs it through the same fixed parameters, and produces an output. Inference happens every time anyone uses the model, which is why a popular model can process far more total requests in its lifetime than the training run ever touched. NVIDIA's overview of training and inference gives a longer introduction to the distinction.

For most developers and companies, training is someone else's problem. You pick a model a model provider has already trained, and every interaction with it after that is inference.

What happens during an inference request?

A request goes in, the model reads all of it at once, then writes its answer one token at a time until it is finished. Those two stages are usually called prefill and decode.

Step one: the input becomes tokens

Models do not read characters or words directly. Text is split into tokens, which are common chunks of text such as a short word, part of a longer word or a punctuation mark. Your prompt, any system instructions and any earlier conversation all become a sequence of input tokens. The total that fits in one request is limited by the model's context window.

Step two: prefill reads the input

In the prefill phase, the model processes all the input tokens together. Because the whole input is known in advance, this work can be done largely in parallel on a GPU. The result is an internal representation of the input that the model keeps in memory for the next phase. Prefill is a large part of how long you wait before the first piece of the answer appears, a measure known as time to first token.

Step three: decode writes the output

In the decode phase, the model generates the answer one token at a time. For each new token it looks at everything so far, the input plus the tokens it has already written, predicts the next token, appends it, and repeats. This loop continues until the model produces a stop signal or reaches a length limit.

Decode is sequential by nature. Token 200 cannot be produced before token 199 exists. That is why long answers take visibly longer than short ones, and why many APIs offer streaming, which sends each token to you as soon as it is ready instead of waiting for the full answer. NVIDIA's technical guide to LLM inference optimization describes the prefill and decode phases in more depth.

Step four: the response comes back

When decoding ends, the API returns the generated text along with usage information: how many input tokens were read and how many output tokens were produced. Those two numbers are what you are billed for.

What does an inference request look like in practice?

It is an HTTP request containing a model name and some messages, and a response containing generated text and token counts. Here is a small worked example using the OpenAI-compatible Chat Completions format, sent to Gatewayz.

curl https://api.gatewayz.ai/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAYZ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5-mini",
    "messages": [
      {"role": "system", "content": "You answer in one short sentence."},
      {"role": "user", "content": "Why is the sky blue?"}
    ]
  }'

An abridged response has this general shape. The token counts below are illustrative, and the exact numbers depend on the model's tokenizer and on what it writes.

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Sunlight scatters off air molecules, and blue light scatters the most."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 24,
    "completion_tokens": 15,
    "total_tokens": 39
  }
}

Reading it the way the model processed it:

1. The system instruction and the question were turned into input tokens, reported as prompt_tokens.
2. Prefill processed those tokens together.
3. Decode produced the answer one token at a time, reported as completion_tokens.
4. finish_reason says why decoding stopped. Here the model ended its answer naturally, rather than hitting a length limit.
5. The bill for this call is the input tokens at the input price plus the output tokens at the output price.

If you send the identical request again, the model runs again from the start and you pay again. There is no stored answer being looked up, unless you or the provider explicitly use a caching feature such as prompt caching, which reuses the processed input but still generates new output.

Why is inference the recurring cost of AI?

Because every request uses compute, and nothing about the first request makes the next one free. Training is a large bill paid once by the model developer. Inference is a smaller bill paid on every call, by whoever makes the call.

Three properties of inference drive that cost:

• Every call runs on hardware. Each request occupies accelerator memory and processing time on a machine somewhere. What is AI compute? explains what that hardware is and who owns it.
• Output is generated token by token. Each output token requires another pass through the model, so a long answer costs more work than a short one.
• Context gets re-read. A chat or an agent that sends its full history with each turn pays for those input tokens again on every turn.

This is why API pricing is quoted per token, with separate prices for input and output. Providers publish these rates, and output tokens are typically priced higher than input tokens. The Anthropic pricing page and the OpenAI pricing page both list input and output rates separately. What are AI tokens? goes further into how that pricing works.

A rough way to estimate cost

For any feature that calls a model, the cost per use is roughly:

1. Count the typical input tokens per call, including system prompt and history.
2. Count the typical output tokens per call.
3. Multiply each by its per-token price and add them.
4. Multiply by the number of calls per user action, and then by how many actions you expect.

Step four is where estimates most often go wrong. A single user action in an agent can trigger many model calls, not one.

How do agents change inference?

An agent turns one request into a loop of requests. It calls a model, reads the result, perhaps runs a tool, and calls the model again with the growing history. Each turn is a separate inference request, and each one re-sends the context so far.

That has two practical effects. Total token usage grows faster than the number of tasks, because later turns carry more input. And the calls often run without a person watching, so limits matter. A per-key request cap is one way to keep an unattended loop from spending more than intended, as covered in Spend ceilings for unattended agents.

Common misconceptions about inference

A few beliefs about inference are widespread and wrong.

• The model learns from my requests as I use it. During inference the weights are fixed. A model may be retrained later on data a provider is permitted to use, but your request does not change the model that answers it.
• Once a model is trained, using it is basically free. Every request uses compute. At scale, inference is a large ongoing cost.
• Asking the same question twice returns a stored answer. By default the model runs again and generates a fresh output, which may differ, and you pay for both.
• Only the answer costs money. Input tokens are billed too, including system prompts and conversation history sent with each call.
• A faster response means a smaller model. Response time also depends on input length, output length, streaming and how busy the serving hardware is.

Frequently asked questions

Is inference the same as running a model?

Yes. Inference is the standard term for running a trained model on new input to get an output. Each API call to a language model is one inference request, whether it comes from a chat window, an application or an agent.

Does inference change the model?

No. During inference the model's weights are fixed. Your prompt shapes the output of that one request, but it does not update the model. Changes to a model come from training or fine-tuning, which are separate processes run by the model developer.

Why are output tokens more expensive than input tokens?

Input tokens are processed together in the prefill phase, while output tokens are generated one at a time in the decode phase, each needing its own pass through the model. That sequential work is a large part of why providers usually charge more per output token. Check each provider's published price list for the actual rates.

Do I pay for inference if the request fails?

It depends on where it fails. A request rejected before it reaches a model, for example because of an invalid key or an unknown model id, does not run inference. A request that starts generating and fails partway may already have consumed compute, and how that is billed depends on the provider. Checking the usage field and the error code on each response tells you which case you are in.

What is the difference between inference and an inference layer?

Inference is the computation a model performs on a request. An inference layer is a service in front of models that handles keys, billing, model names and errors for those requests. Gatewayz is an inference layer: for the provider models in its catalog, it forwards the request and the provider running the model performs the inference.

Related

• What is Gatewayz?. One key and one endpoint for models from several providers.
• What is AI compute?. The hardware every inference request runs on.
• What are AI tokens?. How token counts become a bill.
• What is an AI agent?. Why agents multiply inference calls.
• Prompt caching through a gateway. Reusing processed input across requests.