What is an AI agent? How agents change inference

An AI agent is a program that pairs a language model with tools and a loop. The model reads the current situation and decides the next step, the program executes that step with a tool, feeds the result back, and repeats until the task is finished or something stops it. No person needs to review each step.

What is an AI agent, exactly?

An agent is a loop around a model call where the model's output can trigger actions, and the results of those actions become the next input. The model plans; the surrounding program executes.

Anthropic's engineering essay Building effective agents draws a useful line between workflows, where code follows a predefined path and calls a model at fixed points, and agents, where the model directs its own process and decides which tools to use and when. Both involve many model calls. The agent version is harder to predict, because the number of steps depends on what the model decides along the way.

The three parts

1. The model. It receives the task, the conversation so far, and a description of the tools available. It responds either with a final answer or with a request to use a tool.
2. The tools. Functions the program exposes: search a codebase, run a test, query a database, send an email. Model APIs support this directly. In Anthropic's tool use documentation, the model returns a structured tool call with a name and arguments, your code runs it, and you send the result back in the next request.
3. The loop. The program keeps calling the model with updated context until the model says it is done, a step limit is reached, or an error ends the run.

A coding agent fixing a failing test is a typical example. It reads the error, opens the relevant file, proposes an edit, runs the tests, reads the new output, and tries again. Each of those turns is at least one inference request, and each request carries the growing history of the task as input tokens.

How is an agent different from a chatbot?

A chatbot waits for a person between every request; an agent does not. That single difference changes the volume of calls, who handles failures, and how costs accumulate.

The last two rows are where agents fail in production. A person who hits the same error three times gives up. A loop does exactly what it was written to do, indefinitely.

Why do error codes matter so much for agents?

Because the first reader of every error in an agent system is a program, and programs do not read prose. They read the status code and, if the API provides one, a machine-readable error code.

HTTP status classes already carry most of the meaning. RFC 9110 defines 4xx as a client error, meaning the request itself needs to change, and 5xx as a server error, meaning the request may succeed later. The 429 Too Many Requests status comes from RFC 6585 and tells the client to slow down, sometimes with a Retry-After header saying how long to wait.

The question an agent has to answer for each failure is simple: could sending the identical request again ever succeed? If yes, it is a retryable status. If no, it is a terminal status, and retrying only burns time and money. The article Who reads the error? walks through how a single misclassified error can turn a config typo into what looks like a provider outage.

The shape of a Gatewayz error

Gatewayz returns errors in one consistent body, so an agent can branch on fields instead of matching strings:

{
  "error": {
    "message": "Model 'anthropic/claude-sonet-4-6' does not exist. See GET /v1/models for available model ids.",
    "type": "invalid_request_error",
    "code": "model_not_found"
  }
}

That is the response to a misspelled model id: status 400, with a code that will not change when someone rewrites the message. The message is for the log. The status and the code are for the loop.

How should an agent react to each Gatewayz error?

Stop on errors that need a human or a config change, and retry with backoff only on errors that can clear by themselves. The table below is the contract an agent can rely on.

Two details are worth stating plainly. The two 402 responses share a status because both require money or limits to change, and they have different codes because different people usually fix them: an engineer raises a per-key cap, a billing owner tops up the balance. And a 5xx retry must still be bounded. An upstream that has failed five times in a row is telling you something, and a loop that retries forever turns a short incident into a long bill of failed attempts.

Streaming responses need the same discipline. A stream that fails partway through should end with an explicit error event, not simply stop, so the agent does not mistake a truncated answer for a complete one. The details are in Streaming failures agents can detect.

What does that handling look like in code?

A few dozen lines are enough. The sketch below calls the OpenAI-compatible endpoint directly with httpx, so the retry policy is explicit rather than hidden inside an SDK default.

import os, random, time
import httpx

URL = "https://api.gatewayz.ai/v1/chat/completions"
HEADERS = {"Authorization": f"Bearer {os.environ['GATEWAYZ_API_KEY']}"}

class StopAgent(Exception):
    """A terminal failure. Do not retry; alert a human."""

def call_model(messages, model="anthropic/claude-sonnet-4-6", max_attempts=5):
    for attempt in range(max_attempts):
        r = httpx.post(URL, headers=HEADERS, timeout=120,
                       json={"model": model, "messages": messages})
        if r.status_code == 200:
            return r.json()

        err = r.json().get("error", {}) if r.content else {}
        code = err.get("code")

        if r.status_code in (400, 401, 402):
            # Terminal: model_not_found, invalid_api_key,
            # request_cap_exhausted, insufficient_credits.
            raise StopAgent(f"{r.status_code} {code}: {err.get('message')}")

        if r.status_code == 429 or r.status_code >= 500:
            retry_after = r.headers.get("Retry-After")
            if retry_after and retry_after.isdigit():
                delay = int(retry_after)
            else:
                delay = min(60, pow(2, attempt)) + random.random()
            time.sleep(delay)
            continue

        raise StopAgent(f"Unexpected status {r.status_code}")

    raise StopAgent(f"Gave up after {max_attempts} attempts")

The important property is not the backoff formula. It is that every path out of the function is either a success, a bounded retry, or a named stop. If you use an official SDK instead, check its default retry behavior for your version and make sure it treats 400, 401 and 402 as terminal.

How do agent loops run away?

A runaway loop is an agent that keeps making requests after it has stopped making progress. It rarely looks dramatic from the inside; each step seems reasonable. The common patterns are:

• Retrying a terminal error. A misspelled model id or an empty balance is retried on a timer. Nothing will ever change, but the loop cannot tell.
• No step limit. The model keeps asking for one more tool call, and the program has no maximum number of turns.
• Oscillation. The agent edits a file, a test fails, it reverts the edit, the original test fails, and the cycle repeats.
• Context growth. Each turn appends tool output, so each request is larger and more expensive than the last, until it hits the context window and starts failing.
• Fan-out. One agent spawns sub-agents, each with its own loop, multiplying every one of the problems above.
• Unattended schedules. A job that fails at night keeps failing until someone checks in the morning.

A checklist before you let an agent run unattended

1. Set a maximum number of turns per task, and a maximum wall-clock time.
2. Classify every error as retryable or terminal in code, using status and code, not message text.
3. Bound retries on 429 and 5xx, with backoff and jitter.
4. Detect lack of progress, for example the same tool call with the same arguments twice in a row.
5. Give each agent or environment its own API key, so one runaway does not exhaust everyone's access.
6. Set a request cap on that key, so the worst case has a ceiling you chose in advance.
7. Log the model, status and code for every request, so a failure can be diagnosed without replaying it.

Why does an agent need a spend ceiling?

Because the loop's own logic is not a reliable limit. A bug in the exit condition, a misclassified error, or a model that never declares the task finished can all keep requests flowing. A ceiling enforced outside the agent still holds when the agent's logic does not.

On Gatewayz, a key can carry a request cap, set when the key is created or updated. When the cap is spent, the key returns 402 with code request_cap_exhausted, and an agent following the table above stops and alerts instead of retrying. That turns an unbounded failure into a bounded one: the worst case is the cap you set, not whatever the loop managed before someone noticed. The design reasoning is in Spend ceilings for unattended agents.

A cap does not replace a good loop. It is the backstop for the day the loop is wrong.

Frequently asked questions

Is an AI agent the same as a chatbot?

No. A chatbot answers one message at a time and waits for a person to read the reply and send the next one. An agent uses the model to choose actions, runs tools, and keeps going on its own until a stopping condition is met, so it makes many more requests and handles its own failures in code.

Does an agent need a special kind of model?

Not necessarily. Any model that supports tool use, where it can return a structured request to call a function, can drive an agent loop. Capability differences between models affect how well the agent plans, but the loop, the tools and the error handling are written in your own code and work the same way whichever model you choose.

Should an agent retry a 402 error?

No. On Gatewayz, both 402 codes are terminal: request_cap_exhausted means the key's request cap is spent and insufficient_credits means the account balance is empty. Neither clears by waiting. The agent should stop and alert whoever can raise the cap or add credits, then resume once that has happened.

What is a safe number of retries for a 5xx error?

There is no universal number, but it should be small and fixed, with exponential backoff and some random jitter between attempts. A handful of attempts covers brief upstream failures. Beyond that, stop, record the failure, and let a person or a scheduler decide when to try the task again, rather than retrying inside the loop indefinitely.

How do I stop an agent from overspending?

Combine limits inside and outside the agent. Inside, cap turns per task, bound retries, and treat terminal errors as stops. Outside, give each agent its own key with a request cap, so that even a loop with a bug cannot send more requests than you allowed. See Spend ceilings for unattended agents for the reasoning.

Related

• Who reads the error?. Why status codes are control flow for agents.
• Spend ceilings for unattended agents. How per-key caps bound a runaway loop.
• Streaming failures agents can detect. Telling a truncated stream from a finished one.
• What is an AI gateway? Why model routing matters. One integration for many models.
• What is AI inference?. The request an agent makes on every turn.