What is AI compute? CPUs, GPUs and who owns them

AI compute is the physical processing capacity used to train and run AI models: the chips that do the arithmetic, the fast memory that holds a model's weights, and the servers and power behind them. When you send a prompt to a model, compute turns it into a response, and it is the main cost behind the price.

What does AI compute actually mean?

Compute is the hardware capacity that performs the calculations a model needs, measured in practice by how many chips you have, how fast they are, and how much memory each one carries. The word covers both a single chip and the rented capacity of a whole data center.

AI work splits into two phases that use compute differently. Training is the one-time, very large job of adjusting a model's numbers until it performs well. Inference is every later use of the finished model: each chat message, each agent step, each API call. Training happens rarely and in bulk. Inference happens constantly, one request at a time, for as long as the model is in service. If you use AI through an API, the compute you are paying for is inference compute. The inference guide covers that distinction in more depth.

A useful mental model: a trained model is a file of numbers, called weights or parameters. Inference loads that file into a processor's memory and pushes your input through it, layer by layer, until a result comes out. Compute is the machine doing the pushing.

What is the difference between a CPU and a GPU?

A CPU is a general purpose processor built to run many different kinds of instructions quickly, one after another, with complex branching. A GPU is a processor built to run the same simple operation on huge amounts of data at the same time.

Your laptop's CPU is excellent at the work of an operating system, a browser or a database: lots of different small tasks, decisions and jumps. A GPU started life drawing graphics, where the same calculation is applied to millions of pixels at once. That parallel design turned out to fit neural networks closely.

GPUs are the best known accelerator, but they are not the only one. Several companies build other accelerators designed specifically for neural network math. The principles in this guide apply to all of them: parallel arithmetic plus fast memory close to the chip.

Why do AI models need GPUs and other accelerators?

Models need accelerators because running a model is dominated by one kind of arithmetic, multiplying large matrices, and accelerators do that arithmetic in parallel while CPUs mostly do it in sequence. Three factors explain the gap.

Parallel matrix math

A language model is built from layers, and most of the work in each layer is multiplying a matrix of inputs by a matrix of learned weights. NVIDIA's documentation describes general matrix multiplication as a fundamental building block of fully connected, recurrent and convolutional layers, and describes the GPU as a highly parallel processor architecture composed of processing elements and a memory hierarchy. See NVIDIA's GPU Performance Background guide and its matrix multiplication guide.

Each element of a matrix product can be computed independently of the others. A processor with thousands of units working at once finishes the whole product far sooner than one that works through it a few pieces at a time. Software stacks such as NVIDIA's CUDA platform exist to let frameworks hand this arithmetic to the GPU.

Memory bandwidth

Calculation speed is only half the story. Before a processor can multiply numbers, it has to load them from memory, and after it has to store the result. For some operations, the limit is how fast data moves in and out of memory, not how fast the chip can calculate. NVIDIA's documentation calls these memory-bound calculations and notes their performance depends on the number of inputs, outputs and weights loaded and stored per pass. See its guide to memory-limited layers.

Text generation is a clear case. A model produces its answer one piece at a time, and every new piece requires reading through the model's weights again. The chip spends much of that time waiting on memory. That is why accelerators pair their processors with very high bandwidth memory mounted close to the chip, and why memory capacity and speed are headline specifications for AI hardware.

The weights must fit

A model's weights have to sit in the accelerator's own memory while it runs. Moving them back and forth from slower system memory on every step would make generation impractical.

The arithmetic is simple. Each parameter stored at 16-bit precision takes 2 bytes. So a hypothetical model with 70 billion parameters needs about 140 GB just to hold its weights, before counting working memory for the conversation in progress. If one accelerator does not have that much memory, the model is split across several accelerators linked by fast interconnects, which adds cost and engineering complexity. Techniques such as storing weights at lower precision shrink the footprint, with trade-offs in output quality that vary by model.

This is the practical reason large models are served from data centers and not from ordinary servers or laptops: the memory requirement alone rules most hardware out.

Who owns AI compute?

Most AI compute is owned by three groups: the companies that build models, the large cloud platforms that rent hardware to others, and independent owners who run GPUs outside those two. Ownership matters because the owner decides who can use the hardware and what it costs.

Model providers

A model provider is the company that trains a model and offers it for use, such as OpenAI or Anthropic. Providers operate or reserve very large amounts of compute, first to train models and then to serve them. When you call a provider's API, your request runs on hardware the provider controls, under the provider's own data and capacity policies.

Cloud platforms

Cloud platforms buy accelerators at scale and rent them by the hour or on longer commitments. Some teams rent raw GPU servers and run open-weight models themselves. Others use the cloud's managed model services. Either way, the cloud owns the hardware and the renter pays for capacity whether or not it is fully used.

Independent GPU owners

Outside the large operators, there are many GPUs in smaller data centers, research labs and individual machines. Networks that pool this capacity can offer inference on hardware that would otherwise sit idle. The trade-off is trust: when the operator is not a large, audited company, you need some way to check that the work was done correctly, and you should understand that the machine running your request can see its content.

Gatewayz runs a community GPU network in beta built on this model. Independent node operators run requests, and their work is checked by sampled replay, meaning a sample of completed requests is run again to confirm the result matches. It is opt-in per request: you choose it by using a model id with the community/ prefix, and requests without that prefix do not use it. A community node sees the prompt and response content it runs, but not the identity of the requester. If that visibility is not acceptable for a given workload, use a provider model id instead.

How does compute ownership connect to what inference costs?

The price you pay per request is, underneath, the cost of the compute that ran it plus the owner's margin. Hardware purchase, power, cooling, networking and idle capacity all have to be recovered from usage.

Providers usually express that cost as a price per token, the small chunks of text a model reads and writes. A few consequences follow directly from the hardware:

• Bigger models cost more per token. More parameters means more memory and more arithmetic for every token, so each token uses more compute.
• Output usually costs more than input. Input can be processed in parallel in one pass. Output is generated one token at a time, each step reading through the weights again, so it ties up the hardware for longer.
• Long conversations cost more. The model has to process everything in its context on each request, and working memory grows with context length.
• Reused prefixes can cost less. When a provider can reuse work already done for an identical prompt prefix, it saves compute and often passes some of that through as a lower cached input price.

The tokens and pricing guide walks through these charges with a worked example.

When you use a provider model through Gatewayz, the compute belongs to the provider that runs the model. Gatewayz bills that usage per token at the provider's list price plus a routing fee, from one prepaid balance. Gatewayz has no model of its own and does not change which hardware or which model the provider uses for your request. The public catalog at https://api.gatewayz.ai/v1/models lists the models available and where each comes from.

Common misconceptions about AI compute

A few beliefs come up often and are worth correcting.

• More GPUs always means faster answers. Splitting a model across more chips adds communication between them. For generation, memory speed and interconnects often matter as much as chip count.
• Compute is only a training problem. Training is a large one-time cost. Inference runs continuously for every user and every agent step, so over a model's service life it can use a great deal of compute.
• Any GPU can run any model. A model only runs if its weights, plus working memory, fit in the available accelerator memory.
• The API provider and the hardware owner are always the same company. Sometimes they are. Often a provider leases capacity from a cloud, and a gateway sits in front of providers without owning any hardware.

Frequently asked questions

Can a CPU run a large language model?

A CPU can run small models, and some tools make this practical for local experiments, but large models run far more slowly on a CPU because it performs matrix arithmetic with much less parallelism and has much slower access to the memory holding the weights. For production workloads serving many requests, accelerators are the norm.

Why is GPU memory so important for AI?

A model's weights must sit in the accelerator's memory while it runs, and generating each new token requires reading through those weights again. If the memory is too small, the model does not fit at all. If it is too slow, the processor waits and generation slows down, regardless of how fast the chip calculates.

Does Gatewayz own GPUs?

For provider models, no. Requests to models such as anthropic/claude-sonnet-4-6 or openai/gpt-5 run on the provider's infrastructure, and Gatewayz forwards them. Separately, Gatewayz runs a community GPU network in beta that you can opt into per request with a community/ model id, where independent node operators run the work and it is checked by sampled replay.

Who can see my prompt on the community GPU network?

The node that runs a community request sees the prompt and response content for that request, because it has to process them. It does not receive the requester's identity. The network is opt-in per request, so if a workload should not be visible to an independent operator, use a provider model id instead.

Why does output cost more than input for most models?

Input tokens can be processed together in a single parallel pass, while output tokens are generated one at a time, with each step reading through the model's weights again. That makes each output token occupy the hardware for longer, and providers generally price output tokens higher to reflect it.

Related

• What is AI inference?. How running a model differs from training it.
• What are AI tokens?. How compute cost becomes a per-token price.
• What is an inference layer?. Where a gateway sits between you and the hardware.
• Prompt caching through a gateway. Paying less for input the provider has already processed.