Why Output Tokens Cost More Than Input Tokens — A Hardware Drill-Down

Every AI provider charges you separately for input and output tokens, and output is the expensive half — four to eight times the input rate (Anthropic prices every Claude model at exactly x5 for output tokens). That gap isn’t arbitrary. If you’d like to know why, this blog might be the one for you.

The usual explanation you will find is that generating text is more “computational” work than reading it. My understanding is a bit different. To understand it we have to go all the way down to the hardware — what the chip actually does, where the different pieces of a model live (the weights, above all — more on those shortly), and how long the trip from one of those places to the other takes.

What “computation” actually means

To understand computation we need to understand FLOP. It stands for FLoating-point OPeration, and it means one arithmetic act on a decimal number: one multiply, or one add. 3.7 × 0.02 is one FLOP. 1.4 + 0.9 is one FLOP. The units stack in thousands — KFLOP (thousand), MFLOP (million), GFLOP (billion), TFLOP (trillion), PFLOP ( quadrillion). And one distinction is worth noting:

• FLOPs (lowercase s) is an amount of work. Like miles.
• FLOPS (capital S) is work per second. Like miles per hour.

An NVIDIA H100, one of the cards serving the models you use, is rated at 989 TFLOPS — 989 trillion operations every second. That’s the chip we’ll be using throughout.

The arithmetic in an LLM is: multiply, then add.

To understand what arithmetic an LLM is doing, let’s understand tokens. A token is a point in a space with thousands of directions — and a point is just its coordinates, one number per direction.

Imagine it like this. The model has a vocabulary: every word fragment it can read or write, each one is an entry with a fixed number. For example Llama 3 70B has 128,256 of them. And it has an embedding dimension: how many directions that space has, which for this model is 8,192. Take the word “bank”. That’s one entry in the vocabulary, and what the model looks up under that entry is a list of 8,192 numbers.

That’s the whole representation. A token is a list of 8,192 numbers and nothing else, with points close together meaning similar things.

And that lookup is fixed. There is one entry for “bank,” holding one list, so every “bank” the model ever reads starts from the same 8,192 numbers — which can’t be right, because the right numbers depend on the neighbours. Take “river bank” and “bank loan” — “bank” arrives with the identical list in both, and the model has to pull it toward the water region of the space in one and the money region in the other. You did that just now without noticing, you never even considered the money sense of the first one. The model has no such instinct. All it has is arithmetic, so it adjusts each token’s numbers using the other tokens’ numbers, and the only way to do that is by multiply and add.

That step is called attention: each token compares itself against every other token, then takes most from the ones that matter to it.

So every layer, every bit of what looks like understanding is one operation, run many many times:

take a number, multiply it by a weight, add it to a running total

That’s a multiply-accumulate: 2 FLOPs, one multiply and one add.

A weight is just a number — the part that was learned. Training worked through an enormous amount of text, settled on values that make these multiply-adds come out useful, then froze them. The same weights run for every token, every request, every user. That’s what the “70B” counts: 70 billion weights, also called parameters.

To see the shape of it, shrink a token’s list to three numbers. Inputs a b c, outputs x y z:

1
2
3

x = a·w1 + b·w2 + c·w3
y = a·w4 + b·w5 + c·w6
z = a·w7 + b·w8 + c·w9

Each output gets its own row of weights, and each weight is used exactly once — once in a multiply, once in an add. 9 weights → 18 FLOPs. Three numbers in, three out — so the next layer does the same thing again with nine fresh weights. Real models do this with the 8,192-number lists from above, which means more weights and more arithmetic, but the ratio between them never moves:

Every weight in the model participates in exactly one multiply and one add, per token. So: 2 FLOPs per weight, per token.

That covers every weight — but not quite every multiply-add. Attention also compares tokens against each other, multiplying token numbers by other token numbers, with no weight involved. Those sit outside the 2-FLOPs-per-weight count, and we’ll come back to them later in the post.

Why moving the weights matters to the cost

A multiply-add can only happen if the weights are sitting right where the multiplying happens. And on a GPU, those are two separate pieces of hardware.

There is the chip, which does all the arithmetic — for a 70B model, 70 billion multiply-adds per token, 140 GFLOP, all of it inside the chip and nowhere else. And there is the memory, the GPU’s VRAM, where the weights are kept. Memory doesn’t compute anything; it only holds numbers. So every weight has to be carried from VRAM over to the chip before it can be multiplied by anything.

How big is a model, in bytes?

A weight is a number, and a number takes up space. How much depends on the precision you store it at. The serving standard is BF16: 16 bits, so 2 bytes per weight. Which turns a parameter count into a file size:

1

70,000,000,000 weights × 2 bytes = 140,000,000,000 bytes = 140 GB

That’s where the 140 GB comes from. The chip does have a little memory of its own — SRAM, much faster but there is very little of it: tens of megabytes, against that model’s 140,000 megabytes of weights. So a chip can never hold all of the model weights. The weights live in VRAM, and they have to make the trip.

That trip has a speed limit, and it has a name — memory bandwidth. For an H100 it’s 3.35 TB/s.

140 GB also doesn’t fit on one card, an H100 holds 80 GB. Two would hold it, but that leaves almost nothing spare, and you’ll see at the end why that isn’t enough. So in practice you spread it wider — say four cards, each holding 35 GB, every layer cut four ways, each card doing a quarter of the multiply-adds.

And 35 GB still doesn’t fit on a chip. So the weights never leave VRAM. The card sends the chip a small block at a time, the chip uses it and throws it away to make room for the next one:

That is the trip every weight has to make. Now let’s see how many times it has to happen when the model reads your input, and when it writes its response.

Reading happens all at once. Writing happens one token at a time.

When you send your input to the LLM there are two phases:

Prefill is the reading phase. The model takes your whole input and runs it through itself once, start to finish — that single trip through every layer is called a forward pass. Decode is the writing phase. The model produces the response, one forward pass per token.

The difference between them comes down to one question. Before the model can do any arithmetic on a token, that token’s numbers have to exist:

For the tokens I’m about to work on, do their numbers already exist — or do I have to produce them first?

During prefill, they exist. Your whole prompt exists before the model starts — the first token and the nine-hundredth are equally available, and nothing in it is waiting on a guess the model hasn’t made yet. So every token can be worked on at the same time: the model runs layer 1 for all thousand tokens, then layer 2 for all thousand, and so on down the stack. A thousand tokens of input is one forward pass. (Input tokens do depend on each other — that’s attention, token 900 has to look at all 899 before it. But you typed those; they’re already there.)

Output is where that breaks. To write token 5 the model must know token 4 — and token 4 doesn’t exist until it has been produced. It has to be made first, then fed back in. There’s no way to do the two at once, because one of them is an input to the other.

Note what does not happen: the prompt isn’t run through the model again. Each pass carries only the newest token, because everything earlier was saved the first time it was computed — that’s the KV cache, and we’ll come back to it.

That’s what “autoregressive” means: a thousand output tokens, a thousand forward passes, in strict order. Same model, same arithmetic per token — one phase gets to do it in bulk, the other isn’t allowed to.

What one forward pass actually costs

One output token. A forward pass walks through the model’s layers in order, and a 70B model has 80 of them. At each layer, that layer’s weights are pulled from VRAM to the chip, used, and then overwritten by the next layer’s — there’s nowhere to keep them. So producing a single token means:

1. Pulling all 140 GB of weights out of memory — 140 GB ÷ 3.35 TB/s = 41.8 ms
2. Doing 140 GFLOP of arithmetic with them — 140 GFLOP ÷ 989 TFLOPS = 0.14 ms

The chip can’t multiply a weight that hasn’t arrived yet, so memory bandwidth sets the pace: the token takes 41.8 ms, and only 0.14 ms of that is arithmetic. The chip spends 99.7% of the token waiting for bytes.

A note on these numbers. They’re one H100’s specs, so they describe one card doing all the work. On the four-card split each card reads only its own 35 GB, all four at once, and a real decode step is closer to 10 ms. I’ll keep the single-card figures throughout: every ratio below is unchanged, and the ratios are the entire argument.

Then it starts over for the next token, and it can’t reuse a thing. Token 2 begins at layer 1, whose weights left the chip 79 layers ago. The copy in VRAM is still sitting there, untouched — it just has to make the trip again.

One thousand input tokens. Same model, same weights, same memory. But prefill has all 1,000 tokens in hand, so they travel through the layers together:

1. Pulling all 140 GB of weights out of memory — the same 41.8 ms (once, not 1,000 times)
2. Doing 140 TFLOP of arithmetic with them — 1,000 × 0.14 ms = 141 ms

Nothing was skipped. That 141 ms is a thousand tokens’ worth of multiply-adds. What changed is step 1: layer 1’s weights arrive once and serve all 1,000 tokens before they’re discarded. A thousand output tokens means a thousand trips through 140 GB. A thousand input tokens means one.

Which is the whole comparison. Loading and math happen at the same time, so you wait for whichever is slower:

The math bar is the same length in both — same model, same chip, same arithmetic per token. Only the loading bar moved, because output drags 140 TB past the chip where input drags 140 GB — a thousand trips through 140 GB is 140 TB, so that bar is just the single-token read done a thousand times: 1,000 × 41.8 ms = 41,791 ms. Output’s total goes from 141 milliseconds to 41.8 seconds, with the chip using 0.3% of its arithmetic capacity the entire time.

The compute is the same on both sides. The waiting is what differs.

Input isn’t linear

That 141 ms was measured at a thousand tokens. Change the prompt length and it doesn’t move in proportion, because a forward pass contains two kinds of multiply-add and only one has appeared so far.

Token × weight. Every token gets multiplied by every weight, so every token costs the same: twice the prompt, twice the work. This is the arithmetic the 141 ms measured.

Token × token. Attention compares each token against the ones before it, no weights involved. Here the tokens don’t cost the same. Token 10 has 9 tokens to compare itself against, token 100,000 has 99,999. Late tokens cost far more than early ones, and the longer your prompt, the more late tokens it has.

So input cost is two costs added together. Double your prompt and the weight part doubles with it. The attention part doesn’t — double the tokens and you get four times the comparisons.

At a thousand tokens that second part is too small to matter, which is why every figure above ignored it. Somewhere around fifty thousand tokens the two are equal. By a hundred thousand, the comparing costs twice as much as the weight math.

One GPU serves many users at once

To keep the arithmetic simple, everything above assumed the GPU is generating for exactly one person. In reality no provider runs that way, and serving many people at once changes what the weight reloading costs.

Remember what made decode so expensive: 41.8 ms pulling weights, 0.14 ms using them — for 99.7% of every token, the chip has nothing to do. That idle time is what makes serving many users at once nearly free: the weights are being read anyway, so the same trip through memory can feed more than one request.

Batching does exactly that. Gather more users who all need the next token of their own response — say 295 of them — load layer 1’s weights once, and use them for all 295 before letting them go. 295 is the batch size, and the number comes from the hardware: the read takes 41.8 ms, one user’s math takes 0.14 ms, so that many users fit inside the time the read was going to take anyway. It’s the point where the chip is finally busy.

It’s the same 140 GB read every time — that’s why the column “Reading the weights” never moves, and why the step takes 41.8 ms whether it serves one person or 295. Only the math column grows, filling time the chip was already spending on the read.

Which leaves the obvious question: if 295 people can split one read, why is output still the expensive side?

The KV cache is what batching can’t fix

To write its next token, a sequence needs attention over everything before it — every key and value for every prior token. Calculating those again each time would mean redoing the entire prompt over and over, so they’re saved instead. That’s the KV cache, and every decode step re-reads all of it.

The weights are shared. The KV cache is not.

The weights were frozen when training ended, and everyone in the batch is running those same numbers — one fetch serves every user at once. But your KV cache is your conversation. Nobody else can use it. It must be read for you, separately, every step.

And it has to sit somewhere — the same VRAM holding the weights. Those four H100s have 320 GB between them; take out the 140 GB for model weights and the ~45 GB the server needs, and there’s roughly 135 GB left to hand out. The cache grows with every token of context, so the longer the conversations, the fewer of them fit:

The math had room for 295 people. Memory runs out at 50 — or at three. * Memory is what you run out of, never the chip.*

And that’s the answer to the question. Take the 8K row: 50 users in the batch, so a decode step still reads all 140 GB in 41.8 ms but only does 7 ms of math. The chip sits at 17% — while prefill, with all 1,000 tokens in hand, runs it at 100%.

1
2

Output, batch 50 : 50 tokens per 41.8 ms ≈ 1,200 tokens/second
Input, prefill   : 1,000 tokens per 141 ms ≈ 7,000 tokens/second

Batching was supposed to close that gap, and it would have — the cache runs out of room and bandwidth first. And it compounds: longer contexts mean fewer users in the batch, and fewer users mean a costlier output token.

Conclusion

Input and output cost the same arithmetic per token. What differs is memory, not math. Writing one token on its own means reading all 140 GB of weights to produce that one token.

Serve hundreds of users at once and they split that read between them, so the cost per token drops to what prefill was already getting. The KV cache never splits. Every conversation carries its own, nobody can share it, and it grows with every token of context — until what you run out of is memory, not arithmetic.

So what you’re actually paying for is time on the GPU, and three different things decide how much:

Input is billed as compute, output is billed as bandwidth, and context length is billed as space in memory.

Sources

Hardware

• NVIDIA H100 product page — 989.5 TFLOPS dense BF16; 3.35 TB/s HBM3; 80 GB
• NVIDIA — KV cache offload with CPU–GPU memory sharing — ~ 140 GB of weights for Llama 3 70B at FP16; ~40 GB of KV cache at a 128K context

Model mechanics

• Llama 3 70B config.json — vocab_size 128,256 and hidden_size 8,192
• Transformer Explainer (Georgia Tech) — interactive visualization of GPT-2 Small
• Attention Is All You Need — §3.2.3 on preserving “the auto-regressive property,” which is what forces decode to run one token at a time and lets prefill run all at once

Cost structure at long context

• Meta Engineering — Scaling LLM inference — Llama 3 405B: 128K-token prefill in 3.8 s, 1M-token prefill in 77 s — 8× the tokens, 20× the time, which is “double the tokens, four times the comparisons” showing up in a measurement

Pricing

• Anthropic pricing documentation — the input/output rates behind the x5 in the opening