The two phases
Prefill processes the input prompt. All prompt tokens are available at once, so the matrix multiplications are large and parallel: high arithmetic intensity (many FLOPs per byte of weights fetched). Prefill is compute-bound — it saturates the FLOP units of a GPU and benefits from raw TFLOPS. This is the phase GPUs are good at.
Decode generates output tokens one at a time, autoregressively. Each new token requires a full pass over the model weights, but produces only a single token’s worth of compute. The weight reuse factor is ~1: you fetch the entire weight set from memory to produce one token. Decode is therefore memory-bandwidth-bound — it is limited by how fast weights can be streamed from memory to the compute units, not by FLOPs. Adding more FLOPs does nothing; adding (or avoiding the need for) memory bandwidth is everything.
The roofline, quantified. Because reuse ≈ 1, single-stream decode throughput ≈ aggregate memory bandwidth ÷ model size. A frontier model (several hundred billion parameters, a few hundred GB at FP8) is sharded across several GPUs; divide its weights by the combined HBM bandwidth feeding them (a B200 is ~8 TB/s, HBM (High-Bandwidth Memory)) and a single conversation tops out in the tens of tokens per second, with the accelerators’ petaflops of compute mostly idle, waiting on memory. Batching many concurrent requests raises effective reuse and hardware utilisation (it amortises each weight read across requests), which is why serving economics favour large batches, but per-stream latency stays bandwidth-set. This is the precise sense in which the The Memory Wall is the binding inference cost.
Why this matters for the thesis
The memory wall binds at decode, not prefill. HBM exists to feed decode bandwidth in the datacentre; the Hbm Bottleneck is fundamentally a decode-phase problem. Three consequences:
-
Edge AI and datacentre decode are the same problem. Edge inference is bandwidth-bound because power and form factor rule out HBM; datacentre decode is bandwidth-bound because of the weight-reuse-factor-1 physics. The same architectural fix — compute-in-memory that bypasses the memory–compute boundary — addresses both. This is the core of the Memcapacitor Compute Memory Bound Ai device thesis and the Hbm Free Inference Architectures market thesis.
-
Decode’s cost share is rising. Reasoning models (long chains of thought), agentic workloads, and long-context generation all lengthen the decode phase relative to prefill. The more decode dominates, the larger the addressable market for decode-optimised silicon.
Where each player sits
Caveat
Speculative decoding, parallel sampling, and batching partially recover arithmetic intensity in decode by processing multiple candidate tokens or requests together. None of these eliminate the underlying weight-streaming cost, but they soften how memory-bound decode is in practice — a watch item for whether the HBM-free thesis window stays open.