The growing gap between how fast processors can compute and how fast memory can feed them. Coined by Wulf and McKee (1995): if compute throughput grows faster than memory bandwidth and latency, then sooner or later every workload becomes memory-bound, and adding more compute stops helping.
By the numbers (the divergence)
- 20-year divergence (Gholami et al., “AI and Memory Wall”, IEEE Micro 2024 / Hot Chips 2023, 2024 Gholami Ai And Memory Wall): peak hardware compute grew ~60,000x, DRAM bandwidth ~100x, interconnect bandwidth ~30x. Underlying rates: FLOPS ~3.0x / 2yr, DRAM bandwidth ~1.6x, interconnect ~1.4x. Compute compounds ~2x faster than the memory feeding it, so the memory-bound share of work only grows.
- Energy (Horowitz, ISSCC 2014): a floating-point op costs on the order of a picojoule (FP range ~0.4-3.7 pJ); an off-chip DRAM fetch costs ~1.3-2.6 nJ, i.e. hundreds to ~1,000x more energy to fetch a value than to compute with it. Data movement, not arithmetic, sets the energy bill.
- Bandwidth today: an HBM3E stack delivers ~1.2 TB/s on a 1,024-bit bus; HBM4 doubles the bus to 2,048-bit for >2 TB/s/stack (see HBM (High-Bandwidth Memory)). At the accelerator level: H100 (HBM3) 3.35 TB/s, H200 (HBM3E) 4.8 TB/s, B200 (HBM3E) ~8 TB/s, versus a DDR5 channel in the tens of GB/s.
- The decode consequence: LLM decode reads the full weight set per token (reuse ≈ 1), so single-stream throughput ≈ aggregate memory bandwidth ÷ model size, landing in the tens of tokens/s for a frontier model while the compute sits mostly idle (see Prefill vs Decode (LLM inference phases)).
Why it binds now
AI inference made the wall the dominant constraint. A modern accelerator can do far more multiply-accumulates per second than its memory can supply operands for, so on memory-bandwidth-bound workloads the expensive silicon sits idle waiting on data. The economics compound it: HBM (High-Bandwidth Memory) (stacked DRAM with through-silicon vias) is the industry’s answer to push the wall back, but it is supply-constrained and expensive (see Hbm Bottleneck and HBM / CoWoS Bottleneck).
Where it bites hardest
- LLM decode: each generated token streams the full weight set from memory, so decode is bandwidth-bound while prefill is compute-bound (see Prefill vs Decode (LLM inference phases)).
- The edge: a hard power and thermal budget caps how much memory traffic is affordable.
Architectural responses
Two families. Move memory closer to compute (Near-Memory Compute, HBM), or do the compute inside the memory so the data never moves (In-Memory Computing, Processing-in-Memory (PIM), Charge-Domain Compute). Most of the post-Moore accelerator landscape is a response to this one wall.