The memory wall is the widening gap between how fast processors can compute and how fast memory can feed them: peak hardware compute grew roughly 60,000x over two decades while DRAM bandwidth grew about 100x, which is why modern AI inference is limited by bytes moved rather than arithmetic 2024 Gholami Ai And Memory Wall.
Summary
What it is. “The memory wall” names a structural imbalance in computer architecture: arithmetic throughput has compounded far faster than the bandwidth and capacity of the memory that supplies operands to it. The canonical quantification comes from Gholami et al. (blog 2021, Hot Chips 2023, IEEE Micro 2024): over twenty years peak hardware compute rose ~60,000x, DRAM bandwidth ~100x, and interconnect bandwidth ~30x. Underneath those totals, peak server FLOPS scale at ~3.0x every two years, DRAM bandwidth at ~1.6x and interconnect bandwidth at ~1.4x 2024 Gholami Ai And Memory Wall. Because the rates differ, the share of workloads that are memory-bound rather than compute-bound grows monotonically with each hardware generation, regardless of any single design decision.
How it bites. Two mechanisms. First, throughput: if a kernel has low arithmetic intensity (few operations per byte fetched), the arithmetic units idle while weights arrive. In LLM serving the decode phase is the bandwidth-bound one, and one commentary describes GPU arithmetic units as spending most of their time stalled waiting for weights from HBM ref. Second, energy: an arithmetic operation costs sub-picojoule while an off-chip DRAM fetch costs roughly 1.3 to 2.6 nJ, so data movement, not computation, dominates the energy bill 2024 Gholami Ai And Memory Wall. Reasoning-style models make both worse, because long output sequences, low arithmetic intensity and tight latency budgets all push demand for bandwidth up and system utilisation down.
The parameters that decide it. For any given workload the wall is set by bytes fetched per useful operation (arithmetic intensity), by achievable bandwidth per package, by memory capacity per package (which decides how much of a model or KV cache stays local), and by the fraction of peak bandwidth a design can actually realise. That last parameter is where the commercial fight is: Positron’s second-generation Asimov silicon claims over 90% memory bandwidth utilisation on real transformer workloads against under 30% claimed for GPUs, using commodity LPDDR5x rather than HBM, at 864GB to 2.3TB capacity per chip and 2.76 TB/s realisable bandwidth ref. Groq attacks the same parameter from the other end, placing 230MB of on-die SRAM at 80 TB/s, roughly an order of magnitude above HBM bandwidth, at the cost of tiny capacity ref.
Evidence base
- Over twenty years peak hardware compute grew ~60,000x while DRAM bandwidth grew ~100x and interconnect bandwidth ~30x; per-generation rates are 3.0x, 1.6x and 1.4x every two years 2024 Gholami Ai And Memory Wall (2024-03-21).
- Arithmetic costs sub-picojoule per operation while an off-chip DRAM fetch costs ~1.3-2.6 nJ, so data movement dominates energy (Horowitz ISSCC 2014, via 2024 Gholami Ai And Memory Wall, 2024-03-21).
- Reasoning LLM workloads combine long output sequences, low arithmetic intensity and tight latency constraints, driving higher bandwidth demand and falling system utilisation on GPUs (2026-06-05, paper published 2026-02-20).
- Digital neuromorphic processors that distribute memory on chip find SRAM and STT-MRAM become significant consumers of area and energy, creating a new memory wall (2026-04-09).
- Real processing-in-memory hardware remains scarce, so simulation is the primary way to explore the PIM design space (2026-07-28).
- Positron’s Asimov claims LPDDR5x rather than HBM, 864GB-2.3TB per chip, 2.76 TB/s realisable bandwidth, over 90% memory bandwidth utilisation versus under 30% for GPUs, and 5x tokens per dollar and per watt versus NVIDIA Rubin; tape-out October 2026, production early 2027 ref (2026-07-31).
- Groq places 230MB of on-die SRAM at 80 TB/s, roughly one order of magnitude above HBM bandwidth, in an arrangement with Nvidia described as a $20 billion deal ref (2026-01-22).
Open questions
Assessment drafted 2026-08-31 from up to 10 KB sources using the technology-scorecard framework; scores are a draft read pending review.