Physics / mechanism
Inference is the execution phase of a trained model: converting inputs into outputs (tokens) rather than updating weights. Its hardware demands differ from training in a way that the memory hierarchy makes explicit. The decode stage of autoregressive generation produces one token at a time, so each step must stream model weights and cached attention state past the arithmetic units with little opportunity to amortise those reads across a large batch. Throughput is therefore governed by memory read bandwidth and by how much state fits close to the accelerator, not primarily by peak floating-point rate.
That framing is what drives interest in near-package memory alternatives. High Bandwidth Flash (HBF) stacks NAND dies vertically, connects them with through-silicon vias, and places the stack next to the GPU on the package interposer, in the same topology used for HBM. The claimed result is read bandwidth comparable to an HBM4 stack with roughly ten times the capacity, because the medium is NAND rather than DRAM ref.
The key parameters for inference are consequently: read bandwidth per stack, capacity per stack, and the asymmetry of the storage medium. Flash carries tradeoffs relative to DRAM, and the argument advanced for HBF is that those tradeoffs are tolerable specifically for inference decode workloads, which are read-dominated ref. Capacity matters because larger resident weight sets and longer key-value caches otherwise force model sharding across more accelerators or eviction to slower tiers.
Competitive landscape
The relevant comparison for inference memory is HBM versus HBF. HBM (DRAM) offers symmetric read and write behaviour and is the incumbent for both training and inference; HBF (NAND) targets the read-heavy decode path, trading DRAM’s write characteristics for roughly an order of magnitude more capacity at similar read bandwidth ref. The sources do not support a comparison across accelerator architectures, quantisation approaches, or serving-software strategies, all of which also bear on inference cost.
Evidence base
- HBF is described as a stack of NAND dies built the way an HBM stack is built, with TSVs, sitting next to the GPU on the package interposer ref.
- HBF is claimed to have the same read bandwidth as an HBM4 stack with roughly 10x the capacity, as of 7 July 2026 ref.
- First samples of HBF memory were expected from Sandisk in the second half of 2026 ref.
- Samples of the first AI inference devices built with HBF were expected in early 2027 ref.
- Flash tradeoffs are argued to be acceptable specifically for inference decode workloads rather than for inference generally ref.
Frontier (open questions)
- Do shipped HBF stacks match HBM4 read bandwidth in measured decode workloads, or only in peak specification?
- What write endurance and write-latency limits does HBF impose on key-value cache updates, and do they force a hybrid HBM plus HBF memory tier?
- Did Sandisk HBF samples ship in H2 2026 and did inference-device samples follow in early 2027 as projected?
- What is the cost per token, not cost per GB, of HBF-based inference relative to an all-HBM baseline at equal model size?
Synthesised 2026-08-31 from 2 KB sources by the resynth pipeline; citations are KB source slugs.