Processing-in-Memory puts arithmetic and logic inside or immediately beside the memory arrays so that memory-bound workloads such as LLM decoding, graph traversal and vector search stop paying for data movement; the physics is sound and simulated speedups are large, but real PIM silicon remains scarce and almost every published number is a simulator number.
Summary
The parameters that decide whether a PIM design wins are consistent across the papers. First, kernel arithmetic intensity: LLM decoding is dominated by memory-bound GEMV or flat GEMM and is therefore the canonical target. Second, the area cost of putting processing elements inside DRAM, which reduces usable memory capacity, and the weakness of logic that can be built in a DRAM process at all. Third, communication inside the memory: prior transformer PIM designs suffered costly inter-bank traffic and could not scale past bank capacity, and PIM units have small local memory, expensive inter-unit communication and relatively weak compute. Fourth, for non-volatile PIM, device reliability and thermal noise, which perturb stored weights and degrade inference accuracy. Fifth, the host interface: how a CPU or GPU shares address space and bus with the PIM device without bank conflicts and bus congestion eating the gain.
The response to those constraints in 2025-2026 work is architectural rather than device-level: decouple logic and memory into chiplets built in different process nodes and connect them over an interposer, then attach the module over CXL; mix substrates so each kernel lands on the device that suits it, for example SRAM-PIM for latency-critical attention and HBM-PIM for weight-heavy GEMV, or a matrix tile plus a traversal tile in one 2.5D package; and add a scheduler that decides at runtime what goes to PIM and what stays on the host.
A separate strand treats content-addressable memory as the PIM primitive rather than arithmetic. Reformulating Insert, Search, Update and Delete as in-situ content-addressable operations turns indexing itself into a memory operation, and a hardware-first graph database model has been specified around a small content-addressable instruction set (CAR, CAR2, AAR) with a dually-addressable memory implemented in ordinary FPGA block RAM rather than exotic devices.
Evidence base
- July 2026: a full-system PIM simulator paper states that real PIM hardware remains scarce and simulation is therefore the primary way to explore the design space, spanning eleven memory technologies (seven DRAM standards, SRAM, three non-volatile memories).
- November 2025 (published June 2026): Sangam decouples logic and memory into chiplets in heterogeneous process nodes on an interposer and attaches the module over CXL, explicitly to escape the capacity loss and weak PEs of processing elements built inside DRAM chips.
- April 2026: GEN-Graph’s processing-using-memory matrix tile reports 42.8x speedup and 392x energy efficiency over an NVIDIA H100 on all-pairs shortest path, using a two-tile heterogeneous 2.5D chiplet.
- July 2026: dynamic operator scheduling across heterogeneous NPU plus PIM systems achieves geometric-mean speedups of only 1.20x to 2.23x over a prefill-decode disaggregation baseline, plus 1.28x from weight layout selection.
- May 2026: Sieve documents that modern MoE models’ increasingly bimodal token-to-expert distributions dramatically reduce the efficiency of state-of-the-art PIM systems for LLMs, which relied on static offload rules.
- February 2026: a review of memristor-based digital PIM (RRAM, PCM, MRAM) identifies reliability and device-level optimisation as the key open challenges for scalable, commercially viable PIM.
- May 2025: a peer-reviewed ISCAS paper implements a dually-addressable (address and content) memory in FPGA block RAM at 8x512x36 bits on a Xilinx Virtex-7, showing the content-addressable PIM primitive does not require exotic devices.
Open questions
- Do the chiplet and CXL-attached designs survive fabrication, and how do measured end-to-end numbers compare with the simulated 1.20x-2.23x system speedups reported for NPU plus PIM platforms?
- Can a PIM device designed today remain efficient across two generations of model-architecture change, given that bimodal MoE expert distributions already break existing static offload rules?
- Does non-volatile PIM reach the device reliability and thermal-noise tolerance that the 2026 review identifies as the precondition for commercial viability, or does DRAM and SRAM near-memory logic take the whole market?
- Does any of the competing scheduling and compiler frameworks consolidate into a portable programming model, or does every PIM part continue to require its own bespoke stack?
Assessment drafted 2026-08-31 from up to 18 KB sources using the technology-scorecard framework; scores are a draft read pending review.