Edge inference is running trained neural networks on the device that captured the data rather than in a datacentre, and it has moved from lab demos to shipping silicon in a market that buys against a fixed power envelope rather than a cost per token.
Summary
Edge inference means executing the forward pass of a trained model locally: on a microcontroller, a hearing-aid SoC, a phone NPU, a Raspberry Pi, a Jetson module or an FPGA, instead of sending data to a cloud GPU. The span of workloads is enormous. At one end sits keyword spotting drawing 140 microwatts on a dedicated neural decision processor; at the other, small language models of a few billion parameters generating tokens on consumer hardware 2025 06 Slm Future Agentic Ai Nvidia. Treating these as one technology is a mistake, and most disagreement about whether edge inference “works” comes from conflating them.
There are three engineering levers. First, shrink the model: post-training INT8 quantisation, and more radically the substitution of task-specialised small models for general-purpose large ones. NVIDIA Research argues existing agentic systems could serve 40-70% of their LLM queries with specialised SLMs, and that paying 10-30x for unneeded capability is unsustainable at scale 2025 06 Slm Future Agentic Ai Nvidia. Second, specialise the silicon: sparsity-aware digital accelerators, spiking-neural-network accelerators on low-cost FPGAs, and analogue in-memory or in-materia computing that performs multiply-accumulate in the physical device rather than in logic gates. Third, re-plumb memory, which is increasingly the binding constraint: SRAM is fast but low density, DRAM must hold both static weights and a growing KV cache and so suffers bandwidth contention, and Flash is dense but sits idle during inference. Proposals such as QMC split weights across multi-level ReRAM for inliers and on-chip MRAM for precision-critical outliers to work around this. In practice most deployment work is unglamorous runtime engineering against ONNX Runtime, ExecuTorch and TensorFlow Lite with XNNPACK.
Evidence base
Open questions
- Does the NVIDIA estimate that 40-70% of agentic LLM queries can be served by specialised small models survive independent replication on production agent traces, or is it an upper bound derived from narrow benchmarks?
- Can the ReRAM/MRAM hybrid memory approach in QMC be manufactured at edge BOM prices, or does it remain a simulation-and-calibrated-model result?
- Does the H1+H2 structural argument for INT8 dispatch-invariance hold on transformer and SLM graphs, not just CNNs, and would certification bodies accept it as a determinism guarantee?
- Is there any measured path to sustained multi-hour generative inference inside a 41-42C skin-temperature design target, or does the category stay burst-only regardless of silicon efficiency?
Assessment drafted 2026-08-31 from up to 14 KB sources using the technology-scorecard framework; scores are a draft read pending review.