Edge AI

Physics / mechanism

Inference of ML models at or near data sources—sensors, endpoints, microcontrollers—rather than routing to cloud. Works by quantizing trained models (INT8, INT4, binary) and mapping them onto hardware optimized for MAC operations: NPUs, DSPs, or purpose-built inference ASICs. Key parameters: TOPS/W (efficiency), latency (ms), SRAM footprint (often <1 MB), and BOM cost. State of the art sits around 10–50 TOPS/W for edge NPUs (Arm Ethos-U85, Syntiant NDP120, Hailo-8). Sparse and in-memory computing architectures are pushing efficiency toward 100+ TOPS/W at 7–12nm nodes. Deployment constrained by model compression fidelity loss and memory bandwidth walls.

Competitive landscape

Cloud inference (AWS Inferentia, NVIDIA H100 clusters) competes on model capability and developer tooling but loses on latency, privacy, and connectivity independence. On-device competition splits by tier: MCU-class (Ambiq, Nordic + TinyML), mobile-class (Apple Neural Engine, Qualcomm Hexagon), and embedded vision (Hailo, Axelera, Kneron). Adjacent techniques include neuromorphic (Intel Loihi, BrainChip) and analog in-memory compute (Mythic, Weebit). Photonic inference is an emerging adjacency with orders-of-magnitude efficiency potential but TRL 3–4.

Where value accrues (vehicle-agnostic)

The largest, most durable value in edge AI sits with the platform owners who bundle the NPU into an SoC they already ship (Qualcomm, Apple, MediaTek, Arm via IP, NXP/ST/TI in industrial+auto) and the foundries that supply the low-power process + embedded NVM. Standalone edge-inference silicon is a graveyard (see Low Power Edge Compute): the honestly-capturable wedge is the sub-watt, no-HBM, memory-device niche merchant NPUs cannot serve, sold as a component.

Companies using

Connected ideas

Sources

Frontier (open questions)

• To be added.