Inference silicon is bought against two different customer KPIs depending on where it sits: cost per million tokens (spoken as tokens per watt per dollar under a power cap) in the datacentre, and energy per inference inside a fixed power and thermal envelope at the edge, and confusing the two is the most common analytical error in the sector.
Summary
“Inference economics” is shorthand for the fact that buyers of inference compute do not evaluate chips on peak throughput. They evaluate them on a delivered unit-economic figure, and there are two such figures, not one, split cleanly by segment.
In the datacentre the operative KPI is cost per million tokens. Nvidia has published the identity explicitly: $/Mtok = [$/GPU-hr ÷ (tok/s × 3600)] × 1M, under the headline “Cost per Token Is the Only Metric That Matters” (15 April 2026). Because the binding constraint is electrical rather than financial, the same KPI is usually spoken with power in the denominator: Satya Nadella’s “tokens per watt per dollar” (January 2025), used on Microsoft’s Q2 FY26 call on 28 January 2026 to claim Maia 200 is more than 30% better on tokens per dollar; Jensen Huang’s “tokens per second per watt”, and his GTC 2026 formulation on 16 March 2026 that Revenue = Tokens/Watt × Available GW. The levers inside the identity are therefore box cost per hour, achieved (not peak) tokens per second, and the power cap that decides how many boxes can be energised at all. Achieved throughput is dominated by memory: reasoning workloads with long output sequences have low arithmetic intensity and tight latency constraints, so utilisation collapses on compute-optimised GPUs, and challengers pitch straight at the KPI terms, as Positron does with claims of 5x tokens per dollar and 5x tokens per watt versus Nvidia Rubin on the back of over 90% memory-bandwidth utilisation against under 30% claimed for GPUs ref.
At the edge the KPI is different in kind. Silicon is bought against a fixed power and thermal envelope, and the metric is energy per unit of work: cost per token is absent from procurement, which runs on per-unit BOM (typically $15-45) and 12-18 month RFQ cycles. The envelopes are hard physical facts rather than budget choices: smart glasses at hundreds of milliwatts sustained and 1-2W peak with 150-500mAh batteries and a skin-temperature cap around 48C; hearing aids at 1-3mW steady state; Meta’s Ray-Ban Live AI rated at roughly 30 minutes and thermally gated. Vendors answer in energy or efficiency, not throughput: Syntiant quotes 140µW keyword spotting, ST leads the STM32N6 with 3 TOPS/W rather than its 600 GOPS, and Qualcomm argues publicly that TOPS is insufficient. Third-party energy anchors exist but are thin: Syntiant’s MLPerf Tiny submissions measure 31.5-43.8 µJ per keyword-spotting inference and 71.7-97.2 µJ for visual wake, and are the only verified edge energy figures in the corpus.
The parameters that decide whether the framing is useful are: the gap between claimed and measured delivered economics; whether price and latency move together (they do not); and the rate at which the incumbent baseline improves, since any challenger’s advantage must exceed it. Measured provider data for Llama 3.3 70B shows Groq fastest at 311 tok/s and $0.61 per million blended against mainstream GPU providers at 71-163 tok/s and $0.71-0.88, but a cheapest-GPU floor of $0.12-0.17 per million at FP8, putting Groq 3.5-5x above the price floor: the demonstrated advantage is speed, not cost. And the baseline moves fast: 8x B200 delivers 3.0x 8x H200 on MLPerf Llama-2-70B Server (about 2x iso-precision), software alone lifted H100 by up to 1.5x in a year, and the defensible floor for decline in GPU delivered cost per token is put at roughly 2.5-3x per year.
Evidence base
- 15 April 2026: Nvidia publishes “Cost per Token Is the Only Metric That Matters” with the identity $/Mtok = [$/GPU-hr ÷ (tok/s × 3600)] × 1M; Huang’s GTC 2026 keynote on 16 March 2026 frames Revenue = Tokens/Watt × Available GW with token tiers of $1/M to $150/M.
- 28 January 2026: Microsoft’s Q2 FY26 call uses tokens per dollar per watt and claims Maia 200 is over 30% better on tokens per dollar; Alphabet’s 4 February 2026 call reports Gemini serving unit costs down 78% over 2025.
- 10 July 2026: measured Llama 3.3 70B provider data shows Groq fastest at 311 tok/s and $0.61/M blended versus mainstream GPU at 71-163 tok/s and $0.71-0.88/M, with a cheapest-GPU FP8 floor of $0.12-0.17/M; the advantage is speed (2-4.4x), not price. Defensible GPU delivered $/token decline floor: 2.5-3x per year.
- 10 July 2026: edge silicon is bought against a fixed power/thermal envelope with cost per token absent from procurement (per-unit BOM $15-45, 12-18 month RFQs); anchors include 1-3mW hearing-aid budgets, Syntiant’s 140µW keyword spotting and ST leading the STM32N6 with 3 TOPS/W.
- 31 July 2026: Positron markets Asimov on 5x tokens per dollar and 5x tokens per watt versus Nvidia Rubin, based on over 90% memory-bandwidth utilisation against under 30% claimed for GPUs, with LPDDR5x instead of HBM; tape-out reported October 2026, production early 2027 ref.
- 21 July 2026: Minack argues the contestability case has firmed up and that AI provision is a competitive, mediocre-return industry, partly due to cheap Chinese competition; Chinese open-weight models are reported at coding parity with 5-30x lower cost per token (28 June 2026) ref.
Open questions
- Does the speed premium survive? One GPU provider already hit 295 tok/s at $0.20 per million against Groq’s 311 tok/s at $0.61; if FP8 GPU serving closes the latency gap, the speed-versus-price split collapses into a single price KPI.
- Will an audited benchmark for delivered cost per token emerge beyond Artificial Analysis’s 72-hour P50 panel, so that vendor tokens-per-dollar claims such as Maia 200’s 30% or Positron’s 5x can be independently checked?
- Does the datacentre KPI split permanently into per-phase figures (prefill $/Mtok and decode $/Mtok) as disaggregation and phase-specific silicon reach volume procurement, and does that change which chip wins?
- Does tokens/sec/W become an actual line item in edge RFQs at the on-device-LLM tier, or does edge procurement stay on BOM plus envelope plus energy per inference indefinitely?
Assessment drafted 2026-08-31 from up to 16 KB sources using the technology-scorecard framework; scores are a draft read pending review.