Theses›Datacentre & AI compute›Specialised AI compute

Specialised AI compute

Specialised silicon fleets win; the routing and orchestration layer is the toll

  • Closing
  • $338B by 2035

The call, a draftSpecialised AI chips are winning, mostly inside NVIDIA's rack: Vera Rubin shipped with the Groq LPU as a decode accelerator at GTC in Mar 2026. Independent chip vendors trail, and the window on the routing layer is late.

D1D2D3D4R1R2R3C1C2C3foragainst D1D2D3D4R1R2R3C1C2C3foragainst
Fig. 1The balance. Drivers and catalysts against restraints, each block sized by its weight; catalysts hang dashed because they have not happened yet, sized by their odds. It tips toward the drivers.
  1. D1NVIDIA shipped the Groq 3 LPU as a decode accelerator inside the Vera Rubin rack, first live at Microsoft Azure, with no CUDA changes needed.
  2. D2AWS paired Trainium 3 for prefill with Cerebras CS-3 for decode on Bedrock from H2 2026.
  3. D3Custom ASICs grow about 44.6% a year against 16.1% for merchant GPUs, with ASIC shipments set to triple the GPU rate.
  4. D4Decode is splitting again, into fast decode sold on tokens per second per user and cheap decode sold on tokens per dollar.
  5. R1A leading-edge tape-out costs $50-150M+, and models change faster than a 2-3 year chip cycle.
  6. R2LLM kernel writing collapses on unfamiliar silicon: a model scoring 92% on CUDA tasks scored 14% on Ascend C.
  7. R3Today's routers dispatch by model and provider, and none places arbitrary workloads onto non-GPU silicon in production.
  8. C1A hyperscaler runs prefill- or decode-specialised inference silicon in production.31 Dec 2027
  9. C2A production router places arbitrary workloads onto non-GPU silicon at volume, and merchant GPU inference share falls as a result.31 Dec 2027
  10. C3LLM kernel synthesis reaches expert performance on low-corpus, non-CUDA silicon.31 Dec 2028
Rests onthe beliefs underneath
Supported by · confidence
What would prove itdated tests, open and resolved
20272028202920302031today123456789predictionswatch 20272028202920302031today123456789predictionswatch
Fig. 2Each ring is a dated test: dashed while open, struck through when overdue, ticked or crossed once resolved.
  1. 1Merchant GPU share of inference spend and whether any 2023-founded chip vendor nears $10B.Mar 2027 · open
  2. 2By end-2027 at least one hyperscaler runs a phase-split inference accelerator in production deployment, not only in research.Dec 2027 · open
  3. 3Hardware-routing across NON-SUBSTITUTABLE silicon does not arrive: through GTC 2027 no production routing layer dispatches ARBITRARY workloads onto a categorically different, non-GPU substrate as an abstracted buyer choice, at production volume, with the merchant GPU losing inference share as a result.Dec 2027 · open
  4. 4A hyperscaler running phase-split inference silicon in production.Dec 2027 · open
  5. 5A production router placing arbitrary workloads on non-GPU silicon by GTC 2027.Dec 2027 · open
  6. 6The founding why-now does NOT clear: LLM kernel SYNTHESIS (not translation) does not reach expert-level performance on novel, low-public-corpus silicon by end-2028, so cheap recompilation alone does not materially shift inference share to non-NVIDIA specialised silicon.Dec 2028 · open
  7. 7LLM kernel synthesis results on unfamiliar hardware.Dec 2028 · open
  8. 8By end-2030, general-purpose merchant data-centre GPUs serve <50% of datacentre AI INFERENCE compute spend; the balance goes to workload-specialised silicon fronted by routing/abstraction software.Dec 2030 · open
  9. 9Through 2030 no net-new (founded 2023+) standalone merchant inference-ASIC vendor reaches a durable >$10B standalone outcome; the pattern stays absorption-into-incumbent (Groq->NVIDIA) or hyperscaler-captive (TPU/Trainium/MTIA/Maia).Dec 2030 · open
Expressionslisted, private, technologies
Private · 6 private companies, 3 of them in the US; none in continental Europe.

Shinso · Gimlet Labs · AlphaTango · Callosum · Groq · FuriosaAI

4 more private companies tracked, not shown

Technologies
What changedthe record, newest first
  • 3 Aug 2026The specialisation story is fractal: having split training from prefill from decode, decode is now splitting again into a fast product and a cheap product. The honest counter is that one architecture, compute-in-memory, might not have to choose at all.
  • 1 Aug 2026A heterogeneous-compute orchestrator I hold is about to announce one of Europe's largest-ever seed rounds. The tell in the product line-up: the cheap tier is honest multi-cloud spot arbitrage anyone can copy, and the moat claim rests on learned router models choosing the chip per task — the proprietary placement layer everyone in this category promises is still, industry-wide, unshipped.
  • 31 Jul 2026The memory hierarchy has been one ladder sorted by speed since the 1960s. It is splitting into two lanes sorted by what the data is for: one for the model, one for the conversation.
5 earlier
  • 26 Jul 2026The falsifier was written a month after the event that would have resolved it wrong.
  • 23 Jul 2026A Gemini-specific ASIC is the specialisation ladder's next rung; the merchant-ASIC startups get squeezed and the durable value is the orchestration/placement layer over the chip zoo.
  • 22 Jul 2026Passed on a heterogeneous-compute routing layer this week where the founder, unprompted, made the bear case for me: the near-term product is really a classical CPU/GPU cost optimiser, the exotic-silicon routing is mostly modelled rather than running, and his own exit thesis is that neutral layers of this kind get bought by the chip vendors. That last point is the whole question, said out loud: if the routing layer is durable it is a toll, and if it is not it is an acqui-hire, and even a smart builder in the seat is not sure which.
  • 20 Jul 2026Passed on a company this week partly because it pitched a general-purpose architecture meant to replace the standard computing model wholesale. That framing now reads as a positioning error rather than ambition, which is itself a marker of how far the market has moved: buyers and investors have internalised that chips are narrowing to specific workloads. Worth noting the limit of that observation though. It says specialisation is the default direction, it says nothing about where the specialising stops, or who ends up collecting the money.
  • 14 Jun 2026Specialised AI chips are clearly winning on efficiency, but the brake that's falling (the labour cost of porting software) was never the one holding generality in place. The real brakes are the capital to tape out a chip and the fact that AI models change faster than chips can be designed. So the likeliest end-state isn't a hundred chip startups beating NVIDIA; it's NVIDIA, the foundries and the interconnect owners selling you the specialised rack, with the genuinely new value pooling in the software layer that routes work across mixed silicon.
Reading
Show the reading