The “CUDA moat” is NVIDIA’s accumulated software stack and code corpus, which makes its GPUs the default compute target; its durability now hinges on whether LLM-based kernel synthesis can port performance to low-corpus silicon, and as of mid-2026 the published evidence says it cannot.
Summary
CUDA is NVIDIA’s proprietary programming model for its GPUs, plus the libraries, runtimes and domain platforms layered on top of it. The “moat” is not the language itself but three compounding effects. First, hand-tuned kernels and libraries that competitors must re-implement to match performance. Second, ecosystem default status: papers across unrelated fields ship CUDA-first implementations, from diagonal-sparse Hamiltonian simulation kernels in C++/CUDA to GPU fully homomorphic encryption via CUDA-Python bindings, and NVIDIA has extended the brand into adjacent domains with CUDA-Q for quantum simulation, used in protein-folding VQE, 40-qubit chemistry pipelines and quantum-network control studies. Third, and newest, corpus asymmetry: because almost all public GPU kernel code is CUDA, frontier language models are far better at writing CUDA than at writing kernels for anything else.
The third effect is what turned the moat from a commercial observation into a testable technical question. The bull case for challenger silicon (AMD, Ascend, NPUs, ASICs) is that porting cost is collapsing because agentic models can synthesise kernels and compilers on demand. The bear case is that models only look good where a large public corpus already exists, so the moat is self-reinforcing precisely in the dimension supposed to dissolve it.
The parameters that decide the outcome are: (a) how much of delivered inference performance actually rides on kernel quality rather than silicon, (b) whether LLM kernel synthesis generalises from CUDA to low-corpus instruction sets and programming models, (c) whether portable abstractions such as OpenMP target offloading, HIP or ONNX Runtime can reach hand-tuned CUDA performance without per-vendor work, and (d) whether hardware evolution (chiplets, confidential computing) invalidates CUDA’s own abstractions faster than the corpus accumulates.
On (a) the evidence is that kernel and runtime quality is worth large multiples of a percentage point: a task-based megakernel runtime on AMD MI350 cut Qwen3-8B decode latency 1.3-1.5x below vLLM at small batch and lifted L2 hit rate from 12% to 54% at batch 32, cutting HBM traffic by up to 37%. Software, not silicon, was the variable.
Viability (4/5)
The clearest quantification available: GPT-5.2 scores 92% on CUDA level-1 kernel tasks but 14% on Ascend C, and on level-2 tasks for emerging hardware models fail entirely ref. Independently, KernelBench and TritonBench results show state-of-the-art LLMs struggle to produce even correct GPU kernels and perform substantially worse than expert-crafted ones ref. The dated arc of automated kernel work confirms both the progress and its ceiling: parallel-search generation reached 4.8x over PyTorch on layer-norm and about 180% on Conv2D, but only 9% of Flash Attention and 52% of FP16 matmul performance, and multi-turn RL (Kevin-32B) moved correctness from 56% to 82% and mean speed from 0.53x to 1.10x baseline, which is parity on easy targets rather than expert performance on hard ones ref.
So the mechanism holds today: the cheap-recompilation escape route from CUDA does not yet exist for hardware without a large public kernel corpus. The score is 4 rather than 5 because the trend line is steeply upward from a low base, targeted attacks on exactly this weakness have begun (KForge synthesising for Apple Metal and OpenCL, explicitly addressing scarce training data on low-corpus hardware ref), and the horizon under test runs to end-2028.
TLDR: The moat is real and measurable: model kernel competence collapses off CUDA, from 92% to 14% on one benchmark.
Drivers (4/5)
Supply: NVIDIA is extending the stack faster than rivals close it, and the sampled literature shows CUDA-branded platforms colonising domains with no prior GPU tradition, including quantum simulation and quantum networking. Each such paper adds public CUDA code, feeding the corpus effect. Cryptography researchers note plainly that most existing GPU implementations rely on CUDA-specific optimisations.
Demand: there is visible, funded pull in the other direction. Portable single-source implementations targeting both NVIDIA and AMD via OpenMP offload are being published as a deliberate alternative to CUDA-specific code; performance work is being done natively on AMD Instinct silicon; edge stacks already dispatch across PyTorch CUDA and ONNX Runtime CPU backends per operator; and CUDA-kernel optimisation is now a standard long-horizon task in agentic-AI evaluation suites, which means capability here is being explicitly trained and measured. Both forces are strong, which is why the outcome is contested rather than settled.
TLDR: Supply side keeps widening CUDA’s surface area; demand side is now actively funding the escape attempt.
Novelty (3/5)
Judged as differentiated technology rather than incumbency, CUDA looks weaker than its market position implies. Fleet finds that CUDA and HIP alike expose a flat execution hierarchy that cannot express chiplet-level locality or synchronisation on modern multi-die GPUs, causing redundant memory traffic in memory-bound LLM inference, and that fixing this at the runtime level yields 1.3-1.5x decode latency improvements on AMD hardware. Confidential computing exposes another seam: NVIDIA B300 BF16 matmul runs at 0.998x of non-confidential performance, yet LLM serving loses 13-27% of throughput because secure copies do not gain CUDA-stream concurrency and small crossings pay a fixed toll, with small alloc-and-copy operations 44x slower. CUDA’s model, not its silicon, is the failure point in both cases.
Where CUDA is genuinely and quantifiably better than the alternative is in the corpus: roughly a 6.5x gap in model kernel-writing success versus a low-corpus target ref. That is a real and currently large advantage, but it is an asset-accumulation advantage, replicable in principle by anyone who generates enough kernel code and training data. Hence 3, not higher.
TLDR: CUDA’s advantage is accumulated corpus and libraries, not superior abstractions, and its abstractions are visibly ageing.
Diffusion (5/5)
Diffusion of the incumbent is close to total in this sample. CUDA or CUDA-branded platforms appear in quantum Monte Carlo benchmarking, Hamiltonian simulation kernels, homomorphic encryption, sparse LLM inference kernels distinguishing CUDA-core and Tensor-Core paths, and edge inference on Jetson.
The barriers to displacing it are structural rather than a single technical gate. Groups dissatisfied with the platform tend to patch rather than exit: Graph-VQE responds to CUDA-Q lacking Qiskit support and restricting parallelisation by building a custom Qiskit-CUDA-Q integration layer instead of moving off the platform. Escaping requires simultaneously replacing kernels, libraries, runtimes and inference servers, and portable routes typically appear as explicitly research-grade exceptions to the CUDA norm. The sources do not quantify market share; the score reflects observed technical default status in the literature only.
TLDR: CUDA is the default target across AI, quantum simulation, FHE and cryptography in the sampled literature; displacement requires work at every layer at once.
Impact (4/5)
The sources establish that the contested layer is economically heavy even without market data. Runtime and kernel design alone produce 1.3-1.5x decode latency changes and up to 37% less HBM traffic on fixed silicon; a mismatch between the CUDA execution model and a new deployment mode costs 13-27% of serving throughput, of which a scheduling flag recovers 57% and a worker-thread drain up to 92%; and backend dispatch decisions across 9,584 operator instances measurably change edge latency. Performance of this magnitude sitting in software rather than silicon is what allows an incumbent software stack to determine which chips get bought.
The honest limit: none of the supplied sources contains revenue, market share or pricing data, so the size of the economic prize is inferred from the technical mechanism rather than measured. A resolution either way propagates directly into whether specialised non-NVIDIA inference silicon can take share, which is the framing the evidence collections themselves use ref.
TLDR: The software layer decides double-digit-percent to 1.5x differences in delivered inference performance, so whoever owns it captures the routing decision.
Timing Now (0-2yr)
This is not a future technology; it is a present constraint that is being actively tested. The evidence arc runs continuously from June 2025 through mid-2026, with each step improving from a low base ref, and the most damaging result for the dissolution thesis is dated March 2026 ref.
The question is framed by its own trackers as resolving by end-2028: whether KernelBench-style results show expert-performance LLM kernel synthesis on unfamiliar hardware with little public kernel code, with roughly 70% weight on the bear case (the moat holds) ref. Anyone making silicon or capacity commitments in the next two years is implicitly taking a side.
TLDR: The moat is operative today and the decisive benchmark evidence is arriving on a 2025-2028 clock.
Overrated or underrated? Underrated
The consensus narrative through 2025-2026 was that frontier models would make porting trivial and the CUDA moat would erode quickly. The measured results do not support that. Models are excellent at the thing there is already a corpus for and near-useless at the thing there is not: 92% versus 14% on comparable task tiers ref, with expert-level kernels still out of reach even on CUDA itself ref and hard targets like Flash Attention reached at 9% of expert performance ref. The moat’s durability is therefore underrated relative to the porting-is-solved thesis.
Two qualifications keep this from being a strong call. The moat is not built on better abstractions: CUDA’s flat hierarchy is already a documented liability on chiplet GPUs and under confidential computing, and third parties are recovering large performance wins on AMD silicon by working around it. And the specific weakness, corpus scarcity, is now the explicit target of research such as KForge and cold-start drafting with continual refinement ref ref. If a synthetic-data flywheel closes the gap, this verdict flips within the stated horizon.
Prediction
By 31 December 2027, no publicly reported KernelBench-style evaluation will show a frontier model reaching within 20 percentage points of its CUDA level-1 correctness rate on a low-public-corpus non-NVIDIA target such as Ascend C.
Evidence base
- 2025-11-20: KernelBench and TritonBench show state-of-the-art LLMs struggle to generate correct GPU kernels and perform substantially worse than expert-crafted kernels ref.
- 2026-03-15: GPT-5.2 scores 92% on CUDA level-1 kernel tasks but 14% on Ascend C, and fails entirely on level-2 tasks for emerging hardware ref.
- 2026-06-22: Dated arc of automated kernel synthesis: layer-norm 4.8x over PyTorch and Conv2D to about 180% (Jun 2025), but Flash Attention 9% and FP16 matmul 52%; Kevin-32B moved correctness 56% to 82% and mean speed 0.53x to 1.10x (Jul 2025); KForge targets non-NVIDIA accelerators (Nov 2025) ref.
- 2026-06-05: CUDA and HIP both expose a flat execution hierarchy that cannot express chiplet locality; a task-based runtime on AMD MI350 with Qwen3-8B cut decode latency 1.3-1.5x versus vLLM and raised L2 hit rate from 12% to 54% at batch 32, cutting HBM traffic up to 37%.
- 2026-06-24: On NVIDIA B300 confidential computing, BF16 matmul runs at 0.998x of non-confidential performance, yet LLM serving loses 13-27% of throughput because secure copies do not gain CUDA-stream concurrency; small alloc-and-copy operations are 44x slower.
- 2026-07-29: A portable LWE KEM implementation using OpenMP target offloading runs one source base on both NVIDIA and AMD accelerators, in contrast to most existing GPU implementations which rely on CUDA-specific optimisations.
- 2026-07-07: Graph-VQE responds to CUDA-Q lacking Qiskit support and restricting parallelisation by building a custom Qiskit-CUDA-Q integration layer rather than leaving the platform.
Open questions
- Do cold-start drafting plus continual refinement approaches close the 92%-to-14% CUDA/Ascend C gap, and how much target-specific human-authored kernel data does that still require?
- Can KForge-style synthesis for diverse accelerators (Metal, OpenCL) reach expert performance on hard kernels such as Flash Attention and FP16 matmul, where automated CUDA generation still hits only 9% and 52% of expert performance?
- If CUDA’s flat execution hierarchy is already inadequate for chiplet GPUs and confidential computing, does the corpus advantage transfer to the next programming model or reset with it?
- Do portable single-source routes such as OpenMP target offloading close enough of the performance gap on NVIDIA and AMD to remove the need for per-vendor hand-tuned kernels in production inference?
Assessment drafted 2026-08-31 from up to 18 KB sources using the technology-scorecard framework; scores are a draft read pending review.