The robot “brain”: a single neural network that ingests camera images + a natural-language instruction and emits robot actions, end-to-end. Lineage is VLM -> VLA: take a web-pretrained vision-language model (inheriting semantic grounding), bolt on an action head trained on robot trajectories.
The key architectural axis
- Single-network end-to-end (RT-2, OpenVLA): one transformer emits action tokens. Simpler; bottlenecked by slow VLM inference for high-frequency control.
- Dual-system (System 2 / System 1): a slow VLM planner (~7-10 Hz) + a fast visuomotor policy (~200 Hz, usually a diffusion or flow-matching transformer). Now the consensus design for humanoid/whole-body control. Used by Figure Helix, NVIDIA GR00T, Gemini Robotics 1.5.
Leading models (mid-2026)
- Physical Intelligence π0 / π0.5 — π0 weights open; π0.5 (CoRL 2025) is the most credible open-world generalisation result (cleans unseen homes, noisily). Flow-matching action expert.
- Google DeepMind Gemini Robotics 1.5 / -ER 1.5 — dual model; the ER reasoning model is open to devs; “Motion Transfer” across embodiments.
- NVIDIA GR00T N1->N1.7 — the open humanoid foundation model; the point is data-efficiency (GR00T-Dreams synthetic data “36 hours vs 3 months”). Adopters: Boston Dynamics, Agility, XPENG.
- Figure Helix (closed), Skild Brain (closed, “omni-bodied”, least verified), 1X (paired with its world model).
Open weights: π0, GR00T, OpenVLA, Gemini-ER (reasoning only). Closed: Helix, Skild, π0.5 (paper not weights). NVIDIA is deliberately the open-infra player (to sell GPUs).
What works vs fails
Works: language grounding, short-horizon pick-place/sort in semi-structured settings, modest transfer to new objects, π0.5-class noisy open-world generalisation. Fails: cross-embodiment action transfer, fine spatial/temporal reasoning, robustness to noise/lighting/distractors/paraphrase, long-horizon tasks, and anything contact-rich (Dexterous Manipulation & Robot Hands). New “action hallucination” failure literature is emerging.
Data sources
Teleop demonstrations (dominant, expensive, doesn’t scale, the Teleoperation Bridge argument), pooled cross-robot datasets (Open X-Embodiment ~970k trajectories; DROID ~76k), web/VLM pretraining for semantics, egocentric human video (the emerging cheap-data bet), and synthetic/world-model data.
Why it matters here
This is the layer Robot Foundation Models bets on (value capture: commoditising from below via open releases). Generalisation is improving fast (~80% confidence) but maybe not as fast as valuations imply (~60%). The honest read: no GPT-3 moment yet, which is why Robot Autonomy Destination stays contested.
Connections
Physical AI (robotics cluster hub) · Robot Foundation Models · World Models (for robotics & autonomy) · Sim-to-Real, Robot Simulation & Synthetic Data · Robot Autonomy Destination · Teleoperation Bridge · runs on Nvidia Jetson