AgentRoboCritic: Can Frontier AI Agents Control Robots?
AgentRoboCritic grades GPT-6 Astra's robot commands action by action: 57.7% of failures recovered, grasping and alignment lagged.

Contents8 sections
- How AgentRoboCritic grades frontier agents on robot control
- What frontier agents did well: recovery, grounding and zero-shot skill
- Where they fail: grasping, alignment and uneven recovery
- LLM agent vs VLA: success rate traded for speed
- Limits to keep in mind before citing these results
- How to evaluate robot agent claims after this study
- Frequently asked questions
- Sources and references
Key takeaways
- AgentRoboCritic makes the agent state a short local intent with every robot command, then has a critic judge each action against that intent and name a failure mode.
- Across 12 manipulation benchmarks, 57.7% of failed actions were verifiably recovered later in the same episode, and 49.8% of successful episodes contained at least one failed action.
- Physical execution is the bottleneck: Grasp was the top failure mode on 9 of 10 non-tactile benchmarks, and only 35.7% of Alignment failures were recovered.
- Zero-shot, GPT-6 Astra succeeded in 91% of RoboTwin 2.0 and 90% of YAMLab episodes versus 6% and 0% for pi0.5, but took 33.3 seconds per decision versus 0.12 seconds.
- It is a preprint, the same model acts as policy and main critic, and the zero-shot comparison covers two benchmarks.
Yes, frontier AI agents can drive a robot arm directly, but AgentRoboCritic shows the real question is which actions they get right. The study, posted to arXiv on November 19, 2026 by researchers at Carnegie Mellon University, the University of Illinois Urbana-Champaign and Bosch, graded GPT-6 Astra's robot commands one action at a time across 12 manipulation benchmarks (abstract). The short version: the agent picks the right object almost every time and often fixes its own mistakes, but precise physical work such as grasping and alignment is where it breaks down, and each decision is slow.
Figures below are as of November 2026 and come from the v1 paper and the authors' project page. The paper has not been peer reviewed yet, so numbers may change in later versions.
How AgentRoboCritic grades frontier agents on robot control
Most robot benchmarks record a single pass/fail bit per episode. That hides two things: how many mistakes happened inside a successful run, and which step sank a failed one. Grading a single action against the overall goal does not fix this, because many different trajectories can complete the same task. The same motion might be an incomplete reach, a missed alignment or a failed grasp depending on what it was supposed to do.
The authors' fix is a local intent. Every tool call that sends a numerical command must also include a plain-language note on what that action is meant to achieve, and the harness rejects any call without one. A critic model then sees the task goal, that note and multi-view camera frames from before and after the action.
- 1
Agent sends a command plus its intent
For example, an end-effector pose and gripper value alongside a note like 'close on the blue piece at its current height'.
- 2
Robot executes and returns new images
The loop is closed with no high-level primitives. Gripper closures are judged with the lift frames that follow, so a grasp only counts if the object stays held.
- 3
Critic labels the outcome and failure mode
Outcomes are Achieved, Failed, Partial or Unverifiable. Failures get one of seven modes: Grasp, Alignment, Interference, Effect, Incomplete motion, Choice or Unresolved, plus a short evidence note.
- 4
Humans check the critic
People labeled 400 actions from the same inputs to measure how far the critic can be trusted.
GPT-6 Astra served as the critic. On the 400 human-labeled actions (34% successful, 66% failed) it scored 79.25% outcome accuracy and 80.68% failure-mode accuracy, the best of the models tested, according to the full paper and project page. Without the local intent those numbers collapsed to 25.25% and 25.00%.
What frontier agents did well: recovery, grounding and zero-shot skill
They recover from their own mistakes
Imitation-trained robot policies suffer from covariate shift: one error pushes them outside their training data and errors compound. The project page reports that the agent behaved differently. 49.8% of successful episodes contained at least one failed action. Episodes with one or two local failures still succeeded 40.3% of the time, compared with 75.1% for failure-free episodes.
Tracing 12,163 failed actions across 1,989 graded episodes, 38.9% were verifiably recovered within 10 later actions and 57.7% by the end of the episode. A regrasp only counted once the object visibly rose with the gripper. Across benchmarks, recovery rate and success rate correlated at r = 0.81. One caution: RoboDojo alone contributed 9,923 of those failures, so the pooled figures lean heavily on one suite.
They rarely act on the wrong object
Choice failures, where the action contradicted the object, order or category in the stated intent, made up 0.2% of all failures and never more than 3.3% on any benchmark. That contrasts with specialized models known to degrade under camera shifts and rearranged scenes.
They work without task-specific training
With no fine-tuning or demonstrations, GPT-6 Astra succeeded in 91% of RoboTwin 2.0 episodes and 90% of YAMLab episodes. Under the same zero-shot setup, the VLA pi0.5 reached 6% and 0%, and InternW0-Delta succeeded in no episode on either.
Where they fail: grasping, alignment and uneven recovery

The weak spot is physical precision, not understanding. Grasp was the most frequent failure mode on 9 of the 10 non-tactile benchmarks. On vision-only, tight-tolerance insertion tasks (UniVTAC and OpenViTac), Alignment made up 75% and 86% of failures. Cluttered scenes raised Interference.
Incomplete motion
- What it means
- A commanded move or opening did not finish
- Recovered by episode end
- 68.2% (479 of 702)
Grasp
- What it means
- Object not acquired or not retained
- Recovered by episode end
- 63.7% (4,155 of 6,523)
Interference
- What it means
- Unintended collision or displacement
- Recovered by episode end
- 59.1% (1,602 of 2,711)
Alignment
- What it means
- Held object misses its target pose
- Recovered by episode end
- 35.7% (681 of 1,908)
Effect
- What it means
- Tool held but intended effect not produced
- Recovered by episode end
- 29.6% (87 of 294)
Recovery was uneven. Grasp failures were mostly fixed by regrasping, while only about one in three Alignment and Effect failures were recovered, and those rates moved by less than three percentage points after 30 more actions. More attempts do not solve a precision problem.
Three in five Grasp failures happened as the gripper closed: pinching a rim or edge, or closing beside or above the object. The authors point to fine 3D localization of the gripper relative to the object. Another 9.5% of Grasp failures were actions issued with an already empty gripper, meaning the agent sometimes misjudged its own state.
LLM agent vs VLA: success rate traded for speed

Control interface
- Frontier agent (GPT-6 Astra)
- Numerical commands as text tool calls
- VLA (pi0.5)
- Action head outputs continuous actions
RoboTwin 2.0 zero-shot success
- Frontier agent (GPT-6 Astra)
- 91%
- VLA (pi0.5)
- 6%
YAMLab zero-shot success
- Frontier agent (GPT-6 Astra)
- 90%
- VLA (pi0.5)
- 0%
Decision latency, 74 matched tasks
- Frontier agent (GPT-6 Astra)
- 33.3 ± 27.9 s
- VLA (pi0.5)
- 0.12 ± 0.02 s
New tasks
- Frontier agent (GPT-6 Astra)
- Attempted with no fine-tuning
- VLA (pi0.5)
- Usually needs task-specific fine-tuning
General reasoning
- Frontier agent (GPT-6 Astra)
- Kept in one model
- VLA (pi0.5)
- Reported to degrade after robot training
Latency is the catch. GPT-6 Astra can issue several commands in one turn, so the median of all its individual intervals was 0.17 seconds, but the per-run medians averaged 33.3 seconds. The authors state plainly that the agent reaches its zero-shot success at a decision latency more than two orders of magnitude above a VLA. Catching moving objects or reacting next to people is out of reach with that loop.
The head-to-head covers two benchmarks where the agent did well, and the VLAs were not fine-tuned. It does not show how a tuned VLA would compare.
Limits to keep in mind before citing these results
- Preprint. Version 1 appeared November 19, 2026 and has not been peer reviewed.
- Same model as policy and critic. The authors tested for self-preference: GPT-6 Astra labeled only 19 of 264 human-judged failures (7.2%) as Achieved, and Claude Fable 5.2 from another model family reached similar outcome accuracy (78.25%). Human validation still covers only 400 actions.
- Essentially one policy model. Other frontier models may fail in different ways.
- Heavy on simulation, uneven by benchmark. The real bimanual desk suite, StationeryBench, had 7% episode success across 100 episodes, and RoboDojo's 2,100 episodes succeeded 22% of the time.
- Critic error. Even the best critic stayed below 80% outcome accuracy, so treat failure-mode shares as trends, not exact rates.
How to evaluate robot agent claims after this study
- Ask for more than episode success. Check whether intermediate failures and recoveries are reported.
- Separate pick-and-place work from tight-tolerance insertion or alignment, which this study found much harder.
- Ask about decision latency and whether the task can tolerate tens of seconds per step.
- If you run robot-agent experiments, look at the released dataset and the Muse-Glimmer-30B critic for low-cost action-level analysis, and verify key labels by hand.
- Follow reactions and follow-up work through theddari's AI Telegram and blog trend trackers.
AgentRoboCritic gives a clear picture of frontier agents as robot brains: strong scene understanding and self-correction, with physical precision and slow decisions as the open problems. The next papers to watch are those that improve grasp and alignment accuracy or shorten the decision loop.
Frequently asked questions
What does direct robot control by an LLM agent mean?
Does AgentRoboCritic show LLM agents beat vision-language-action models?
Why does stating a local intent make action-level evaluation work?
Can I use the released critic model and dataset?
Sources and references
Related articles
AIAnthropic's Riemann Hypothesis Proof: Is It Verified?
On November 20, 2026, Anthropic researcher Levent Alpöge said he had proved the Riemann Hypothesis with Claude. A day later the Lean and Mathlib communities certified the formal proof. What that means, what is still missing, and a checklist for judging big proof claims.
AIInkling 2 Mini vs Inkling 2 Small: Specs, Local Use, Limits
Thinking Machines released Inkling 2 Mini on November 20, 2026: a 26.4B-parameter open-weights model with a ~20GB checkpoint that runs on 32GB Macs. Here is how it compares with Inkling 2 Small and what to check before you deploy it.
AIFrostbite CPU: What the AI-Designed Chip Claims Really Show
Architect Labs says AI agents took Frostbite from instruction set to final GDS layout. Here is what the AI actually did, what the 4.59x efficiency figure is measured against, and which claims still lack silicon or independent replication.