ラウンジ
接続中…
接続中…
すべて見る

AgentRoboCritic: Can Frontier AI Agents Control Robots?

AgentRoboCritic grades GPT-6 Astra's robot commands action by action: 57.7% of failures recovered, grasping and alignment lagged.

AI最新6分で読めます閲覧 0公開 更新
目次8セクション
  1. How AgentRoboCritic grades frontier agents on robot control
  2. What frontier agents did well: recovery, grounding and zero-shot skill
  3. Where they fail: grasping, alignment and uneven recovery
  4. LLM agent vs VLA: success rate traded for speed
  5. Limits to keep in mind before citing these results
  6. How to evaluate robot agent claims after this study
  7. よくある質問
  8. 出典・参考資料

要点

  • AgentRoboCritic makes the agent state a short local intent with every robot command, then has a critic judge each action against that intent and name a failure mode.
  • Across 12 manipulation benchmarks, 57.7% of failed actions were verifiably recovered later in the same episode, and 49.8% of successful episodes contained at least one failed action.
  • Physical execution is the bottleneck: Grasp was the top failure mode on 9 of 10 non-tactile benchmarks, and only 35.7% of Alignment failures were recovered.
  • Zero-shot, GPT-6 Astra succeeded in 91% of RoboTwin 2.0 and 90% of YAMLab episodes versus 6% and 0% for pi0.5, but took 33.3 seconds per decision versus 0.12 seconds.
  • It is a preprint, the same model acts as policy and main critic, and the zero-shot comparison covers two benchmarks.

Yes, frontier AI agents can drive a robot arm directly, but AgentRoboCritic shows the real question is which actions they get right. The study, posted to arXiv on November 19, 2026 by researchers at Carnegie Mellon University, the University of Illinois Urbana-Champaign and Bosch, graded GPT-6 Astra's robot commands one action at a time across 12 manipulation benchmarks (abstract). The short version: the agent picks the right object almost every time and often fixes its own mistakes, but precise physical work such as grasping and alignment is where it breaks down, and each decision is slow.

Figures below are as of November 2026 and come from the v1 paper and the authors' project page. The paper has not been peer reviewed yet, so numbers may change in later versions.

How AgentRoboCritic grades frontier agents on robot control

Most robot benchmarks record a single pass/fail bit per episode. That hides two things: how many mistakes happened inside a successful run, and which step sank a failed one. Grading a single action against the overall goal does not fix this, because many different trajectories can complete the same task. The same motion might be an incomplete reach, a missed alignment or a failed grasp depending on what it was supposed to do.

The authors' fix is a local intent. Every tool call that sends a numerical command must also include a plain-language note on what that action is meant to achieve, and the harness rejects any call without one. A critic model then sees the task goal, that note and multi-view camera frames from before and after the action.

  1. 1

    Agent sends a command plus its intent

    For example, an end-effector pose and gripper value alongside a note like 'close on the blue piece at its current height'.

  2. 2

    Robot executes and returns new images

    The loop is closed with no high-level primitives. Gripper closures are judged with the lift frames that follow, so a grasp only counts if the object stays held.

  3. 3

    Critic labels the outcome and failure mode

    Outcomes are Achieved, Failed, Partial or Unverifiable. Failures get one of seven modes: Grasp, Alignment, Interference, Effect, Incomplete motion, Choice or Unresolved, plus a short evidence note.

  4. 4

    Humans check the critic

    People labeled 400 actions from the same inputs to measure how far the critic can be trusted.

GPT-6 Astra served as the critic. On the 400 human-labeled actions (34% successful, 66% failed) it scored 79.25% outcome accuracy and 80.68% failure-mode accuracy, the best of the models tested, according to the full paper and project page. Without the local intent those numbers collapsed to 25.25% and 25.00%.

What frontier agents did well: recovery, grounding and zero-shot skill

They recover from their own mistakes

Imitation-trained robot policies suffer from covariate shift: one error pushes them outside their training data and errors compound. The project page reports that the agent behaved differently. 49.8% of successful episodes contained at least one failed action. Episodes with one or two local failures still succeeded 40.3% of the time, compared with 75.1% for failure-free episodes.

Tracing 12,163 failed actions across 1,989 graded episodes, 38.9% were verifiably recovered within 10 later actions and 57.7% by the end of the episode. A regrasp only counted once the object visibly rose with the gripper. Across benchmarks, recovery rate and success rate correlated at r = 0.81. One caution: RoboDojo alone contributed 9,923 of those failures, so the pooled figures lean heavily on one suite.

They rarely act on the wrong object

Choice failures, where the action contradicted the object, order or category in the stated intent, made up 0.2% of all failures and never more than 3.3% on any benchmark. That contrasts with specialized models known to degrade under camera shifts and rearranged scenes.

They work without task-specific training

With no fine-tuning or demonstrations, GPT-6 Astra succeeded in 91% of RoboTwin 2.0 episodes and 90% of YAMLab episodes. Under the same zero-shot setup, the VLA pi0.5 reached 6% and 0%, and InternW0-Delta succeeded in no episode on either.

Where they fail: grasping, alignment and uneven recovery

Where they fail: grasping, alignment and uneven recovery

The weak spot is physical precision, not understanding. Grasp was the most frequent failure mode on 9 of the 10 non-tactile benchmarks. On vision-only, tight-tolerance insertion tasks (UniVTAC and OpenViTac), Alignment made up 75% and 86% of failures. Cluttered scenes raised Interference.

Incomplete motion

What it means
A commanded move or opening did not finish
Recovered by episode end
68.2% (479 of 702)

Grasp

What it means
Object not acquired or not retained
Recovered by episode end
63.7% (4,155 of 6,523)

Interference

What it means
Unintended collision or displacement
Recovered by episode end
59.1% (1,602 of 2,711)

Alignment

What it means
Held object misses its target pose
Recovered by episode end
35.7% (681 of 1,908)

Effect

What it means
Tool held but intended effect not produced
Recovered by episode end
29.6% (87 of 294)

Recovery was uneven. Grasp failures were mostly fixed by regrasping, while only about one in three Alignment and Effect failures were recovered, and those rates moved by less than three percentage points after 30 more actions. More attempts do not solve a precision problem.

Three in five Grasp failures happened as the gripper closed: pinching a rim or edge, or closing beside or above the object. The authors point to fine 3D localization of the gripper relative to the object. Another 9.5% of Grasp failures were actions issued with an already empty gripper, meaning the agent sometimes misjudged its own state.

LLM agent vs VLA: success rate traded for speed

LLM agent vs VLA: success rate traded for speed

Control interface

Frontier agent (GPT-6 Astra)
Numerical commands as text tool calls
VLA (pi0.5)
Action head outputs continuous actions

RoboTwin 2.0 zero-shot success

Frontier agent (GPT-6 Astra)
91%
VLA (pi0.5)
6%

YAMLab zero-shot success

Frontier agent (GPT-6 Astra)
90%
VLA (pi0.5)
0%

Decision latency, 74 matched tasks

Frontier agent (GPT-6 Astra)
33.3 ± 27.9 s
VLA (pi0.5)
0.12 ± 0.02 s

New tasks

Frontier agent (GPT-6 Astra)
Attempted with no fine-tuning
VLA (pi0.5)
Usually needs task-specific fine-tuning

General reasoning

Frontier agent (GPT-6 Astra)
Kept in one model
VLA (pi0.5)
Reported to degrade after robot training

Latency is the catch. GPT-6 Astra can issue several commands in one turn, so the median of all its individual intervals was 0.17 seconds, but the per-run medians averaged 33.3 seconds. The authors state plainly that the agent reaches its zero-shot success at a decision latency more than two orders of magnitude above a VLA. Catching moving objects or reacting next to people is out of reach with that loop.

The head-to-head covers two benchmarks where the agent did well, and the VLAs were not fine-tuned. It does not show how a tuned VLA would compare.

Limits to keep in mind before citing these results

  • Preprint. Version 1 appeared November 19, 2026 and has not been peer reviewed.
  • Same model as policy and critic. The authors tested for self-preference: GPT-6 Astra labeled only 19 of 264 human-judged failures (7.2%) as Achieved, and Claude Fable 5.2 from another model family reached similar outcome accuracy (78.25%). Human validation still covers only 400 actions.
  • Essentially one policy model. Other frontier models may fail in different ways.
  • Heavy on simulation, uneven by benchmark. The real bimanual desk suite, StationeryBench, had 7% episode success across 100 episodes, and RoboDojo's 2,100 episodes succeeded 22% of the time.
  • Critic error. Even the best critic stayed below 80% outcome accuracy, so treat failure-mode shares as trends, not exact rates.

How to evaluate robot agent claims after this study

  1. Ask for more than episode success. Check whether intermediate failures and recoveries are reported.
  2. Separate pick-and-place work from tight-tolerance insertion or alignment, which this study found much harder.
  3. Ask about decision latency and whether the task can tolerate tens of seconds per step.
  4. If you run robot-agent experiments, look at the released dataset and the Muse-Glimmer-30B critic for low-cost action-level analysis, and verify key labels by hand.
  5. Follow reactions and follow-up work through theddari's AI Telegram and blog trend trackers.

AgentRoboCritic gives a clear picture of frontier agents as robot brains: strong scene understanding and self-correction, with physical precision and slow decisions as the open problems. The next papers to watch are those that improve grasp and alignment accuracy or shorten the decision loop.

よくある質問

What does direct robot control by an LLM agent mean?
In the paper's definition, a multimodal language agent reads the task instruction, camera images and interaction history, then outputs low-level numerical commands such as end-effector poses, joint targets or gripper openings as structured tool calls. It does not call prebuilt skills like 'pick up cup' or write code against a robot API, and in this study it ran zero-shot, with no task-specific fine-tuning or in-context demonstrations.
Does AgentRoboCritic show LLM agents beat vision-language-action models?
Only under specific conditions. On RoboTwin 2.0 and YAMLab, without fine-tuning, GPT-6 Astra reached 91% and 90% success while pi0.5 reached 6% and 0% and InternW0-Delta 0%. VLAs are normally fine-tuned per task, which this comparison did not do, and the agent's decision latency averaged 33.3 seconds against about 0.12 seconds for pi0.5. For fast, reactive tasks that gap matters more than the success rate.
Why does stating a local intent make action-level evaluation work?
Manipulation is multimodal: many trajectories can advance the same goal, so one motion compared only to the overall task is ambiguous. In the paper's ablation, removing the local intent dropped GPT-6 Astra's critic outcome accuracy from 79.25% to 25.25% and failure-mode accuracy from 80.68% to 25.00%, with most actions labeled Unverifiable.
Can I use the released critic model and dataset?
The authors say they released 12,844 annotated actions, a 400-action human-labeled test set and a fine-tuned open critic, Muse-Glimmer-30B. Its failure-mode accuracy rose from 32.58% to 74.24% after fine-tuning, and scoring the test set cost $0.79 versus $26.06 for GPT-6 Astra. It is cheaper but less accurate, so spot-check important labels by hand and read the license on the release pages before use.

出典・参考資料

  1. 1arXiv:2611.26613 AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control (abstract)(新しいタブで開きます)arxiv.org
  2. 2AgentRoboCritic full paper, HTML version (arXiv)(新しいタブで開きます)arxiv.org
  3. 3AgentRoboCritic project page(新しいタブで開きます)adu2021.github.io