Outcome 12
Is the task done, or will it fail?
success · grade · failure forecast
Survey · Benchmark · Toolbox
A Survey and Capability Benchmark
1National University of Singapore 2Nanyang Technological University *Equal contribution †Corresponding author
Success probabilities, completion percentages, preferences, expected returns and diagnoses all go by reward. We group 114 methods by the question their native output answers.
Is the task done, or will it fail?
success · grade · failure forecast
How far along is it?
pₜ ∈ [0, 1] on a whole-task scale
Which is better?
pairwise order · utility from comparisons
Which stage, and how well is it going?
stage · time to go · Vπ, Qπ
What happened, and on what evidence?
several judgments from one evaluator
one method; hover or tap for its name* evaluates generated behavior‡ a protocol or harness
* evaluates generated behavior · ‡ a protocol or harness
Real closed-loop executions, failures and retries kept whole, labeled by which task conditions hold at each moment and by whether each camera shows enough to judge.
Ten real executions from the suites. Play one and watch nine evaluators' outputs against the human task-state label.
Saved outputs at each model's native timestamps after a fixed unit conversion, no fitting. VOC is the benchmark-level value from the paper; r is the per-video Pearson correlation with the label, not a paper metric. Qualitative examples picked from the benchmark, not a random sample.
A unified study of 27 evaluators separates capabilities that a single leaderboard number merges.
For the strongest evaluators, the highest final score picks the most complete execution of a task in 75–85% of cases. Their top tenth by final score holds 90–100% fully completed executions. RoboReward filters this well with the lowest falling F1 among specialized evaluators: endpoint judgment and dense credit are separate capabilities.
Highest final score picks the most complete execution
Fully completed executions among the top tenth by final score
Selection: Sequence-queried GPT-6.1 Sol and GPT-5.6 Terra, then Jev 1.14 and Flora-Rewarder. Filtering: Robometer and RoboReward, then several evaluators at 90%.
A control that reads only elapsed time tops the standard progress metric, yet never detects a fall. Where task state and time diverge, 17 of 27 evaluators follow time.
Hiding the decisive view lowers Macro-F1 for 24 of 26 single-view evaluators, and no evaluated interface can report that its evidence is insufficient. Extra cameras help where the interface fuses them, yet the judgment shifts with the camera in the primary slot.
Direction metrics over 17,381 non-flat 1 FPS transitions, 1,702 of them falling. Because nine in ten transitions rise, always answering “rising” already scores Macro-F1 0.474.
| # | Evaluator | Macro-F1 | Falling F1 | VOC | τa | V | P/X | Pair |
|---|---|---|---|---|---|---|---|---|
| Always “rising” reference | 0.474 | 0.000 | — | — | — | — | — | |
| Normalized elapsed time reference | 0.474 | 0.000 | 1.000 | 0.000 | — | — | — |
General VLMs and Jev 1.14 use Sequence queries; the Pair column gives their Macro-F1 under Pair queries. † Curve interpolated to 1 FPS. V and P/X: Macro-F1 on the 347 paired views, same transitions. Robo-ValueRL takes only multiview input and is reported with the paper's multiview rows. Bold marks the best value among the rows shown; click a column to sort.
Adapters keep each reward model's native inputs and outputs, so existing models run on new robot videos and new evaluators are tested on the same capabilities.
import rewardbox
rm = rewardbox.create_reward("robometer-4b")
out = rm("robot.mp4", "open the drawer")
out.progress # native scale, never rescaled
out.frame_indices # source frame of each value
out.success # models that output it
$ pip install -e .
$ rewardbox list # all configurations
$ rewardbox doctor robometer-4b # deps · checkout · GPU
$ rewardbox run robometer-4b robot.mp4 \
--instruction "open the drawer" -o result.json
$ rewardbox bench \
--manifest RRM-Bench-1K/inputs/single_view.jsonl \
--model robometer-4b --strategy fps --fps 1 \
--no-include-last -o runs/robometer-4b
$ python RRM-Bench-1K/tools/evaluate.py \
--predictions runs/robometer-4b/robometer-4b.json
@article{liang2026rrm,
title = {Robotic Reward Models in the Foundation-Model Era},
author = {Liang, Guoqiang and Lin, Yiqi and Zhou, Shangchen and
Zeng, Wenzheng and Chen, Yanzhe and Bai, Zechen and
Shou, Mike Zheng},
journal = {arXiv preprint arXiv:2611.20741},
year = {2026},
url = {https://arxiv.org/abs/2611.20741}
}