Survey · Benchmark · Toolbox

Robotic Reward Models
in the Foundation-Model Era

A Survey and Capability Benchmark

Guoqiang Liang*1 Yiqi Lin*1 Shangchen Zhou2 Wenzheng Zeng1 Yanzhe Chen1 Zechen Bai1 Mike Zheng Shou†1

1National University of Singapore 2Nanyang Technological University *Equal contribution †Corresponding author

Task state (human label) Claude Opus 5.5 reward Real execution · put the books back · 2× speed · replay it in the Lab ↓
01 / Survey

One word. Five questions.

Success probabilities, completion percentages, preferences, expected returns and diagnoses all go by reward. We group 114 methods by the question their native output answers.

Outcome 12

Is the task done, or will it fail?

success · grade · failure forecast

Absolute Progress 29

How far along is it?

pₜ ∈ [0, 1] on a whole-task scale

Relative 23

Which is better?

pairwise order · utility from comparisons

Temporal Value 26

Which stage, and how well is it going?

stage · time to go · Vπ, Qπ

Hybrid 24

What happened, and on what evidence?

several judgments from one evaluator

one method; hover or tap for its name* evaluates generated behavior‡ a protocol or harness

All 114 methods by family and design route

* evaluates generated behavior · ‡ a protocol or harness

02 / Benchmark

RRM-Bench-1K

Real closed-loop executions, failures and retries kept whole, labeled by which task conditions hold at each moment and by whether each camera shows enough to judge.

1,117real-robot videos
205three-camera inputs
52tasks
5embodiments
17,381scored 1 FPS transitions
1,702task-state falls
24 of the 1,117 videos. ARX-5, UR5, Franka, Cobot Magic ALOHA and DOS-W1, up to three cameras per execution.

Four contrast suites each separates the credited condition from a cue that usually comes with it

03 / Reward Lab

Credit vs. task state

Ten real executions from the suites. Play one and watch nine evaluators' outputs against the human task-state label.

0.0 s

    Task state (GT) Model GT reversal

    Saved outputs at each model's native timestamps after a fixed unit conversion, no fitting. VOC is the benchmark-level value from the paper; r is the per-video Pearson correlation with the label, not a paper metric. Qualitative examples picked from the benchmark, not a random sample.

    04 / Findings

    What one score hides

    A unified study of 27 evaluators separates capabilities that a single leaderboard number merges.

    Endpoint judgment 75–85%

    Final scores can pick the winner

    For the strongest evaluators, the highest final score picks the most complete execution of a task in 75–85% of cases. Their top tenth by final score holds 90–100% fully completed executions. RoboReward filters this well with the lowest falling F1 among specialized evaluators: endpoint judgment and dense credit are separate capabilities.

    Highest final score picks the most complete execution

    best evaluators 75–85%
    0random 44%100%

    Fully completed executions among the top tenth by final score

    best evaluators 90–100%
    0random 47%100%

    Selection: Sequence-queried GPT-6.1 Sol and GPT-5.6 Terra, then Jev 1.14 and Flora-Rewarder. Filtering: Robometer and RoboReward, then several evaluators at 90%.

    Dense credit VOC 1.000

    A clock beats every evaluator on VOC

    A control that reads only elapsed time tops the standard progress metric, yet never detects a fall. Where task state and time diverge, 17 of 27 evaluators follow time.

    Specialized General VLM Decision model Time-only control
    Evidence 24 / 26

    No evaluator says “I can't see”

    Hiding the decisive view lowers Macro-F1 for 24 of 26 single-view evaluators, and no evaluated interface can report that its evidence is insufficient. Extra cameras help where the interface fuses them, yet the judgment shifts with the camera in the primary slot.

    Sufficient view (V) Reduced view (P/X)

    Six directions from the gaps above

    1. Learning from real failuresSigned targets and cause labels, so credit can fall.
    2. Task structure beyond the instructionCheck the constraints that motion does not reveal.
    3. Query designPair-query recall with Sequence-query precision.
    4. Evidence sufficiency and abstentionAbstain or widen uncertainty without the decisive view.
    5. Measuring what mattersTest credit withdrawal apart from chronology.
    6. From evaluators to decisionsMatch fidelity to the optimization pressure of the decision.
    05 / Leaderboard

    26 single-view evaluators

    Direction metrics over 17,381 non-flat 1 FPS transitions, 1,702 of them falling. Because nine in ten transitions rise, always answering “rising” already scores Macro-F1 0.474.

    # Evaluator Macro-F1 Falling F1 VOC τa V P/X Pair
    Always “rising” reference0.4740.000—————
    Normalized elapsed time reference0.4740.0001.0000.000———

    General VLMs and Jev 1.14 use Sequence queries; the Pair column gives their Macro-F1 under Pair queries. † Curve interpolated to 1 FPS. V and P/X: Macro-F1 on the 347 paired views, same transitions. Robo-ValueRL takes only multiview input and is reported with the paper's multiview rows. Bold marks the best value among the rows shown; click a column to sort.

    06 / Toolbox

    RRM-Hub

    Adapters keep each reward model's native inputs and outputs, so existing models run on new robot videos and new evaluators are tested on the same capabilities.

    import rewardbox
    
    rm  = rewardbox.create_reward("robometer-4b")
    out = rm("robot.mp4", "open the drawer")
    
    out.progress        # native scale, never rescaled
    out.frame_indices   # source frame of each value
    out.success         # models that output it
    $ pip install -e .
    $ rewardbox list                  # all configurations
    $ rewardbox doctor robometer-4b   # deps · checkout · GPU
    $ rewardbox run robometer-4b robot.mp4 \
          --instruction "open the drawer" -o result.json
    $ rewardbox bench \
          --manifest RRM-Bench-1K/inputs/single_view.jsonl \
          --model robometer-4b --strategy fps --fps 1 \
          --no-include-last -o runs/robometer-4b
    $ python RRM-Bench-1K/tools/evaluate.py \
          --predictions runs/robometer-4b/robometer-4b.json
    14methods
    35configurations
    1interface
    • Robometer
    • RoboReward
    • TOPReward
    • Robo-Dopamine
    • GVL
    • VLAC
    • VLAC-Cut
    • PRIMO-R1
    • Flora-Rewarder
    • SOLE-R1
    • LRM
    • RynnValue
    • ProcVLM
    • Robo-ValueRL
    • Native outputs. Fractions, percentages, seconds to go or signed scores, never rescaled.
    • Explicit sampling. Every model runs its upstream protocol unless told otherwise.
    • Capabilities apart. Endpoint, dense credit and evidence use are reported separately.
    Citation

    BibTeX

    @article{liang2026rrm,
      title   = {Robotic Reward Models in the Foundation-Model Era},
      author  = {Liang, Guoqiang and Lin, Yiqi and Zhou, Shangchen and
                 Zeng, Wenzheng and Chen, Yanzhe and Bai, Zechen and
                 Shou, Mike Zheng},
      journal = {arXiv preprint arXiv:2611.20741},
      year    = {2026},
      url     = {https://arxiv.org/abs/2611.20741}
    }