Evaluation
The Core fixes data, robot, physics, reset, failures and scoring. Each controller uses its released checkpoint and documented inference path, history and control timing.
Each benchmark declares its motion set, robot, simulator settings, initialization and failure conditions. Evaluate every declared motion with one checkpoint, from first frame to endpoint; record the first failure and continue without resetting.
Tracking failure rules
Root height and tilt conditions follow SuperMimic Appendix A. Compare robot and reference states at the same timestamp; any exceeded limit marks a failure.
| Condition | Measurement | Default limit |
|---|---|---|
| Root height | Absolute world-Z position difference | > 0.25 m |
| Root inclination | Absolute difference between the local Z components of projected unit gravity | > 0.8 |
Height is relative to the reference. Inclination is dimensionless and does not measure yaw or the full relative rotation. No XY or yaw alignment is applied to failure checks.
Equality passes; a failure at the final sample still counts. Each benchmark declares its root body and freezes its thresholds. End-effector deviation does not trigger failure. Different thresholds require a separately versioned configuration.
Policy observations
Ideal proprioception and reference motion, without robot world-position or contact feedback. Rules apply to initialization, preprocessing, history and inference.
| Input or computation | Allowed |
|---|---|
| Joint state, full root orientation (including yaw), base linear and angular velocity, and gravity | ✓ |
| Local body positions and orientations from forward kinematics (FK) | ✓ |
| Reference motion, world trajectory and future frames | ✓ |
| Native proprioceptive history, past actions and recurrent memory | ✓ |
| Robot world positions (root and bodies, including height) | × |
| Robot contact states, forces and impulses | × |
| Tracking errors derived from robot world position | × |
| Velocity integration, odometry or explicit global-position estimators | × |
Metrics
Primary metric: No termination · XY only MPBPE, in mm. Lower is better; yaw and height errors are retained.
| Metric | Definition |
|---|---|
| Success ↑ | Fraction of complete motions without failure |
| Completion ↑ | Last valid sample time before first failure divided by motion duration; averaged equally across motions. Successful motions count as 100%. |
| MPBPE ↓ · mm | Mean per-body position error over the benchmark’s declared body origins; average bodies and samples within each motion, then weight motions equally. |
| Alignment | Removed before measuring error |
|---|---|
| XY only | Per-frame horizontal root translation |
| XY + yaw | Per-frame horizontal root translation and the closest relative rotation about world Z |
| None | Nothing; compare world positions directly |
Height and tilt are retained in all three views. Alignment does not change success or completion.
Tracking-error coverage
| Time range | Samples included |
|---|---|
| Early termination | Through the first failing sample, including that sample; the entire motion if no failure occurs. |
| No termination | The entire motion through its exact endpoint, including post-failure states, without resetting. |
Two time ranges × three alignments give six tracking-error views. Success and completion always use the first failure; later recovery does not erase it.
Model requirements
- Public general-controller checkpoint; the same weights for the complete suite.
- Author-faithful inference and permitted observations; disclose adaptations and training-data overlap.
- Public adapter, pinned Core and dependencies; complete traces, independent verification and a repeated run.
Single-motion checkpoints do not qualify. No minimum score is required. Each model card links its exact-weight handling and evidence.