Agentifying the Evaluation of Visual Worlds
The era of harnessed benchmarking: evaluation defines the taste of evolution.
A benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the reasoning that justifies the score. This is especially critical for world models, where judging a rollout requires understanding whether physics, causality, and world state evolve correctly. Humans spot such violations naturally, yet no existing benchmark automates this capability: metrics are computed brute-force, leaving no reasoning chain that can be examined or verified.
HarnessEval is an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking. Rather than applying a fixed rubric, HarnessEval interprets the context of each evaluation case, decomposes the evaluation question into measurable sub-questions, and spawns specialized sub-agents, each equipped with tailored context and diagnostic tools to reason over its own sub-question. The parent agent then validates the gathered evidence and aggregates it into the final verdict. Every evaluation becomes a transparent evidence tree whose complete reasoning chain justifies the result.
- [2026/08/18] 📄 Paper now available.
- [2026/08/18] 🌐 Homepage with leaderboard-ready reports and benchmark resources is live.
- [2026/08/18] 🚀 Released the full HarnessEval benchmark, evaluation code, fixed plans, and metric backends.
- Case-specific skill routing. Given a case (initial world, action, probe intent), the planner routes it to the skills that can legitimately evaluate it — and records an evidence-grounded reason for every skill it skips.
- Sub-agent reasoning. Each skill decomposes its evaluation into measurable sub-questions, each answered by a dedicated sub-agent against rollout evidence.
- Validated aggregation. The parent agent validates the collected evidence and aggregates it into the case score. The full trace — every question, answer, score, and supporting frame — is saved as an auditable case card.
Routing depends only on the case context, never on the model being evaluated, so every model faces the same questions on the same cases.
Create three environments:
harnesseval-main: launcher / CLIharnesseval-metrics: metric backendsharnesseval-pavrm: physical-plausibility backend
git clone --branch main --single-branch https://github.com/mirros-lab/harnesseval-w.git
cd harnesseval-w
conda env create -f docs/installation/main.environment.yml
conda env create -f docs/installation/metrics.environment.yml
conda env create -f docs/installation/pavrm.environment.yml
conda activate harnesseval-mainConfigure credentials and paths:
cp config/example.env harnesseval.env
set -a; . ./harnesseval.env; set +aThe bundled demo in runs/example/results_example can be evaluated immediately after setup.
Evaluate a model's generated results:
# Evaluate generated videos and write run outputs.
harnesseval eval \
--results runs/example/results_example/generation \
--model-id seedance-2.0-standard \
--run-root runs/example/results_example/run \
--manifest runs/example/results_example/manifest.json \
--plan-root benchmark/plansCheck a completed run:
# Verify that an existing run has all expected scores.
harnesseval verify run \
--eval-root runs/example/results_example/run/harnesseval/models/seedance-2.0-standard/evaluation \
--manifest runs/example/results_example/manifest.json \
--model seedance-2.0-standardOr use the bundled demo end to end:
# Run the bundled demo from the example directory.
cd runs/example/results_example
# Evaluate the demo outputs.
harnesseval eval --results generation --model-id seedance-2.0-standard --run-root run --manifest manifest.json --plan-root ../../benchmark/plans
# Verify the demo run.
harnesseval verify run --eval-root run/harnesseval/models/seedance-2.0-standard/evaluation --manifest manifest.json --model seedance-2.0-standardThe bundled example already produces scores. A completed evaluation writes:
runs/example/results_example/run/harnesseval/models/<model-id>/evaluation/
├── summary.json # overall and per-family scores
├── leaderboard_latest.json
├── leaderboard_latest.csv
└── LEADERBOARD.md
Per-case artifacts and caches live under run/harnesseval/metric_cache/ and run/harnesseval/models/<model-id>/.
HarnessEval is a living benchmark, and contributions are welcome.
- Submit a new case with a new world, action, or probe family.
- Submit a new skill when a case needs a new kind of evaluation.
Keep submissions aligned with the existing benchmark format so they can plug into the bundled flow.
- Hosted submission & evaluation service (submit videos, get scores)
- Full cases on HuggingFace
- Subset cases & weights on HuggingFace
- Model generation example
- Evaluation code & example release
- ArXiv paper, homepage with interactive leaderboard, blog release
This project builds upon the following excellent works:
- VBench — Video quality metrics
- WBench — World model benchmark
- WorldScore - World model benchmark
- Cosmos - Open source world model
- Lingbot World - Open source world model
- MiniMax H3 - Open source world model
- ... and many other excellent open-source projects
If you find HarnessEval useful, please cite:
@article{mirros2026harnessevalw,
title = {HarnessEval-W: Agentifying the Evaluation of Visual Worlds},
author = {{MirroS Team}},
journal = {arXiv preprint arXiv:2608.16859},
year = {2026}
}We release our code under the Apache 2.0 License.