Define the task and interaction.
A benchmark starts with the behavior it is meant to measure. The specification defines the task distribution, initial conditions, observation and action spaces, and the budget for each episode. Multi-agent tasks also define participant roles, private information, and communication rules.
The environment maintains state and applies the consequences of actions. The benchmark defines which tasks run in that environment and how their results are scored. A task can test a single prediction or a sequence of decisions with delayed feedback.
Starting state
A task and action budget
First action
The model chooses a step
New observation
The environment changes
Revised action
The model updates its plan
Task outcome
Completion and constraints
Illustrative episode structure. Each task defines its observations, actions, and completion criteria.
Measure the intended behavior.
Scoring should follow the task specification. Completion can be checked against environment state, while constraints, prediction error, and reporting accuracy require their own measures. Outcomes may come from a simulator, a tool, or an external event, depending on the task.
Prediction quality and action quality answer different questions. A model that completes a task has not necessarily predicted its consequences accurately. World-model evaluation therefore needs explicit prediction targets and held-out transitions, with scoring validated for those targets.
Control the comparison.
A comparison records the model version, prompt, tools, environment, and scorer. Starting conditions, episode budgets, and opponent configurations must be matched or their differences disclosed. When the goal is to isolate a model change, the rest of the evaluation configuration stays fixed.
The protocol defines held-out tasks and appropriate baselines. Repeated trials estimate variation from sampling, initial states, and other participants. Reports should state the number of trials and uncertainty in the scores. Live conditions and unequal account histories limit which comparisons are justified.
Validate the benchmark and inspect failures.
Commissioned benchmarks are checked against agreed acceptance criteria. Validation examines whether the tasks measure the intended behavior and whether a simple policy or grader exploit can earn credit without doing the task. Task access and hold-out procedures are documented so the limits of contamination controls are clear.
Run records connect observations, actions, and outcomes to their evaluation configuration. Reports distinguish model failures from tool or environment failures, include cost and latency where recorded, and state which conclusions the evidence supports.
The commissioning scope defines the deliverables: versioned tasks and environment code where agreed, scoring methods, baseline results, run records, and a validation report. Data access and publication terms are agreed before the evaluation.
Commission a benchmark or environment.
Define the capability or workflow you need to evaluate. We work with you to specify the tasks, validation criteria, and deliverables.