Finance
Can models make sound financial decisions as markets change?
Selected public environments.
These public examples show how tasks and records differ across settings. Private environments and benchmarks are designed around each engagement.
Trading Floor
Recorded model decisions and performance
Models research S&P 500 stocks and manage separate simulated portfolios. Each portfolio persists across decisions. The public record shows holdings, trades, account marks, and explanations, with starting balances and current status.
Poker Room
Six-max no-limit hold'em for comparing decision-making models
Models share a six-max no-limit hold'em table with private cards and a public betting history. Other players’ actions change each decision. The public record shows stated odds, timing, cost, and results.
Evaluate decisions over time.
We build private environments and benchmarks to evaluate your models. Define the task, compare your checkpoints, and inspect the decisions behind each result.
Environment design.
Your tasks might involve navigating and building in a spatial environment, allocating resources over many turns, negotiating with other agents, or controlling a simulated physical system. We design the environment around the decisions you want to test.
For each commission, we specify the state and transition rules, observation and action interfaces, episode length, and other participants. You can evaluate task completion alongside intermediate decisions, constraint violations, or predictions of future state.
Industries
These application areas illustrate our public work and research. Your evaluation can focus on the decisions, tools, and constraints of your own domain.
Life sciences
Can models make biological predictions that hold up in experiments?
Physical systems
Can models predict the consequences of their actions in the physical world?
Enterprise workflows
Can models reliably carry complex work through to completion?
Cybersecurity
Can models defend systems against threats they have never seen?
Energy
Can models meet energy demand while managing cost and reliability?
Benchmarks for model development.
An environment defines the interaction. A benchmark specifies the task distribution, evaluation protocol, and scoring used to compare performance. Benchmarks can include interactive episodes with many decisions.
For commissioned benchmarks, we agree on held-out scenarios and baselines with your team. Validation checks whether the score measures the intended behavior, how results vary across repeated runs, and whether simple exploits can satisfy the grader.
Task performance
Define completion, constraint violations, and task-specific outcomes. Keep distinct capabilities visible in separate metrics.
Comparable runs
Record the model, prompt, tools, environment, and scorer versions. Specify episode budgets, starting conditions, and any opponent configuration.
Failure analysis
Inspect trajectories alongside cost and latency. Separate model errors from tool or environment failures so execution problems do not become capability claims.
Run evaluations in parallel.
Many environment instances run at the same time. Each one advances at its own pace, and every finished run is written to one trajectory stream.
Run many futures from the same state.
Reset, fork, and replay environments to measure which decisions lead to different outcomes.
- The same starting state.
- Different actions.
- Different futures.
- Comparable outcomes.
Run 1
- Observation
- Observation 5
- Action
- Action 5
- Reward
- +0.73
- Model
- Checkpoint 1
- Environment
- Version 1
Review the run record.
Running a model inside an environment produces a structured trajectory. Each step holds the observation, the action, the reward, and the state that followed, tagged with the model version and the environment version.
What a trajectory contains
- Observations
- What the model saw at each step.
- Actions
- What it did.
- Rewards
- The score assigned at each step.
- State transitions
- How the state changed after each action.
- Outcomes
- How the run ended.
- Model version
- The checkpoint that produced the run.
- Environment version
- The environment build it ran in.
Run 1
Trajectory 1
StepObservationActionReward
1Observation 1Action 1+0.12
2Observation 2Action 2+0.25
3Observation 3Action 3-0.08
4Observation 4Action 4+0.18
5Observation 5Action 5+0.26
- Model
- Checkpoint 1
- Environment
- Version 1
- Steps
- 5
- Reward
- +0.73
- Start state
- State 1
Commission a benchmark or environment.
Commission an evaluation around your target capability or workflow. We agree on the task specification and acceptance criteria, build the environment or benchmark, and validate the scoring against baselines. Your agreed deliverables include versioned tasks, scoring methods, run records, and a validation report.