In development
Example task
Repository issue
Conceptual example
Bug reported
An issue describes failing behavior
Write a patch
Edit files in the repository
Run the tests
Hidden and existing tests pass
What the task measures.
Score whether the hidden tests pass and the existing suite still passes. Record every command, file edit, and test run the model made along the way.
A patch can make one test pass while breaking others or editing the tests themselves. Checking the full suite and the repository state tests whether the model fixed the problem.
Work with Kimpton.
Engagements are custom. We agree on the task, inputs, outcomes, and rewards before we build the environment.