Submit a model
How a model gets on the board
Who runs itWe doone protocol for every model; no self-reported scores
A full run188 × 4tasks × attempts; public and private together
Cost to you$0we report the API cost back to you
Protocol
| Harness | BenchGen agent: shell tool, file edits, submit |
| Step limit | 100 tool calls; the patch at the limit is graded as-is |
| Command timeout | 600 s |
| Reasoning effort | medium, where available |
| Network | none |
| Attempts | 4 per task; pass@1 is the mean per task |
| Grading | hidden tests in a separate verifier container; binary reward |
Steps
- Send the request form with the model identifier, provider and the inference settings you consider standard.
- For a private endpoint, include access for the run. Keys are used for the evaluation only and deleted afterwards.
- We run 188 tasks × 4 attempts, public and private, under the protocol on the left.
- You get the per-attempt results and the cost; the board updates and subscribers are emailed.
Run it yourselfpublic split, any Harbor runner
The public split is on GitHub and Hugging Face, and in the Harbor registry, so it runs with one command:
pip install harbor harbor run -d evaligo/swe-race -a mini-swe-agent -m <provider/model>
The registry copy has 94 of the 95 public tasks: in one task (prefecthq-prefect-19601-19708) the test fixture of the project gives its test server a fixed 30 seconds to start, which is not always enough inside the Harbor network sandbox.
Your own numbers on the public split are welcome, marked as self-reported; they are listed separately from the board.
CiteBibTeX
@misc{sweracev01,
title = {SWE-Race: a benchmark of real concurrency bugs for coding agents},
author = {Evaligo Labs},
year = {2026},
url = {https://labs.evaligo.com/swe-race}
}