Submit a model

How a model gets on the board

Who runs itWe doone protocol for every model; no self-reported scores
A full run188 × 4tasks × attempts; public and private together
Cost to you$0we report the API cost back to you
Contactrequest formwe reply within two working days
Protocol
HarnessBenchGen agent: shell tool, file edits, submit
Step limit100 tool calls; the patch at the limit is graded as-is
Command timeout600 s
Reasoning effortmedium, where available
Networknone
Attempts4 per task; pass@1 is the mean per task
Gradinghidden tests in a separate verifier container; binary reward
Steps
  1. Send the request form with the model identifier, provider and the inference settings you consider standard.
  2. For a private endpoint, include access for the run. Keys are used for the evaluation only and deleted afterwards.
  3. We run 188 tasks × 4 attempts, public and private, under the protocol on the left.
  4. You get the per-attempt results and the cost; the board updates and subscribers are emailed.
Run it yourselfpublic split, any Harbor runner

The public split is on GitHub and Hugging Face, and in the Harbor registry, so it runs with one command:

pip install harbor
harbor run -d evaligo/swe-race -a mini-swe-agent -m <provider/model>

The registry copy has 94 of the 95 public tasks: in one task (prefecthq-prefect-19601-19708) the test fixture of the project gives its test server a fixed 30 seconds to start, which is not always enough inside the Harbor network sandbox.

Your own numbers on the public split are welcome, marked as self-reported; they are listed separately from the board.

CiteBibTeX
@misc{sweracev01,
  title  = {SWE-Race: a benchmark of real concurrency bugs for coding agents},
  author = {Evaligo Labs},
  year   = {2026},
  url    = {https://labs.evaligo.com/swe-race}
}