What changed
Plannedwhat we are working on next
- More models. Claude Opus 5.5 and GPT-5.6 Sol are next. Request a model.
- Go and Rust versions. Readers asked for languages where threads run in parallel. The same pipeline (real merged fixes, the project's own tests, a private half) works for them.
- Harness comparison. The same model under different agents (mini-SWE-agent, OpenHands, Claude Code) on the same tasks.
v0.15 entries · 25 Sep 2026 → 13 Oct 2026
The public split is in the Harbor registry: harbor run -d evaligo/swe-race. 94 of the 95 public tasks; one task's test fixture has a fixed 30-second server start that Harbor's sandbox does not always meet.
DeepSeek V4 Flash added (387 attempts). The leaderboard now counts every attempt made against the shipped version of a task, including the September calibration runs: GLM-5.3 Flash 441 attempts, GPT-5.6 Luna 1,178. GLM moves from 85.1% (one attempt per task) to 81.9%; ranks are unchanged.
First results: GPT-5.6 Luna (4 attempts per task) and GLM-5.3 Flash (1 attempt). 188 tasks after a release check that excluded 3; see the methodology page.
Public/private split regrouped so that tasks sharing an upstream pull request stay in one half (21 pull requests had crossed). 95 public and 95 private tasks before verification.
Pool frozen at 191 tasks from 108 projects. Copyleft, unlicensed and noncommercial projects excluded.