Changelog

What changed

Currentv0.1updated 3 Oct 2026
Tasks18895 public · 93 private
Models3DeepSeek V4 Flash, GLM-5.3 Flash, GPT-5.6 Luna
Entries5since 25 Sep 2026
Plannedwhat we are working on next
  • More models. Claude Opus 5.5 and GPT-5.6 Sol are next. Request a model.
  • Go and Rust versions. Readers asked for languages where threads run in parallel. The same pipeline (real merged fixes, the project's own tests, a private half) works for them.
  • Harness comparison. The same model under different agents (mini-SWE-agent, OpenHands, Claude Code) on the same tasks.
v0.15 entries · 25 Sep 2026 → 13 Oct 2026
13 Oct 2026

The public split is in the Harbor registry: harbor run -d evaligo/swe-race. 94 of the 95 public tasks; one task's test fixture has a fixed 30-second server start that Harbor's sandbox does not always meet.

3 Oct 2026

DeepSeek V4 Flash added (387 attempts). The leaderboard now counts every attempt made against the shipped version of a task, including the September calibration runs: GLM-5.3 Flash 441 attempts, GPT-5.6 Luna 1,178. GLM moves from 85.1% (one attempt per task) to 81.9%; ranks are unchanged.

2 Oct 2026

First results: GPT-5.6 Luna (4 attempts per task) and GLM-5.3 Flash (1 attempt). 188 tasks after a release check that excluded 3; see the methodology page.

29 Sep 2026

Public/private split regrouped so that tasks sharing an upstream pull request stay in one half (21 pull requests had crossed). 95 public and 95 private tasks before verification.

25 Sep 2026

Pool frozen at 191 tasks from 108 projects. Copyleft, unlicensed and noncommercial projects excluded.