Coding-agent benchmark · Python · two bugs per task
SWE-Pair
161 tasks, each combining two real bugs fixed in the same open-source project. The agent gets one bug report and must fix both; the hidden tests of both pull requests decide. Half the tasks are held out.
Best pass@190.4%DeepSeek V4.1 Flash · medium · tied with DeepSeek V4.1 Flash · max
Hard subset53.8%DeepSeek V4.1 Flash · medium · 15 tasks
Cheapest solved task$0.027GPT-5.6 Luna · medium
Scale161 tasks69 projects · 997 attempts · 5 entries
| Rank | Model · effort | pass@1 | Hard | $ / solved task | Attempts | details |
|---|
Top-left is good: more tasks solved for less money. Full set.
Tap a row for cost, attempts and more.Ranks are shared when 95% intervals overlap.v0.1 · updated 7 Nov 2026 · changelogRequest the full task set →
How to read this table
- pass@1
- Share of attempts where every hidden fail-to-pass test of both pull requests passes and no pass-to-pass test regresses, averaged per task.
- Entries
- Each row is a model at one reasoning effort. Max and medium effort are listed separately, so the cost of extra reasoning is visible.
- Hard
- The 15 tasks neither max-effort model solved in its first attempt. Those two models set the tiers, so they score 0 there by construction and are left out of this view.
- $ / solved task
- Average model API cost per attempt divided by pass@1.
- Interval
- 95% bootstrap over tasks. Rank follows Scale's rule: overlapping intervals share a rank.
- Protocol
- Same agent for every entry, 100-step limit, no network, one bug report naming both bugs. The patch present at the limit is graded as-is. Every attempt on the shipped task version counts.
- Coverage
- DeepSeek V4.1 Flash · medium, GPT-5.6 Luna · medium, GLM-5.3 Flash · medium: no attempts on a few tasks whose bug report was re-worded before release; their scores cover the tasks they ran.
What the tasks are
Two real bugs per task, from 69 Python projects and all bug types: parsing, None handling, serialization, paths, resource leaks, concurrency. The combined fix never existed upstream as one commit.
Browse the 62 public tasks →How it is graded
Sealed repository at one commit, no network, hidden tests of both pull requests in a clean verifier, binary reward. Every task verified three times before release.
Methodology →Run it
pip install harbor harbor run -d evaligo/swe-pair -a mini-swe-agent -m <provider/model>