SWE-PairLeaderboardTasks
Docs
MethodologyChangelog
Methodology

Two real bugs, one bug report

Tasks16162 public · 99 private · 69 projects
Verification3×reference fix and empty submission, in the shipped image
Protocol100 stepsno network · graded as-is at the limit
Left out39 tasks25 for licensing
The pipelinefrom two merged pull requests to a shipped task
  1. 1 · mine

    Pair two real fixes

    Two merged bug-fix pull requests in the same project, usually within a week of each other, each adding tests.

  2. 2 · validate

    Both bugs at one commit

    Both bugs reproduce at a shared base, each fix flips only its own tests, both together pass; three runs against flakiness.

  3. 3 · specify

    One bug report

    A single report that describes both failures and the interface the tests use, never the fix. Audited against the hidden tests.

  4. 4 · seal

    Probe for leaks

    One commit, no remotes, no network, no trace of hidden test names in the sandbox.

  5. 5 · tier

    Measure difficulty

    Two models at maximum reasoning sort tasks into clean, split and hard.

  6. 6 · verify

    Check what ships

    Each image graded three times with the reference fix (must score 1) and an empty submission (must score 0).

Public and private halveshow the split was made
  • Tasks that share an upstream pull request always sit in the same half, so a public reference fix never reveals a private task.
  • 37 public tasks are also SWE-Race public tasks; SWE-Pair tasks built on SWE-Race private pull requests are private here too.
  • Both halves are scored side by side; a model tuned on the public half shows a gap.
Left out before releaseeach with its reason
  • 25 tasks from repositories without a permissive license (no license, GPL/AGPL/LGPL, proprietary or noncommercial).
  • aio-libs__aiohttp-13001-13136: pytest-asyncio 1.4.1 (2026-10-25) warns at startup and the repo's filterwarnings=error turns that into an INTERNALERROR, so no test runs and the reference fix scores 0; every model attempt after 2026-
  • pydantic__pydantic-ai-4419-4502: pytest-asyncio 1.4.1 (2026-10-25) warns at startup and the repo's filterwarnings=error turns that into an INTERNALERROR, so no test runs and the reference fix scores 0; every model attempt after 2026-
  • 12 private task(s) for reasons of the same kinds.
Relation to SWE-Racetwo benchmarks, one pipeline

SWE-Race covers concurrency bugs only; SWE-Pair covers all bug types and always asks for two fixes at once. On SWE-Race, multi-fix tasks were where models separated, which is why SWE-Pair exists. The two leaderboards are independent: SWE-Race v0.1 stays as published.