Evaligo Labs
Benchmarks for coding agents
Evaluations built from real bugs in real projects, graded by hidden tests in sealed containers, with a private split held back to catch overfitting. How we build them.
Concurrency · Python
SWE-Race
188 race conditions, deadlocks and cancellation bugs from 106 projects.
81.9% top pass@1 · GLM-5.3 Flash · updated 3 Oct 2026
Two real bugs per task · Python
SWE-Pair
161 tasks, each two real bugs from one of 69 projects, fixed together from one bug report.
90.4% top pass@1 · DeepSeek V4.1 Flash · medium · released 7 Nov 2026