SWE-RaceLeaderboardTasks
Docs
MethodologyLeak auditSubmit a modelChangelog
SWE-Race · public split

95 public tasks

tasks.csvresults.json

Prompt, hidden tests, reference fix and every agent attempt for each. 93 more are held out and never listed. Request the full set →

TaskProjectKindTierf2pMergedGPT-5.6 LunaDeepSeek V4 FlashGLM-5.3 Flash
a2aproject-a2a-python-440a2aproject/a2a-pythonsingleclean32025-09-036/62/22/2
agronholm-anyio-1110agronholm/anyiosingleclean42026-04-146/62/22/2
agronholm-anyio-1145agronholm/anyiosinglehard22026-06-156/60/22/2
agronholm-anyio-1217agronholm/anyiosinglesplit12026-07-125/62/22/2
agronholm-anyio-1218agronholm/anyiosingleclean22026-07-116/62/25/5
agronholm-anyio-1223agronholm/anyiosingleclean12026-08-236/62/22/2
agronholm-anyio-1279agronholm/anyiosingleclean12026-08-236/62/22/2
agronholm-anyio-1279-1223agronholm/anyiocompositeclean22026-08-2310/104/43/3
agronholm-anyio-485agronholm/anyiosingleclean42022-10-206/62/22/2
agronholm-anyio-790agronholm/anyiosinglehard32024-12-055/61/22/2
aio-libs-aiobotocore-1537aio-libs/aiobotocoresingleclean12026-05-036/62/22/2
aio-libs-aiohttp-10923aio-libs/aiohttpsingleclean22025-05-216/62/22/2
aio-libs-aiohttp-12493aio-libs/aiohttpsingleclean12026-05-106/62/22/2
aio-libs-aiohttp-8445-8482aio-libs/aiohttpcompositeclean32024-07-176/62/22/2
aio-libs-aiohttp-9140aio-libs/aiohttpsingleclean12024-09-136/62/22/2
aiven-open-astacus-259Aiven-Open/astacussingleclean12024-11-146/62/22/2
aws-aws-advanced-python-wrapper-1089aws/aws-advanced-python-wrappersingleclean22026-01-306/62/22/2
berriai-litellm-16672-16708BerriAI/litellmcompositeclean32025-11-156/62/25/5
berriai-litellm-20666-22884BerriAI/litellmcompositehard52026-03-056/61/20/2
berriai-litellm-21658-22884BerriAI/litellmcompositehard32026-03-054/61/20/2
berriai-litellm-8874BerriAI/litellmsinglehard12025-02-276/60/20/2
bumpy-croc-ai-trading-bot-636-637bumpy-croc/ai-trading-botcompositehard52026-06-014/61/21/2
celery-celery-10086celery/celerysingleclean16/62/22/2
celery-celery-10179-10189celery/celerycompositehard192026-03-080/60/20/2
celery-celery-10189-10158celery/celerycompositesplit152026-04-044/62/21/2
celery-celery-10204celery/celerysingleclean32026-03-266/62/22/2
celery-celery-10218celery/celerysinglesplit36/62/21/2
celery-celery-10218-10204celery/celerycompositehard62026-03-2610/106/82/2
celery-celery-10408-10404celery/celerycompositehard42026-07-150/60/20/2
celery-kombu-2492celery/kombusinglesplit52026-03-265/61/22/2
celery-kombu-2492-2552celery/kombucompositehard62026-06-102/61/21/2
celery-kombu-2498celery/kombusingleclean42026-03-256/62/22/2
celery-kombu-2498-2492celery/kombucompositehard92026-03-266/60/21/2
celery-kombu-2498-2492-2552celery/kombucompositehard102026-06-100/60/21/2
celery-kombu-2552celery/kombusinglesplit12026-06-105/62/21/2
d0ttino-ume-561d0tTino/UMEsingleclean12025-07-226/62/22/2
dewet22-givenergy-hass-51-73dewet22/givenergy-hasscompositehard22026-05-306/80/22/2
eandersson-amqpstorm-152eandersson/amqpstormsingleclean12026-05-246/62/22/2
fastapi-fastapi-14349-14297fastapi/fastapicompositesplit32025-11-137/82/22/2
flyteorg-flyte-sdk-1142-1178flyteorg/flyte-sdkcompositeclean52026-06-026/62/22/2
fossasia-voxbento-319fossasia/voxbentosingleclean22026-08-046/62/22/2
frequenz-floss-frequenz-sdk-python-1166frequenz-floss/frequenz-sdk-pythonsingleclean12025-02-275/62/22/2
giampaolo-psutil-2515giampaolo/psutilsingleclean12025-03-046/62/22/2
helicopterrun-aiophyn-6helicopterrun/aiophynsingleclean22025-11-086/62/22/2
hkuds-lightrag-3056-3129HKUDS/LightRAGcompositehard152026-05-241/60/20/2
hkuds-lightrag-3129HKUDS/LightRAGsingleclean22026-05-245/62/22/2
huggingface-huggingface-hub-2528huggingface/huggingface_hubsingleclean12024-09-106/62/22/2
jonathan-nascimento51-glpi-dashboard-cau-71jonathan-nascimento51/glpi_dashboard_causingleclean12025-07-226/62/22/2
joopsnijder-multi-search-api-2Joopsnijder/multi-search-apisingleclean22025-11-036/62/22/2
k1rl3s-maxo-348K1rL3s/maxosinglehard62026-09-144/60/22/2
kalarb-pykalshi-47Kalarb/pykalshisingleclean112026-05-296/62/25/5
komalharshita-devpath-1846-1890komalharshita/DevPathcompositehard192026-08-133/60/20/2
kpn-combadge-132kpn/combadgesingleclean12024-10-046/62/22/2
laixintao-prometheus-http-sd-39laixintao/prometheus-http-sdsinglesplit22023-04-186/61/22/2
langflow-ai-langflow-15026langflow-ai/langflowsinglesplit32026-09-116/61/22/2
learningcircuit-local-deep-research-2517-2502LearningCircuit/local-deep-researchcompositesplit102026-03-066/61/22/2
litestar-org-litestar-3479litestar-org/litestarsinglehard12024-05-084/60/21/2
mapillary-mapillary-tools-779mapillary/mapillary_toolssinglesplit102025-08-276/62/21/2
matterminers-tardis-371MatterMiners/tardissingleclean22025-04-286/62/22/2
miquido-haiway-146miquido/haiwaysinglesplit82025-08-266/61/22/2
mitmproxy-mitmproxy-5589mitmproxy/mitmproxysinglehard12022-09-223/60/20/2
mmcdermott-meds-transforms-374mmcdermott/MEDS_transformssinglesplit62026-04-176/62/21/2
nousresearch-hermes-agent-32018-33241NousResearch/hermes-agentcompositeclean102026-05-276/62/25/5
nousresearch-hermes-agent-83536-99176NousResearch/hermes-agentcompositesplit92026-08-315/61/22/2
pavua-krab-ear-1161-1290Pavua/Krab-Earcompositeclean42026-05-278/82/25/5
pavua-krab-ear-1264-1290Pavua/Krab-Earcompositeclean32026-05-276/62/25/5
pavua-krab-ear-1297-1290Pavua/Krab-Earcompositesplit42026-05-278/80/12/2
peterager-lennoxs30api-87PeteRager/lennoxs30apisingleclean12025-02-086/62/22/2
pipecat-ai-pipecat-4186pipecat-ai/pipecatsingleclean32026-03-286/62/22/2
prefecthq-fastmcp-2803PrefectHQ/fastmcpsingleclean42026-01-075/62/22/2
prefecthq-fastmcp-3480PrefectHQ/fastmcpsinglehard32026-03-132/61/22/2
prefecthq-fastmcp-3899-3889PrefectHQ/fastmcpcompositesplit152026-04-145/61/22/2
prefecthq-prefect-19601-19708PrefectHQ/prefectcompositehard42025-12-020/60/20/2
protolabsai-protoagent-302-304protoLabsAI/protoAgentcompositesplit42026-05-308/81/22/2
pydantic-pydantic-ai-4419-4502pydantic/pydantic-aicompositehard72026-03-025/61/21/2
pydantic-pydantic-ai-4502pydantic/pydantic-aisingleclean12026-03-026/62/22/2
pytest-dev-pytest-14540-14567pytest-dev/pytestcompositehard62026-06-080/60/20/2
pytest-dev-pytest-14566-14540pytest-dev/pytestcompositeclean52026-06-085/62/25/5
python-poetry-poetry-8517python-poetry/poetrysinglehard22023-10-291/61/22/2
python-roborock-python-roborock-632Python-roborock/python-roborocksingleclean32025-12-056/62/22/2
redis-redis-py-3863redis/redis-pysinglesplit22025-12-045/62/22/2
rhysu-droll-182-190RhysU/drollcompositehard42026-03-048/80/22/2
sblibs-pyswitchbot-368sblibs/pySwitchbotsingleclean12025-07-076/62/22/2
strands-agents-harness-sdk-2800strands-agents/harness-sdksingleclean12026-06-156/62/22/2
tornadoweb-tornado-3553-3554tornadoweb/tornadocompositeclean22025-12-106/62/22/2
tox-dev-python-discovery-79-85tox-dev/python-discoverycompositehard62026-06-115/60/21/2
tox-dev-sphinx-autodoc-typehints-605-606tox-dev/sphinx-autodoc-typehintscompositeclean22026-02-188/82/25/5
tox-dev-sphinx-autodoc-typehints-607-605-606tox-dev/sphinx-autodoc-typehintscompositehard42026-02-181/60/21/2
tox-dev-sphinx-autodoc-typehints-607-606tox-dev/sphinx-autodoc-typehintscompositeclean32026-02-188/82/25/5
twingate-kubernetes-operator-96Twingate/kubernetes-operatorsinglesplit252024-01-196/61/22/2
vllm-project-vllm-42585vllm-project/vllmsinglehard22026-05-266/82/42/3
vyperlang-vyper-3925-4059vyperlang/vypercompositehard132024-05-280/60/20/2
vyperlang-vyper-3925-4060vyperlang/vypercompositehard172024-05-280/60/20/2
zauberzeug-nicegui-5931zauberzeug/niceguisingleclean12026-04-026/62/22/2
zauberzeug-nicegui-6295zauberzeug/niceguisingleclean22026-09-106/62/25/5