SWE-bench Verified

Real GitHub issues

benchmarkAgents% of issues resolved (tests pass), 'Verified' subset500 items

What this measures

The real deal: given an actual open GitHub issue and a whole codebase, can the model write a patch that fixes the bug and passes the project's tests? Measures agentic software engineering.

Sample

Issue #1234 in a real Python repo: 'TypeError when config is empty.' The model must edit the right files so the failing tests pass.

Leaderboard · 5 models

% of issues resolved (tests pass), 'Verified' subset — higher is better

1Claude 3.7 Sonnetvendor-reported · 2025-02

62.3

2GPT-4.1vendor-reported · 2025-04

54.6

3Claude 3.5 Sonnetvendor-reported · 2024-10

4o1vendor-reported · 2024-12

48.9

5Claude 3.5 Haikuvendor-reported · 2024-11

40.6

Benchmark health

headroom

Plenty of room at the top — this benchmark still separates frontier models clearly.

51.1

top-10 mean

62.3

best

models

What improves this score

Datasets & environments shown to raise it.

SWE-Gym+15.0

WebArena+4.0

Research using this benchmark

all →

LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling Dockerless: Environment-Free Program Verifier for Coding Agents Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Matched by name in title/abstract.