AI Agent Benchmark

The defining feature of a benchmark is that it's fixed: the same tasks, run the same way, scored by the same method, regardless of which agent or model is being tested. That fixedness is what makes the resulting numbers comparable — and it's what makes a leaderboard possible at all.

Benchmarks are useful mainly because they save teams from designing a new test every time they want to compare options. Checking how a new model or agent performs on a few well-known benchmark suites is a common first step when evaluating candidates, well before building anything task-specific. SWE-bench is a widely known example in the coding-agent space — it draws on real issues from open-source repositories and checks whether an agent's fix gets the relevant tests passing. Exact scores shift across model versions and over time, so this description stays deliberately high-level rather than citing a specific number.

The limits are just as real. "Agent benchmarks are broken" is a recurring, legitimate criticism in the field: a fixed task suite can only cover so much of the diversity of real-world work; tasks that stay public long enough risk leaking into later training data, inflating scores without a matching gain in real capability; and the existence of a public leaderboard creates a pull toward optimizing for the benchmark itself rather than for general reliability.

How it works

A typical benchmark is built by selecting or constructing a set of tasks with a clear, checkable notion of success — a code change that either does or doesn't make a test pass, for instance — paired with an automated or semi-automated scoring method. Because every agent runs the identical task set under the identical scoring rules, results from different teams and different runs end up comparable in a way that ad hoc testing isn't.

How it differs

Common misconceptions

FAQ