SWE-Bench Pro

Benchmark models and coding agents on long-horizon software engineering tasks.

Stars
★ 569
Last updated
1mo ago
License
MIT
Primary language
Python

At a glance

How it runs
CLILibrary / SDK
Works with
Universal · cross-platform
Cost
Free, no paid service needed
Setup effort
Medium · a few setup steps
You'll need
Python dependencies from requirements.txtDockerModal (recommended for the v1 evaluation pipeline)Harbor (for V2 evaluation)Shell / CLINetwork accessLocal filesystem
Typical use
Model research teams comparing how different models or coding harnesses perform on long-running, repository-level fixes.
Not a fit if
  • Developers seeking a ready-to-use coding assistant
  • Teams unable to run containerized evaluations
  • Users focused only on short, isolated coding problems
Maintenance
Commits in the last 6 monthslast commit 2026-09-22latest release v2.0.0 (2026-09-22)+1.4 ★/day over 7 days

What does this agent do, and when should you use it?

SWE-Bench Pro is a benchmark, dataset, and evaluation toolkit for long-horizon software engineering rather than an end-user coding assistant. Each task gives a model a codebase and an issue and asks it to produce a resolving patch. V2 contains 642 validated tasks across 11 repositories, the HARD-51 subset, and self-contained Harbor task directories with verifiers, reference solutions, and public container images. The repository also preserves the original v1 pipeline, including swe_bench_pro_eval.py, run scripts, Dockerfiles, and Docker Hub images. Users can load the data from Hugging Face, generate .pred files with their chosen harness or the SWE-agent submodule, consolidate them into JSON, and run containerized evaluations.

It reads tasks from the Hugging Face ScaleAI/SWE-bench_Pro dataset and supplies the codebase-and-issue workload for a selected model or harness to solve. The SWE-agent submodule provides one patch-generation scaffold, while other harnesses may also be used; generated patches are stored as per-instance .pred files. helper_code/gather_patches.py consolidates those files into a JSON array containing instance_id, patch, and prefix. For v1, swe_bench_pro_eval.py evaluates that JSON against task samples and run_scripts using container images and configurable parallel workers. V2 instead packages each case as a Harbor directory under v2/tasks/, together with its verifier, reference solution, and GHCR image. V2 also includes HARD-51 and locked-protocol tooling for an offline agent phase followed by re-grading in a fresh sandbox.

  1. Model research teams comparing how different models or coding harnesses perform on long-running, repository-level fixes.
  2. Agent developers who need executable, containerized patch verification instead of judging textual answers alone.
  3. Researchers measuring performance separately on the full V2 set, HARD-51, or the original v1 dataset.
  4. Leaderboard participants reproducing SWE-agent or mini-swe-agent runs and preparing consolidated prediction files.
  5. Evaluation-infrastructure engineers studying a locked protocol with offline execution and fresh-sandbox re-grading.

Key setup facts

Pulled from the install and usage notes below. The project README has the current values.

Install / run
  • pip install -r requirements.txt
Config files
~/.modal.toml sample1_patches.json
Install notes cover
Docker

How do you install or deploy this agent?

Install the repository's declared Python dependencies:

pip install -r requirements.txt

Install Docker for reproducible containerized evaluation. The v1 workflow recommends configuring Modal:

modal setup  # Follow the prompts to generate your token

Then verify that ~/.modal.toml contains:

token_id = <token id>
token_secret = <token secret>
active = true

For the v1 local Docker Beta path, no additional Modal setup is required; add --use_local_docker when running evaluations. V2 uses Harbor for evaluation. The README shows a harbor run invocation but does not provide a Harbor installation command.

How do you use this agent?

Load the default V2 test split, HARD-51, or the original v1 data:

from datasets import load_dataset
v2   = load_dataset('ScaleAI/SWE-bench_Pro', split='test')            # V2, 642 tasks (default config)
hard = load_dataset('ScaleAI/SWE-bench_Pro', 'hard', split='test')    # HARD-51
v1   = load_dataset('ScaleAI/SWE-bench_Pro', 'v1', split='test')      # original 731 tasks

Generate .pred patch files with a harness of your choice. If using SWE-agent, first configure and run the scaffold following the instructions in the SWE-agent git submodule. Consolidate the predictions:

python helper_code/gather_patches.py \
    --directory swe_bench_pro_results/sample1 \
    --prefix sample1 \
    --output sample1_patches.json

Run the v1 evaluator:

python swe_bench_pro_eval.py \
    --raw_sample_path=swe_bench_pro_full.csv \
    --patch_path=<your_patches>.json \
    --output_dir=<output_directory> \
    --scripts_dir=run_scripts \
    --num_workers=100 \
    --dockerhub_username=jefzda

V2 tasks can be executed through Harbor. This example runs the reference patch with 50-way concurrency:

harbor run -p v2/tasks -e modal -n 50 -a oracle

What are this agent's strengths and limitations?

Pros
  • V2 provides 642 validated tasks drawn from 11 repositories, plus the more demanding HARD-51 subset.
  • Every V2 task includes a verifier, reference solution, and public GHCR container image that does not require login to pull.
  • The locked protocol separates offline agent execution from re-grading in a fresh sandbox, creating a defined evaluation boundary.
  • Patch generation is harness-flexible, while documented SWE-agent and mini-swe-agent paths provide concrete starting points.
  • Both v1 and V2 remain available, supporting reproduction of older results alongside adoption of the newer format.
Limitations
  • This is benchmark infrastructure, not a coding assistant that directly handles everyday development work.
  • The full workflow spans Python dependencies, Docker, dataset downloads, patch collection, and evaluation commands.
  • Modal is recommended for v1, while the local Docker alternative is explicitly labeled Beta.
  • V1 and V2 use different task formats, image registries, and evaluation pipelines, adding migration overhead.
  • The news section reports identified leaderboard issues under active remediation, so leaderboard consumers should account for that status.

How does this agent compare with similar options?

SWE-Bench Pro is inspired by SWE-Bench but explicitly targets more challenging, long-horizon software engineering tasks. The repository also presents SWE-agent and mini-swe-agent as patch-generation scaffolds; it reports comparable Sonnet 4.5 results for them, but supplies no broader head-to-head comparison.

Key facts side by side with the most closely related agents.

Agent Source review Form / cost Stars Updated Language Full support on
SWE-Bench Pro This agent 48 · Major gaps CLIFree ★ 569 1mo ago Python —
SWE-bench 48 · Major gaps CLIFree + model costs ★ 6.2k 2mo ago Python —
Multi-SWE-bench 48 · Major gaps CLIFree ★ 364 11mo ago Python —
OSWorld 2.1 Computer-Use Agent Benchmark 70 · Some gaps CLIFree + model costs ★ 428 7d ago Python OpenAI API · Claude API

How does FollowAgents rate this agent?

FollowAgents source review · FARS-2.1
Major gaps
48/ 100 5-point scale 2.4 / 5
Trust 12/29
Reliability 5/14
Adaptability 10/18
Convention 10/18
Effectiveness 7/13
Verifiability 4/8
Why each dimension lost points
Trust12 / 29 · 2.1/5

The README describes the evaluation flow, dataset, and Docker/Modal options, while run_script.sh and test.sh show test execution and log handling, so data flow is partly visible. Permission, external-effect, and rollback controls are limited: the scripts start Redis, install npm dependencies, change configuration, and clean up or restore test files. The README mentions Modal credentials and where they are stored, but the sample shows no credential safeguards, pre-execution confirmation, or general recovery path. Source links and dataset/paper attribution are present, but publisher identity is unknown and maintenance responsibility is unclear.

Reliability5 / 14 · 1.8/5

The README distinguishes the V2 default configuration from the legacy V1 flow and gives reproduction steps, but the sample lacks enough of the main evaluation implementation to verify end-to-end consistency. Dependencies have minimum or open-ended version ranges, with no lockfile shown. The scripts include timeouts and error output, but run_script.sh may emit empty test results after a failed test command, limiting diagnosis. No deduction is made for lack of execution evidence, which is outside this assessment.

Adaptability10 / 18 · 2.8/5

The README defines the task as generating a patch for a given codebase and issue, and describes SWE-agent, other harnesses, vLLM, Modal, and local Docker paths. The repository is clearly positioned as a benchmark and evaluation tool rather than a general-purpose coding agent. Its environment fit is partial because V1 is marked legacy and local Docker is described as Beta.

Convention10 / 18 · 2.8/5

The README is organized into overview, installation, images, usage, and reproduction sections, with command examples, parameter notes, and V1/V2 distinctions. The MIT license text is complete. Release notes identify the V2 release, V1 configuration, and several updates, but the sample does not state a maintainer, support period, or dependency policy. It discloses some limitations, including Beta status, local execution, and leaderboard issues, but the limitations guidance is incomplete.

Effectiveness7 / 13 · 2.7/5

The repository provides task data, container environments, patch collection, and evaluation instructions that support comparing software engineering agents and reproducing a benchmark workflow. The output format is illustrated, but the sampled parser only handles JSON objects matched in stdout and does not use its stderr input; the run script may also emit empty results after test failure. These details reduce output reliability. Modal, Docker, and evaluation across many tasks impose meaningful setup and compute costs, which the sample does not quantify or mitigate with a lightweight path.

Verifiability4 / 8 · 2.5/5

The README links its main claims to a paper, dataset, public leaderboard, and V2 documentation, and gives task counts and configuration examples. The sampled code shows the parser and runner, but does not independently substantiate claims such as 642/642 or reproducing Sonnet 4.5 results. The README does not clearly distinguish reported results from facts directly verifiable in the sample, so corroboration and fact/inference separation receive limited scores.

Risks and how to mitigate them
  • This is a benchmark and evaluation repository, not a general-purpose agent product; reproduction depends on Docker/Modal, external data, and a model harness.
  • The test scripts modify the workspace and prepare services; run them only in a dedicated container or sandbox following the documented workflow.
  • The sampled parser and runner may produce incomplete or empty test results on failure paths. This was a static review and no execution was performed.
Evidence confidence: Low Reviewed Nov 08, 2026 Reviewed revision 66f92766bba6
See the full review method →
Maintain this project? Add the FARS badge to your README

The badge shows the current review score and verdict and links back to this page. It refreshes about once an hour after a re-review.

FARS

[![FARS review](https://followagents.com/badge/swe-bench-pro-os.svg)](https://followagents.com/en/agents/swe-bench-pro-os)
<a href="https://followagents.com/en/agents/swe-bench-pro-os"><img src="https://followagents.com/badge/swe-bench-pro-os.svg" alt="FARS review"></a>

Recent releases

Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.

  1. v2.0.0 SWE-bench Pro V2 (v2.0.0)
    Release notes excerpt
    SWE-bench Pro V2 (v2.0.0) The validated public split of SWE-bench Pro: 642 tasks across 11 repositories, shipped as Harbor task directories under v2/tasks/ with verifier, reference solution and public container image, plus the HARD-51 subset (v2/hard51_ids.txt) and the locked-protocol tooling (v2/tooling/). See v2/README.md for the run recipe. Highlights - 731 -> 642 tasks: 89 dropped after review; 529 instructions rewritten and validated blind by a second engineer; 214 test patches and 38 gold patches revised; test lists regenerated against repaired parsers. …
    View on GitHub ↗

FAQ

Is this a coding agent I can use directly on my repository?
No. It supplies benchmark tasks, containers, verifiers, and evaluation workflows. SWE-agent, mini-swe-agent, or another harness must generate the patches.
How do V2 and v1 differ?
V2 is the default configuration with 642 validated Harbor-format tasks, HARD-51, and locked-protocol tooling. V1 retains the original 731 tasks and uses swe_bench_pro_eval.py, run scripts, and Docker Hub images.
Is Modal mandatory?
No. Modal is recommended for the v1 workflow, but a local Docker Beta path is available through --use_local_docker. The documented V2 example runs Harbor with Modal.
Do the container images require registry credentials?
The V2 GHCR images are public and require no login for docker pull. V1 uses prebuilt images from the jefzda/sweap-images Docker Hub repository.
Can I evaluate my own model or harness?
Yes. The workflow accepts patches generated by a harness of your choice. The documented SWE-agent configuration also supports different models and vllm for local models, though adaptation depends on the selected harness.
Is SWE-Bench Pro free?
Yes. SWE-Bench Pro is free to use and does not need a paid service.
Is SWE-Bench Pro open source?
Yes. The source code is on GitHub (scaleapi/SWE-bench_Pro-os) under the MIT licence.
Is SWE-Bench Pro safe to use?
FollowAgents' FARS review (Nov 08, 2026) scored SWE-Bench Pro 12/29 on trust and safety. None of the eight safety controls the review checks for was missing outright.
Is SWE-Bench Pro still maintained?
Development has slowed: the latest commit on GitHub is from 2026-09-22, within the last six months but not the last 30 days. The latest release is v2.0.0 (2026-09-22).
Who makes SWE-Bench Pro?
SWE-Bench Pro is published on GitHub by the account scaleapi. FollowAgents has not verified which person or company is behind that account.
Was this page helpful?
View on GitHub ↗ Install ↓

Compare agents like this one

The same FARS review applied across the shortlist this agent qualifies for.

Related concepts