SWE-Bench Pro
Benchmark models and coding agents on long-horizon software engineering tasks.
- Source repo
- scaleapi/SWE-bench_Pro-os
- Stars
- ★ 569
- Last updated
- 1mo ago
- License
- MIT
- Primary language
- Python
- FA score
- 48/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free, no paid service needed
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- Model research teams comparing how different models or coding harnesses perform on long-running, repository-level fixes.
- Not a fit if
- Developers seeking a ready-to-use coding assistant
- Teams unable to run containerized evaluations
- Users focused only on short, isolated coding problems
- Maintenance
- Commits in the last 6 monthslast commit 2026-09-22latest release v2.0.0 (2026-09-22)+1.4 ★/day over 7 days
- Source review
- 48/100 · Major gaps
What does this agent do, and when should you use it?
SWE-Bench Pro is a benchmark, dataset, and evaluation toolkit for long-horizon software engineering rather than an end-user coding assistant. Each task gives a model a codebase and an issue and asks it to produce a resolving patch. V2 contains 642 validated tasks across 11 repositories, the HARD-51 subset, and self-contained Harbor task directories with verifiers, reference solutions, and public container images. The repository also preserves the original v1 pipeline, including swe_bench_pro_eval.py, run scripts, Dockerfiles, and Docker Hub images. Users can load the data from Hugging Face, generate .pred files with their chosen harness or the SWE-agent submodule, consolidate them into JSON, and run containerized evaluations.
It reads tasks from the Hugging Face ScaleAI/SWE-bench_Pro dataset and supplies the codebase-and-issue workload for a selected model or harness to solve. The SWE-agent submodule provides one patch-generation scaffold, while other harnesses may also be used; generated patches are stored as per-instance .pred files. helper_code/gather_patches.py consolidates those files into a JSON array containing instance_id, patch, and prefix. For v1, swe_bench_pro_eval.py evaluates that JSON against task samples and run_scripts using container images and configurable parallel workers. V2 instead packages each case as a Harbor directory under v2/tasks/, together with its verifier, reference solution, and GHCR image. V2 also includes HARD-51 and locked-protocol tooling for an offline agent phase followed by re-grading in a fresh sandbox.
- Model research teams comparing how different models or coding harnesses perform on long-running, repository-level fixes.
- Agent developers who need executable, containerized patch verification instead of judging textual answers alone.
- Researchers measuring performance separately on the full V2 set, HARD-51, or the original v1 dataset.
- Leaderboard participants reproducing SWE-agent or mini-swe-agent runs and preparing consolidated prediction files.
- Evaluation-infrastructure engineers studying a locked protocol with offline execution and fresh-sandbox re-grading.
Key setup facts
Pulled from the install and usage notes below. The project README has the current values.
- Install / run
pip install -r requirements.txt
- Config files
~/.modal.tomlsample1_patches.json- Install notes cover
- Docker
How do you install or deploy this agent?
Install the repository's declared Python dependencies:
pip install -r requirements.txtInstall Docker for reproducible containerized evaluation. The v1 workflow recommends configuring Modal:
modal setup # Follow the prompts to generate your tokenThen verify that ~/.modal.toml contains:
token_id = <token id>
token_secret = <token secret>
active = trueFor the v1 local Docker Beta path, no additional Modal setup is required; add --use_local_docker when running evaluations. V2 uses Harbor for evaluation. The README shows a harbor run invocation but does not provide a Harbor installation command.
How do you use this agent?
Load the default V2 test split, HARD-51, or the original v1 data:
from datasets import load_dataset
v2 = load_dataset('ScaleAI/SWE-bench_Pro', split='test') # V2, 642 tasks (default config)
hard = load_dataset('ScaleAI/SWE-bench_Pro', 'hard', split='test') # HARD-51
v1 = load_dataset('ScaleAI/SWE-bench_Pro', 'v1', split='test') # original 731 tasksGenerate .pred patch files with a harness of your choice. If using SWE-agent, first configure and run the scaffold following the instructions in the SWE-agent git submodule. Consolidate the predictions:
python helper_code/gather_patches.py \
--directory swe_bench_pro_results/sample1 \
--prefix sample1 \
--output sample1_patches.jsonRun the v1 evaluator:
python swe_bench_pro_eval.py \
--raw_sample_path=swe_bench_pro_full.csv \
--patch_path=<your_patches>.json \
--output_dir=<output_directory> \
--scripts_dir=run_scripts \
--num_workers=100 \
--dockerhub_username=jefzdaV2 tasks can be executed through Harbor. This example runs the reference patch with 50-way concurrency:
harbor run -p v2/tasks -e modal -n 50 -a oracleWhat are this agent's strengths and limitations?
- V2 provides 642 validated tasks drawn from 11 repositories, plus the more demanding HARD-51 subset.
- Every V2 task includes a verifier, reference solution, and public GHCR container image that does not require login to pull.
- The locked protocol separates offline agent execution from re-grading in a fresh sandbox, creating a defined evaluation boundary.
- Patch generation is harness-flexible, while documented SWE-agent and mini-swe-agent paths provide concrete starting points.
- Both v1 and V2 remain available, supporting reproduction of older results alongside adoption of the newer format.
- This is benchmark infrastructure, not a coding assistant that directly handles everyday development work.
- The full workflow spans Python dependencies, Docker, dataset downloads, patch collection, and evaluation commands.
- Modal is recommended for v1, while the local Docker alternative is explicitly labeled Beta.
- V1 and V2 use different task formats, image registries, and evaluation pipelines, adding migration overhead.
- The news section reports identified leaderboard issues under active remediation, so leaderboard consumers should account for that status.
How does this agent compare with similar options?
SWE-Bench Pro is inspired by SWE-Bench but explicitly targets more challenging, long-horizon software engineering tasks. The repository also presents SWE-agent and mini-swe-agent as patch-generation scaffolds; it reports comparable Sonnet 4.5 results for them, but supplies no broader head-to-head comparison.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| SWE-Bench Pro This agent | 48 · Major gaps | CLIFree | ★ 569 | 1mo ago | Python | — |
| SWE-bench | 48 · Major gaps | CLIFree + model costs | ★ 6.2k | 2mo ago | Python | — |
| Multi-SWE-bench | 48 · Major gaps | CLIFree | ★ 364 | 11mo ago | Python | — |
| OSWorld 2.1 Computer-Use Agent Benchmark | 70 · Some gaps | CLIFree + model costs | ★ 428 | 7d ago | Python | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
The README describes the evaluation flow, dataset, and Docker/Modal options, while run_script.sh and test.sh show test execution and log handling, so data flow is partly visible. Permission, external-effect, and rollback controls are limited: the scripts start Redis, install npm dependencies, change configuration, and clean up or restore test files. The README mentions Modal credentials and where they are stored, but the sample shows no credential safeguards, pre-execution confirmation, or general recovery path. Source links and dataset/paper attribution are present, but publisher identity is unknown and maintenance responsibility is unclear.
The README distinguishes the V2 default configuration from the legacy V1 flow and gives reproduction steps, but the sample lacks enough of the main evaluation implementation to verify end-to-end consistency. Dependencies have minimum or open-ended version ranges, with no lockfile shown. The scripts include timeouts and error output, but run_script.sh may emit empty test results after a failed test command, limiting diagnosis. No deduction is made for lack of execution evidence, which is outside this assessment.
The README defines the task as generating a patch for a given codebase and issue, and describes SWE-agent, other harnesses, vLLM, Modal, and local Docker paths. The repository is clearly positioned as a benchmark and evaluation tool rather than a general-purpose coding agent. Its environment fit is partial because V1 is marked legacy and local Docker is described as Beta.
The README is organized into overview, installation, images, usage, and reproduction sections, with command examples, parameter notes, and V1/V2 distinctions. The MIT license text is complete. Release notes identify the V2 release, V1 configuration, and several updates, but the sample does not state a maintainer, support period, or dependency policy. It discloses some limitations, including Beta status, local execution, and leaderboard issues, but the limitations guidance is incomplete.
The repository provides task data, container environments, patch collection, and evaluation instructions that support comparing software engineering agents and reproducing a benchmark workflow. The output format is illustrated, but the sampled parser only handles JSON objects matched in stdout and does not use its stderr input; the run script may also emit empty results after test failure. These details reduce output reliability. Modal, Docker, and evaluation across many tasks impose meaningful setup and compute costs, which the sample does not quantify or mitigate with a lightweight path.
The README links its main claims to a paper, dataset, public leaderboard, and V2 documentation, and gives task counts and configuration examples. The sampled code shows the parser and runner, but does not independently substantiate claims such as 642/642 or reproducing Sonnet 4.5 results. The README does not clearly distinguish reported results from facts directly verifiable in the sample, so corroboration and fact/inference separation receive limited scores.
- This is a benchmark and evaluation repository, not a general-purpose agent product; reproduction depends on Docker/Modal, external data, and a model harness.
- The test scripts modify the workspace and prepare services; run them only in a dedicated container or sandbox following the documented workflow.
- The sampled parser and runner may produce incomplete or empty test results on failure paths. This was a static review and no execution was performed.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v2.0.0 SWE-bench Pro V2 (v2.0.0)
Release notes excerpt
SWE-bench Pro V2 (v2.0.0) The validated public split of SWE-bench Pro: 642 tasks across 11 repositories, shipped as Harbor task directories under v2/tasks/ with verifier, reference solution and public container image, plus the HARD-51 subset (v2/hard51_ids.txt) and the locked-protocol tooling (v2/tooling/). See v2/README.md for the run recipe. Highlights - 731 -> 642 tasks: 89 dropped after review; 529 instructions rewritten and validated blind by a second engineer; 214 test patches and 38 gold patches revised; test lists regenerated against repaired parsers. …View on GitHub ↗
FAQ
Is this a coding agent I can use directly on my repository?
How do V2 and v1 differ?
swe_bench_pro_eval.py, run scripts, and Docker Hub images.Is Modal mandatory?
--use_local_docker. The documented V2 example runs Harbor with Modal.Do the container images require registry credentials?
docker pull. V1 uses prebuilt images from the jefzda/sweap-images Docker Hub repository.Can I evaluate my own model or harness?
vllm for local models, though adaptation depends on the selected harness.Is SWE-Bench Pro free?
Is SWE-Bench Pro open source?
Is SWE-Bench Pro safe to use?
Is SWE-Bench Pro still maintained?
Who makes SWE-Bench Pro?
Thanks — we use this to improve the page.