SWE-bench
Evaluate model-generated patches against real GitHub software issues.
- Source repo
- SWE-bench/SWE-bench
- Stars
- ★ 6.2k
- Last updated
- 2mo ago
- License
- MIT
- Primary language
- Python
- FA score
- 45/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changes
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A benchmark researcher comparing model patch predictions on the same set of real GitHub issues can run the harness against each prediction file.
- Not a fit if
- Machines with under 120GB free storage or 16GB RAM
- ARM users who need stable, non-experimental support
- Source review
- 45/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
SWE-bench is a benchmark for testing large language models on real-world GitHub software issues: a model receives a codebase and an issue, then produces a patch. The repository contains code and data for SWE-bench, Lite, Verified, and Multimodal, plus a Docker-based evaluation harness. Evaluations run through swebench.harness.run_evaluation and write image-build logs, evaluation logs, and final result files locally. The project also documents inference with existing local and API models and a procedure for collecting tasks from other repositories. Local adoption has a substantial infrastructure footprint: the README recommends x86_64, 120GB free storage, 16GB RAM, and 8 CPU cores.
Users can load the test split with datasets.load_dataset('princeton-nlp/SWE-bench', split='test'), or choose a dataset by name in the evaluator. python -m swebench.harness.run_evaluation reads predicted patches or the gold patches and evaluates them in Docker; --dataset_name, --predictions_path, --max_workers, --instance_ids, and --run_id configure the run. It writes image-build logs to logs/build_images, evaluation logs to logs/run_evaluation, and final results to evaluation_results in the current directory. The documented inference path gives an existing local or API model a repository and issue so it can generate a fix.
- A benchmark researcher comparing model patch predictions on the same set of real GitHub issues can run the harness against each prediction file.
- A model team validating its evaluation environment before a larger run can execute the supplied gold patch against one named instance.
- An engineering research group that needs a smaller regression set can evaluate predictions on
princeton-nlp/SWE-bench_Lite. - A researcher with an existing local or API model can use the documented inference workflow to generate repair candidates from a repository and issue.
- A team creating software-repair tasks from its own repositories can use the collection procedure, while accounting for the documented pause in support for instance-creation questions.
- A team that cannot run evaluations locally can use the documented Modal option or sb-cli's AWS-based automatic evaluation path.
How do you install or deploy this agent?
Install Docker first, then build from source:
git clone [email protected]:princeton-nlp/SWE-bench.git
cd SWE-bench
pip install -e .The README specifies Python 3.8+. No application credentials are documented; downloading datasets and pulling Docker images require network access. Verify the installation with:
python -m swebench.harness.run_evaluation --predictions_path gold --max_workers 1 --instance_ids sympy__sympy-20590 --run_id validate-goldOn Apple Silicon Macs or other ARM systems, add --namespace '' so evaluation images are built locally.
How do you use this agent?
Evaluate a patch-prediction file on SWE-bench Lite:
python -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Lite --predictions_path <path_to_predictions> --max_workers <num_workers> --run_id <run_id>Replace <path_to_predictions> with the prediction file; use gold to verify gold patches. Add --modal true to run on Modal. Inspect all evaluator arguments with python -m swebench.harness.run_evaluation --help.
What are this agent's strengths and limitations?
- It evaluates patches against real GitHub issues and codebases, rather than judging text-only responses.
- Its Docker-based harness establishes a reproducible execution boundary and produces separate build logs, evaluation logs, and result files.
- The repository brings together the standard, Lite, Verified, and Multimodal datasets; Verified is described as 500 problems confirmed solvable by real software engineers.
- It supports evaluation of supplied predictions and documents inference with both local and API models.
- Local evaluation is resource-intensive: the documented recommendation is at least 120GB free storage, 16GB RAM, and 8 CPU cores.
- Docker is required for reproducible evaluation; default images are built for Linux and ARM support is experimental.
- Multimodal test-split evaluation is private and must be submitted through sb-cli, so it is not available as a local test evaluation in the documented workflow.
- Although task collection is included, the project says support for questions about creating SWE-bench instances is temporarily paused.
How does this agent compare with similar options?
For non-local evaluation, the repository names two execution alternatives: sb-cli for automatic evaluation on AWS and Modal for cloud-based evaluation. It does not provide a feature or cost comparison between them.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| SWE-bench This agent | 45 · Major gaps | CLIFree + model costs | ★ 6.2k | 2mo ago | Python | — |
| SWE-Bench Pro | 55 · Major gaps | CLIFree | ★ 557 | 1mo ago | Python | — |
| Multi-SWE-bench | 46 · Major gaps | CLIFree | ★ 364 | 10mo ago | Python | — |
| mini-swe-agent | 54 · Major gaps | CLIFree + model costs | ★ 9.2k | 2mo ago | Python | — |
How does FollowAgents rate this agent?
Why each dimension lost points
This repository is a benchmark tool, not an agent with permission management, user confirmation, data flow transparency, or sensitive data handling. No evidence of malicious behavior, but also no security mechanisms. Hence all trust criteria score 0.
Self-consistency: Installation instructions in README and pyproject.toml are consistent, and test files match examples in docs. Dependency availability: Dependencies are listed but not pinned, which may affect reproducibility. Failure messages: Tests only check return codes, not error message quality.
Target audience is clearly researchers and developers, with clear usage scenarios. Capability boundaries are stated in README (e.g., resource requirements, arm64 support). Trigger precision: CLI arguments are clear. Environment fit: Supports Docker and Modal, but arm64 is experimental.
Information architecture is clear with README, docs links, and directory structure. Install notes are detailed. Naming is stable with package name swebench. Examples and FAQ: README provides example commands, but no FAQ. Known limitations: Mentioned in README (resource requirements, arm64). License: MIT. Versioning/changelog: Not provided. Maintenance responsibility: Contact persons and contribution guidelines exist.
Output usability: Evaluation results are stored in JSON format, easy to use. Marginal value: As a benchmark tool, it provides standardized evaluation methods. Cost-benefit: Resource-intensive but offers cloud options.
Claim traceability: README cites papers and datasets. Cross-source corroboration: Multiple related project links. Fact-inference separation: README distinguishes facts and recommendations.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- This repository is a benchmark tool, not an agent product, so trust-related criteria are not applicable.
- Dependencies are not pinned, which may affect reproducibility.
- No changelog is provided.
FAQ
What machine capacity is recommended for local evaluation?
--max_workers below min(0.75 * os.cpu_count(), 24).Can it run on Apple Silicon or another ARM system?
--namespace '' to build evaluation images locally.Can I evaluate the SWE-bench Multimodal test split locally?
What accounts or credentials are required?
Where are results and troubleshooting logs written?
logs/build_images, evaluation logs go to logs/run_evaluation, and final results go to evaluation_results.