SWE-bench
Evaluate model-generated patches against real GitHub software issues.
Per-dimension scores and reasoning
This repository is a benchmark tool, not an agent with permission management, user confirmation, data flow transparency, or sensitive data handling. No evidence of malicious behavior, but also no security mechanisms. Hence all trust criteria score 0.
Self-consistency: Installation instructions in README and pyproject.toml are consistent, and test files match examples in docs. Dependency availability: Dependencies are listed but not pinned, which may affect reproducibility. Failure messages: Tests only check return codes, not error message quality.
Target audience is clearly researchers and developers, with clear usage scenarios. Capability boundaries are stated in README (e.g., resource requirements, arm64 support). Trigger precision: CLI arguments are clear. Environment fit: Supports Docker and Modal, but arm64 is experimental.
Information architecture is clear with README, docs links, and directory structure. Install notes are detailed. Naming is stable with package name swebench. Examples and FAQ: README provides example commands, but no FAQ. Known limitations: Mentioned in README (resource requirements, arm64). License: MIT. Versioning/changelog: Not provided. Maintenance responsibility: Contact persons and contribution guidelines exist.
Output usability: Evaluation results are stored in JSON format, easy to use. Marginal value: As a benchmark tool, it provides standardized evaluation methods. Cost-benefit: Resource-intensive but offers cloud options.
Claim traceability: README cites papers and datasets. Cross-source corroboration: Multiple related project links. Fact-inference separation: README distinguishes facts and recommendations.
- This repository is a benchmark tool, not an agent product, so trust-related criteria are not applicable.
- Dependencies are not pinned, which may affect reproducibility.
- No changelog is provided.
What does this agent do, and when should you use it?
SWE-bench is a benchmark for testing large language models on real-world GitHub software issues: a model receives a codebase and an issue, then produces a patch. The repository contains code and data for SWE-bench, Lite, Verified, and Multimodal, plus a Docker-based evaluation harness. Evaluations run through `swebench.harness.run_evaluation` and write image-build logs, evaluation logs, and final result files locally. The project also documents inference with existing local and API models and a procedure for collecting tasks from other repositories. Local adoption has a substantial infrastructure footprint: the README recommends x86_64, 120GB free storage, 16GB RAM, and 8 CPU cores.
Users can load the test split with datasets.load_dataset('princeton-nlp/SWE-bench', split='test'), or choose a dataset by name in the evaluator. python -m swebench.harness.run_evaluation reads predicted patches or the gold patches and evaluates them in Docker; --dataset_name, --predictions_path, --max_workers, --instance_ids, and --run_id configure the run. It writes image-build logs to logs/build_images, evaluation logs to logs/run_evaluation, and final results to evaluation_results in the current directory. The documented inference path gives an existing local or API model a repository and issue so it can generate a fix.
- A benchmark researcher comparing model patch predictions on the same set of real GitHub issues can run the harness against each prediction file.
- A model team validating its evaluation environment before a larger run can execute the supplied gold patch against one named instance.
- An engineering research group that needs a smaller regression set can evaluate predictions on
princeton-nlp/SWE-bench_Lite. - A researcher with an existing local or API model can use the documented inference workflow to generate repair candidates from a repository and issue.
- A team creating software-repair tasks from its own repositories can use the collection procedure, while accounting for the documented pause in support for instance-creation questions.
- A team that cannot run evaluations locally can use the documented Modal option or sb-cli's AWS-based automatic evaluation path.
What are this agent's strengths and limitations?
- It evaluates patches against real GitHub issues and codebases, rather than judging text-only responses.
- Its Docker-based harness establishes a reproducible execution boundary and produces separate build logs, evaluation logs, and result files.
- The repository brings together the standard, Lite, Verified, and Multimodal datasets; Verified is described as 500 problems confirmed solvable by real software engineers.
- It supports evaluation of supplied predictions and documents inference with both local and API models.
- Local evaluation is resource-intensive: the documented recommendation is at least 120GB free storage, 16GB RAM, and 8 CPU cores.
- Docker is required for reproducible evaluation; default images are built for Linux and ARM support is experimental.
- Multimodal test-split evaluation is private and must be submitted through sb-cli, so it is not available as a local test evaluation in the documented workflow.
- Although task collection is included, the project says support for questions about creating SWE-bench instances is temporarily paused.
How do you install or deploy this agent?
Install Docker first, then build from source:
git clone [email protected]:princeton-nlp/SWE-bench.git
cd SWE-bench
pip install -e .The README specifies Python 3.8+. No application credentials are documented; downloading datasets and pulling Docker images require network access. Verify the installation with:
python -m swebench.harness.run_evaluation --predictions_path gold --max_workers 1 --instance_ids sympy__sympy-20590 --run_id validate-goldOn Apple Silicon Macs or other ARM systems, add --namespace '' so evaluation images are built locally.
How do you use this agent?
Evaluate a patch-prediction file on SWE-bench Lite:
python -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Lite --predictions_path <path_to_predictions> --max_workers <num_workers> --run_id <run_id>Replace <path_to_predictions> with the prediction file; use gold to verify gold patches. Add --modal true to run on Modal. Inspect all evaluator arguments with python -m swebench.harness.run_evaluation --help.
How does this agent compare with similar options?
For non-local evaluation, the repository names two execution alternatives: sb-cli for automatic evaluation on AWS and Modal for cloud-based evaluation. It does not provide a feature or cost comparison between them.
FAQ
What machine capacity is recommended for local evaluation?
--max_workers below min(0.75 * os.cpu_count(), 24).Can it run on Apple Silicon or another ARM system?
--namespace '' to build evaluation images locally.Can I evaluate the SWE-bench Multimodal test split locally?
What accounts or credentials are required?
Where are results and troubleshooting logs written?
logs/build_images, evaluation logs go to logs/run_evaluation, and final results go to evaluation_results.