Multi-SWE-bench
A Docker-based benchmark for validating issue-resolution patches across programming languages.
- Source repo
- multi-swe-bench/multi-swe-bench
- Stars
- ★ 364
- Last updated
- 10mo ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 46/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free, no paid service needed
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A repair-system team has candidate
fix_patchoutputs and needs a common acceptance benchmark spanning Java, Go, Rust, C, and C++ tasks. - Not a fit if
- Teams evaluating only Python-centric issue resolution
- Users who cannot run Docker or accept image build overhead
- Teams wanting a hosted evaluation service instead of supplying their own patch and dataset files
- Source review
- 46/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
Multi-SWE-bench evaluates real-world code issue resolution; it is not presented as a patch-generating coding agent. The benchmark covers Java, TypeScript, JavaScript, Go, Rust, C, and C++, with 1,632 expert-curated instances stated for the full release. Its evaluation interface is python -m multi_swe_bench.harness.run_evaluation --config /path/to/your/config.json, consuming candidate patch JSONL and dataset JSONL. Docker builds or reuses instance images, while results are written as final_report.json plus logs. The project also links Multi-SWE-RL and smaller mini and flash evaluation collections for teams assessing multilingual repair systems.
You prepare a JSONL patch file whose records contain org, repo, number, and fix_patch, then supply dataset JSONL from Multi-SWE-bench or Multi-SWE-RL. Configuration supplies paths such as workdir, output_dir, repo_dir, and log_dir, along with worker settings. multi_swe_bench.harness.run_evaluation runs in evaluation, instance, instance_only, or image mode; missing Docker images are built during evaluation, or may be downloaded in advance with scripts/download_images.sh or scripts/download_images.ps1. The run produces final_report.json in output_dir, including resolved_instances and unresolved_instances, and stores detailed failure reasons in log_dir.
- A repair-system team has candidate
fix_patchoutputs and needs a common acceptance benchmark spanning Java, Go, Rust, C, and C++ tasks. - A researcher wants to compare issue-resolution results from models or repair frameworks in reproducible Docker environments.
- An evaluation engineer needs to run only selected PR IDs or exclude known instances through
specificsandskips. - A software-engineering RL data contributor wants to follow the Multi-SWE-RL community quick-start and contribution demonstration.
- A compute-constrained team wants to use the documented 400-instance mini collection or 300-instance flash collection for faster evaluation.
How do you install or deploy this agent?
Docker is required. The README does not document API keys or other credentials. Run:
git clone [email protected]:multi-swe-bench/multi-swe-bench.git
cd multi-swe-bench
make installFor development dependencies, run make install-dev. To pre-download images on Linux/macOS, use bash scripts/download_images.sh scripts/images_mini.txt, scripts/images_verified.txt, or scripts/images_rl.txt; Windows has matching scripts/download_images.ps1 commands.
How do you use this agent?
Create a JSON configuration with paths including patch_files, dataset_files, workdir, output_dir, repo_dir, and log_dir; each patch record must include org, repo, number, and fix_patch. Run:
python -m multi_swe_bench.harness.run_evaluation --config /path/to/your/config.jsonThe documented example uses "mode": "evaluation". Read output_dir/final_report.json when it completes. If patch application fails and the workaround fits your case, add the documented fix_patch_run_cmd to use patch --batch in place of git apply.
What are this agent's strengths and limitations?
- It explicitly covers seven languages beyond a Python-centric repair benchmark scope.
- Its JSONL input accepts existing
fix_patchoutputs, so a team can evaluate repair results without adopting a named agent framework. - Docker images can be pre-downloaded or built on demand, with separate concurrency controls for general work, image builds, and instance runs.
- It emits both a
final_report.jsonsummary and per-run logs for investigating unresolved instances and errors.
- Docker is a runtime dependency, and absent images are built during evaluation, adding setup and execution overhead.
- Users must supply candidate patch JSONL and dataset JSONL; no hosted evaluation service is documented.
- Patch application can fail; the README documents a
patch --batch --fuzz=5workaround for cases wheregit applyis problematic. - The documented interface evaluates supplied patches rather than providing an embedded interface to generate them.
How does this agent compare with similar options?
The README positions Multi-SWE-bench against Python-centric benchmarks such as SWE-bench and states that it references and builds on the SWE-bench repository.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Multi-SWE-bench This agent | 46 · Major gaps | CLIFree | ★ 364 | 10mo ago | Python | — |
| SWE-Bench Pro | 55 · Major gaps | CLIFree | ★ 557 | 1mo ago | Python | — |
| SWE-bench | 45 · Major gaps | CLIFree + model costs | ★ 6.1k | 2mo ago | Python | — |
| HUD | 63 · Some gaps | CLIFree + model costs | ★ 323 | 1d ago | Python | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows this is a benchmark framework, with no permission management, user confirmation, data flow transparency, or sensitive data handling. No security audit or dependency vulnerability scanning evidence. External effects mainly involve running Docker containers, but no permission control or rollback mechanisms are described. Source attribution is clear to ByteDance Seed team, but publisher is unverified. Therefore, all trust criteria score 0.
Self-consistency: README description aligns with config example, but no code-level verification. Dependency availability: Depends on Docker and Hugging Face datasets, which are publicly available, but no version pinning or image checksums. Failure messages: Config provides log directory and error handling options, but no detailed error message examples. Thus, self-consistency scores 2, dependency availability 2, failure messages 2.
Audience and scenarios: Clearly aimed at researchers and developers for evaluating LLMs on code issue resolution. Capability boundaries: Describes framework functionality but no explicit limitations. Trigger precision: Detailed config parameters but no trigger conditions. Environment fit: Requires Docker, provides installation guide, but no other environment requirements. Thus, each scores 2.
Information architecture: README well-structured with sections for installation, evaluation, config. Install notes: Provides make install and make install-dev. Naming stability: Project name and commands stable. Examples and FAQ: Provides config example, but no FAQ. Known limitations: Not explicitly listed. License: Apache-2.0 with full LICENSE file. Versioning and changelog: News updates but no formal changelog. Maintenance responsibility: Clearly ByteDance Seed team. Thus, information architecture, install notes, naming stability, examples and FAQ, maintenance responsibility score 2, known limitations 1, license 3, versioning and changelog 1.
Output usability: Evaluation generates final_report.json with summary. Marginal value: Fills gap in multilingual benchmarks, research value. Cost-benefit: Requires Docker and compute, but mini version reduces cost. Thus, each scores 2.
Claim traceability: README cites arXiv paper and Hugging Face datasets, but no code-level verification. Cross-source corroboration: Provides multiple external links (arXiv, Hugging Face, Discord), but no independent verification. Fact-inference separation: README distinguishes facts (e.g., dataset size) from inferences (e.g., 'accelerate AGI'), but not explicitly labeled. Thus, each scores 2.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- Publisher identity is unverified; treat as unknown.
- No evidence of security audit or dependency vulnerability scanning.
- Running Docker containers may have external effects, but no permission control or rollback mechanisms are described.
- Known limitations are not explicitly listed.
- No formal changelog is provided.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v1.0.0 Multi-SWE-bench v1.0.0 – Benchmark Ready
Release notes excerpt
This is the first official release of Multi-SWE-bench. ✅ Key Features in v1.0.0: - Fully functional support for running Multi-SWE-bench - Compatible integration with Multi-SWE-RL - Initial stable API and reproducible environment setup Full Changelog: https://github.com/multi-swe-bench/multi-swe-bench/commits/v1.0.0 This version serves as a baseline for future updates and benchmarking extensions.View on GitHub ↗
FAQ
Does it generate a code fix for me?
fix_patch: it takes candidate patches and dataset files, runs the harness, and reports resolution results. No built-in patch-generation interface is documented.What credentials are required to run it?
What if git apply cannot apply a patch?
fix_patch_run_cmd example replaces the default application command with patch --batch --fuzz=5 for /home/test.patch and /home/fix.patch.Can I limit the run or control resource use?
specifics for selected PR IDs, skips to exclude instances, and max_workers, max_workers_build_image, and max_workers_run_instance to control concurrency.