Multi-SWE-bench
A Docker-based benchmark for validating issue-resolution patches across programming languages.
Per-dimension scores and reasoning
Evidence shows this is a benchmark framework, with no permission management, user confirmation, data flow transparency, or sensitive data handling. No security audit or dependency vulnerability scanning evidence. External effects mainly involve running Docker containers, but no permission control or rollback mechanisms are described. Source attribution is clear to ByteDance Seed team, but publisher is unverified. Therefore, all trust criteria score 0.
Self-consistency: README description aligns with config example, but no code-level verification. Dependency availability: Depends on Docker and Hugging Face datasets, which are publicly available, but no version pinning or image checksums. Failure messages: Config provides log directory and error handling options, but no detailed error message examples. Thus, self-consistency scores 2, dependency availability 2, failure messages 2.
Audience and scenarios: Clearly aimed at researchers and developers for evaluating LLMs on code issue resolution. Capability boundaries: Describes framework functionality but no explicit limitations. Trigger precision: Detailed config parameters but no trigger conditions. Environment fit: Requires Docker, provides installation guide, but no other environment requirements. Thus, each scores 2.
Information architecture: README well-structured with sections for installation, evaluation, config. Install notes: Provides make install and make install-dev. Naming stability: Project name and commands stable. Examples and FAQ: Provides config example, but no FAQ. Known limitations: Not explicitly listed. License: Apache-2.0 with full LICENSE file. Versioning and changelog: News updates but no formal changelog. Maintenance responsibility: Clearly ByteDance Seed team. Thus, information architecture, install notes, naming stability, examples and FAQ, maintenance responsibility score 2, known limitations 1, license 3, versioning and changelog 1.
Output usability: Evaluation generates final_report.json with summary. Marginal value: Fills gap in multilingual benchmarks, research value. Cost-benefit: Requires Docker and compute, but mini version reduces cost. Thus, each scores 2.
Claim traceability: README cites arXiv paper and Hugging Face datasets, but no code-level verification. Cross-source corroboration: Provides multiple external links (arXiv, Hugging Face, Discord), but no independent verification. Fact-inference separation: README distinguishes facts (e.g., dataset size) from inferences (e.g., 'accelerate AGI'), but not explicitly labeled. Thus, each scores 2.
- Publisher identity is unverified; treat as unknown.
- No evidence of security audit or dependency vulnerability scanning.
- Running Docker containers may have external effects, but no permission control or rollback mechanisms are described.
- Known limitations are not explicitly listed.
- No formal changelog is provided.
What does this agent do, and when should you use it?
Multi-SWE-bench evaluates real-world code issue resolution; it is not presented as a patch-generating coding agent. The benchmark covers Java, TypeScript, JavaScript, Go, Rust, C, and C++, with 1,632 expert-curated instances stated for the full release. Its evaluation interface is `python -m multi_swe_bench.harness.run_evaluation --config /path/to/your/config.json`, consuming candidate patch JSONL and dataset JSONL. Docker builds or reuses instance images, while results are written as `final_report.json` plus logs. The project also links Multi-SWE-RL and smaller mini and flash evaluation collections for teams assessing multilingual repair systems.
You prepare a JSONL patch file whose records contain org, repo, number, and fix_patch, then supply dataset JSONL from Multi-SWE-bench or Multi-SWE-RL. Configuration supplies paths such as workdir, output_dir, repo_dir, and log_dir, along with worker settings. multi_swe_bench.harness.run_evaluation runs in evaluation, instance, instance_only, or image mode; missing Docker images are built during evaluation, or may be downloaded in advance with scripts/download_images.sh or scripts/download_images.ps1. The run produces final_report.json in output_dir, including resolved_instances and unresolved_instances, and stores detailed failure reasons in log_dir.
- A repair-system team has candidate
fix_patchoutputs and needs a common acceptance benchmark spanning Java, Go, Rust, C, and C++ tasks. - A researcher wants to compare issue-resolution results from models or repair frameworks in reproducible Docker environments.
- An evaluation engineer needs to run only selected PR IDs or exclude known instances through
specificsandskips. - A software-engineering RL data contributor wants to follow the Multi-SWE-RL community quick-start and contribution demonstration.
- A compute-constrained team wants to use the documented 400-instance mini collection or 300-instance flash collection for faster evaluation.
What are this agent's strengths and limitations?
- It explicitly covers seven languages beyond a Python-centric repair benchmark scope.
- Its JSONL input accepts existing
fix_patchoutputs, so a team can evaluate repair results without adopting a named agent framework. - Docker images can be pre-downloaded or built on demand, with separate concurrency controls for general work, image builds, and instance runs.
- It emits both a
final_report.jsonsummary and per-run logs for investigating unresolved instances and errors.
- Docker is a runtime dependency, and absent images are built during evaluation, adding setup and execution overhead.
- Users must supply candidate patch JSONL and dataset JSONL; no hosted evaluation service is documented.
- Patch application can fail; the README documents a
patch --batch --fuzz=5workaround for cases wheregit applyis problematic. - The documented interface evaluates supplied patches rather than providing an embedded interface to generate them.
How do you install or deploy this agent?
Docker is required. The README does not document API keys or other credentials. Run:
git clone [email protected]:multi-swe-bench/multi-swe-bench.git
cd multi-swe-bench
make installFor development dependencies, run make install-dev. To pre-download images on Linux/macOS, use bash scripts/download_images.sh scripts/images_mini.txt, scripts/images_verified.txt, or scripts/images_rl.txt; Windows has matching scripts/download_images.ps1 commands.
How do you use this agent?
Create a JSON configuration with paths including patch_files, dataset_files, workdir, output_dir, repo_dir, and log_dir; each patch record must include org, repo, number, and fix_patch. Run:
python -m multi_swe_bench.harness.run_evaluation --config /path/to/your/config.jsonThe documented example uses "mode": "evaluation". Read output_dir/final_report.json when it completes. If patch application fails and the workaround fits your case, add the documented fix_patch_run_cmd to use patch --batch in place of git apply.
How does this agent compare with similar options?
The README positions Multi-SWE-bench against Python-centric benchmarks such as SWE-bench and states that it references and builds on the SWE-bench repository.
FAQ
Does it generate a code fix for me?
fix_patch: it takes candidate patches and dataset files, runs the harness, and reports resolution results. No built-in patch-generation interface is documented.What credentials are required to run it?
What if git apply cannot apply a patch?
fix_patch_run_cmd example replaces the default application command with patch --batch --fuzz=5 for /home/test.patch and /home/fix.patch.Can I limit the run or control resource use?
specifics for selected PR IDs, skips to exclude instances, and max_workers, max_workers_build_image, and max_workers_run_instance to control concurrency.