MARTI: Multi-Agent Reinforced Training & Inference for LLMs
Train LLM-based multi-agent systems with reinforcement learning and tree-search-augmented reasoning across debate, chain-of-agents, and mixture-of-agents workflows.
- Source repo
- TsinghuaC3I/MARTI
- Stars
- ★ 560
- Last updated
- 1mo ago
- License
- MIT
- Primary language
- Python
- FA score
- 36/100 · Major gaps
At a glance
- Works with
- Universal · cross-platform
- You'll need
- Typical use
- Researchers running reproducible RL training of LLMs in multi-agent debate or mixture-of-agents setups with GRPO/PPO
- Main limitation
- High hardware barrier: ~8×80G GPUs for single-agent and ~8×80G per agent for multi-agent; the paper's experiments used 3 nodes of 8×H200 with one node per agent
- Source review
- 36/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
MARTI is an open-source framework from Tsinghua University and Shanghai AI Lab for training LLM-based Multi-Agent Systems (MAS) with reinforcement learning, accepted at ICLR 2026. It follows a principle of centralized multi-agent interaction with distributed policy training, built from three core modules — Multi-Agent World, Centralized Rewarding, and Single Agent Trainer — on top of OpenRLHF, with vLLM v1 and Hybrid Engine support. The latest MARTI-v2 release adds tree-search-augmented RL (MARS²) for complex reasoning tasks like code generation, featuring asynchronous multi-agent MCTS training plus GSPO loss, TIS correction, dynamic data filtering, and an overlong buffer supporting sequences up to 32K tokens and heterogeneous multi-agent training (e.g., Qwen3-8B + AreaL-boba-2-8B). Built-in graph-based workflows include debate, chain-of-agents, and mixture-of-agents, with experimental third-party integration for AutoGen and CAMEL. On the LCB code generation benchmark, multi-agent MCTS improves Qwen3-8B by 8.0% over the base model and 4.4% over Vanilla GRPO. Note that this is a research training framework, not a turnkey inference service.
MARTI reads configuration and training scripts (bash scripts under examples/) and runs multi-agent RL training through the OpenRLHF infrastructure with the vLLM engine. The centralized Multi-Agent World module handles inter-agent interaction and reward allocation; Centralized Rewarding implements credit assignment and reward shaping; Single Agent Trainer performs distributed policy training per agent. Supported RL algorithms include PPO, GRPO, REINFORCE++, and TTRL, with workflows such as Multi-Agent Debate, Chain-of-Agents, Mixture-of-Agents, and Review-RL. The MARTI-v2 MARS² module performs asynchronous multi-agent MCTS with adaptive node expansion and refinement to explore solution spaces and produce high-quality reasoning trajectories for policy updates, using GSPO sequence-level optimization, TIS correction for vLLM sampling mismatch, per-agent dynamic sample filtering, and an overlong token penalty buffer. The output is model weights optimized via multi-agent RL (e.g., Qwen3-8B/14B, DeepCoder-14B).
- Researchers running reproducible RL training of LLMs in multi-agent debate or mixture-of-agents setups with GRPO/PPO
- ML engineers training code generation models who want MCTS tree search instead of naive rollouts to lift Pass@1 on LCB-style benchmarks
- Teams jointly training heterogeneous models (e.g., Qwen3-8B + AreaL-boba-2-8B), each agent with its own role, training strategy, and sample filtering
- Users with existing AutoGen or CAMEL workflows who want to plug their orchestration into an RL training pipeline (experimental support)
- Research scenarios requiring asynchronous tool use and multi-turn agentic RL, such as async code or math RL
How do you install or deploy this agent?
Installation steps (from the repository):
- Clone and install:
git clone https://github.com/TsinghuaC3I/MARTI.git
cd MARTI
pip install -r requirements.txt- Follow the repository's setup instructions for dependencies, including OpenRLHF, Ray, and vLLM.
Hardware threshold: single-agent training requires roughly 8×80G GPUs; multi-agent training requires approximately 8×80G GPUs per agent. No Docker image or hosted service is provided; you must supply your own GPU cluster.
How do you use this agent?
Set path variables in the scripts and run the relevant example scripts:
ROOT_DIR="/path/to/MARTI"
MODEL_DIR="/path/to/models"
# Single-agent MCTS training (MARS²)
bash examples/mars2/run_train_single_mcts.sh
# Multi-agent MCTS training
bash examples/mars2/run_train_multi_mcts.sh
# Async multi-turn code RL (single-agent)
bash examples/single-agent/run_train_code_async.sh
# Multi-agent debate
bash examples/multi-agent/run_train_mad.sh
# Chain-of-agents (MathChat)
bash examples/multi-agent/run_train_mathchat.sh
# Review-RL
bash examples/reviewrl/run_train_reviewrl_async.shDocumentation lives in docs/ (Overview, Workflows Integration, Reward and Training, Experiments).
What are this agent's strengths and limitations?
- Backed by an ICLR 2026 paper and an arXiv technical report (2602.07848), giving peer-reviewed credibility
- Unifies multi-agent training and single-agent RL in one framework with built-in credit assignment and reward shaping, avoiding hand-assembled pipelines
- MARS² provides asynchronous multi-agent MCTS training; on the LCB benchmark multi-agent training improves over Vanilla GRPO by up to 4.4%, with support for 32K-token sequences and heterogeneous agents
- Supports multiple RL algorithms (PPO, GRPO, REINFORCE++, TTRL) plus experimental compatibility with existing AutoGen/CAMEL orchestration
- High hardware barrier: ~8×80G GPUs for single-agent and ~8×80G per agent for multi-agent; the paper's experiments used 3 nodes of 8×H200 with one node per agent
- Research-grade framework; the AutoGen/CAMEL integrations are explicitly experimental and production readiness is undocumented
- No hosted service or Docker deployment; you must maintain the OpenRLHF/Ray/vLLM dependency stack yourself
- Published evidence covers specific model families (Qwen3, AreaL-boba, DeepCoder; DeepSeek-R1, Gemma, Llama appear in topics); effectiveness on other models is not publicly validated
How does this agent compare with similar options?
MARTI is explicitly built on the single-agent RL frameworks OpenRLHF and verl — it extends rather than replaces them: OpenRLHF/verl target single-agent policy training, while MARTI adds centralized multi-agent interaction, reward allocation, and multi-agent tree search (MARS²). The README names no other competing multi-agent RL framework.
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| MARTI: Multi-Agent Reinforced Training & Inference for LLMs This agent | 36 · Major gaps | ★ 560 | 1mo ago | Python | — |
| AgentsMeetRL — Awesome List of Agentic Reinforcement Learning | 29 · Major gaps | ★ 1.8k | 8d ago | HTML | — |
| Hands-On Modern RL | 30 · Major gaps | ★ 4.4k | 20d ago | Python | — |
| rLLM: Reinforcement Learning Framework for LLM Agents | 0 · Major gaps | ★ 5.8k | 11d ago | Python | OpenAI API |
How does FollowAgents rate this agent?
Why each dimension lost points
This is a research-oriented multi-agent RL training framework, not a deployed Agent. Source attribution is strong: acknowledgements to OpenRLHF/vLLM/Ray, contributor lists, and contact emails, earning 3. Least privilege: only requirements and script names are visible; no documented permission design, though no credential theft or covert exfiltration was found; wandb telemetry is present but its data reporting is undocumented — deducted for undocumented data flow. No evidence at all on sensitive data handling or rollback. Dependency pinning is partial (some pins, vllm uses an open '>=' constraint) with no vulnerability notes, scoring 1.
Install notes are two commands plus 'follow setup instructions' for OpenRLHF/Ray/vLLM, thin support for dependency availability (1). README is internally consistent in structure, but many MARTI-v2 claims (GSPO, TIS, overlong buffer) lack code-level substantiation in the provided files (1). No failure-message or error-handling documentation exists in the provided evidence (0).
Audience (RL researchers) and scenarios (code/math training) are clearly stated (2). Trigger precision is not applicable to a training framework (0). Capability boundaries are partly stated (8×80G GPU minimum) but third-party integrations are only loosely marked 'experimental' (1). Environment fit gives rough hardware needs only, no OS/CUDA details (1).
Information architecture is good: full TOC, segmented docs, figures (2). Install notes are very thin (1). Naming is stable across MARTI/MARS². Examples are mere script-name pointers with no FAQ (1). Known limitations are entirely absent (0). MIT license is complete, though the README footer 'All rights reserved' mildly conflicts with MIT terms (2). No version numbers or CHANGELOG; News items substitute (1). Maintenance responsibility is clear: named institutions, contacts, update timeline (2).
Output usability: results are shown as images without reproducible tables or numbers (1). Marginal value: unifying tree-search RL with multi-agent training on OpenRLHF infrastructure is a genuine differentiator (2). Cost-benefit: minimum 8×80G GPUs and 3-node H200 clusters for multi-agent runs make it very costly relative to benefit for ordinary users (1).
Benchmark claims (4.6%/8.0% gains) point to an arXiv report and OpenReview, but are presented only as screenshots with no verifiable eval scripts or data tables in-repo (1). Cross-source corroboration exists as external links (ICLR acceptance, third-party adopters) but is unverifiable in static review (1). Fact/inference separation is mediocre: dates and hardware requirements are factual, but some 'Key Features' read as promotional assertions (1).
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Caution: several performance claims are presented only as images without in-repo evaluation scripts or data tables; verify against the paper and code before adoption.
- Install documentation is thin: OpenRLHF, vLLM and other core dependencies must be installed separately with complex version coupling; pin versions in an isolated environment to avoid compatibility risk.
- wandb is a default dependency; confirm the scope of experiment data reporting before use to avoid unintended data egress.
- Hardware cost is very high (8×80G GPUs minimum; 3 H200 nodes for multi-agent); assess cost-benefit for smaller teams first.
- The README footer 'All rights reserved' is not fully consistent with the MIT license; confirm licensing intent with maintainers before commercial use.
- Third-party integrations (AutoGen, CAMEL) are marked experimental and need extra validation before production use.