MARTI: Multi-Agent Reinforced Training & Inference for LLMs
Train LLM-based multi-agent systems with reinforcement learning and tree-search-augmented reasoning across debate, chain-of-agents, and mixture-of-agents workflows.
- Source repo
- TsinghuaC3I/MARTI
- Stars
- ★ 574
- Last updated
- 2mo ago
- License
- MIT
- Primary language
- Python
- FA score
- 40/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free, no paid service needed
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- Researchers running reproducible RL training of LLMs in multi-agent debate or mixture-of-agents setups with GRPO/PPO
- Not a fit if
- Individual developers without ~8×80G GPU clusters
- Teams wanting a turnkey inference service or Docker deployment
- Users unwilling to maintain the OpenRLHF/Ray/vLLM stack themselves
- Source review
- 40/100 · Major gaps 3 safety controls not found
What does this agent do, and when should you use it?
MARTI is an open-source framework from Tsinghua University and Shanghai AI Lab for training LLM-based Multi-Agent Systems (MAS) with reinforcement learning, accepted at ICLR 2026. It follows a principle of centralized multi-agent interaction with distributed policy training, built from three core modules — Multi-Agent World, Centralized Rewarding, and Single Agent Trainer — on top of OpenRLHF, with vLLM v1 and Hybrid Engine support. The latest MARTI-v2 release adds tree-search-augmented RL (MARS²) for complex reasoning tasks like code generation, featuring asynchronous multi-agent MCTS training plus GSPO loss, TIS correction, dynamic data filtering, and an overlong buffer supporting sequences up to 32K tokens and heterogeneous multi-agent training (e.g., Qwen3-8B + AreaL-boba-2-8B). Built-in graph-based workflows include debate, chain-of-agents, and mixture-of-agents, with experimental third-party integration for AutoGen and CAMEL. On the LCB code generation benchmark, multi-agent MCTS improves Qwen3-8B by 8.0% over the base model and 4.4% over Vanilla GRPO. Note that this is a research training framework, not a turnkey inference service.
MARTI reads configuration and training scripts (bash scripts under examples/) and runs multi-agent RL training through the OpenRLHF infrastructure with the vLLM engine. The centralized Multi-Agent World module handles inter-agent interaction and reward allocation; Centralized Rewarding implements credit assignment and reward shaping; Single Agent Trainer performs distributed policy training per agent. Supported RL algorithms include PPO, GRPO, REINFORCE++, and TTRL, with workflows such as Multi-Agent Debate, Chain-of-Agents, Mixture-of-Agents, and Review-RL. The MARTI-v2 MARS² module performs asynchronous multi-agent MCTS with adaptive node expansion and refinement to explore solution spaces and produce high-quality reasoning trajectories for policy updates, using GSPO sequence-level optimization, TIS correction for vLLM sampling mismatch, per-agent dynamic sample filtering, and an overlong token penalty buffer. The output is model weights optimized via multi-agent RL (e.g., Qwen3-8B/14B, DeepCoder-14B).
- Researchers running reproducible RL training of LLMs in multi-agent debate or mixture-of-agents setups with GRPO/PPO
- ML engineers training code generation models who want MCTS tree search instead of naive rollouts to lift Pass@1 on LCB-style benchmarks
- Teams jointly training heterogeneous models (e.g., Qwen3-8B + AreaL-boba-2-8B), each agent with its own role, training strategy, and sample filtering
- Users with existing AutoGen or CAMEL workflows who want to plug their orchestration into an RL training pipeline (experimental support)
- Research scenarios requiring asynchronous tool use and multi-turn agentic RL, such as async code or math RL
How do you install or deploy this agent?
Installation steps (from the repository):
- Clone and install:
git clone https://github.com/TsinghuaC3I/MARTI.git
cd MARTI
pip install -r requirements.txt- Follow the repository's setup instructions for dependencies, including OpenRLHF, Ray, and vLLM.
Hardware threshold: single-agent training requires roughly 8×80G GPUs; multi-agent training requires approximately 8×80G GPUs per agent. No Docker image or hosted service is provided; you must supply your own GPU cluster.
How do you use this agent?
Set path variables in the scripts and run the relevant example scripts:
ROOT_DIR="/path/to/MARTI"
MODEL_DIR="/path/to/models"
# Single-agent MCTS training (MARS²)
bash examples/mars2/run_train_single_mcts.sh
# Multi-agent MCTS training
bash examples/mars2/run_train_multi_mcts.sh
# Async multi-turn code RL (single-agent)
bash examples/single-agent/run_train_code_async.sh
# Multi-agent debate
bash examples/multi-agent/run_train_mad.sh
# Chain-of-agents (MathChat)
bash examples/multi-agent/run_train_mathchat.sh
# Review-RL
bash examples/reviewrl/run_train_reviewrl_async.shDocumentation lives in docs/ (Overview, Workflows Integration, Reward and Training, Experiments).
What are this agent's strengths and limitations?
- Backed by an ICLR 2026 paper and an arXiv technical report (2602.07848), giving peer-reviewed credibility
- Unifies multi-agent training and single-agent RL in one framework with built-in credit assignment and reward shaping, avoiding hand-assembled pipelines
- MARS² provides asynchronous multi-agent MCTS training; on the LCB benchmark multi-agent training improves over Vanilla GRPO by up to 4.4%, with support for 32K-token sequences and heterogeneous agents
- Supports multiple RL algorithms (PPO, GRPO, REINFORCE++, TTRL) plus experimental compatibility with existing AutoGen/CAMEL orchestration
- High hardware barrier: ~8×80G GPUs for single-agent and ~8×80G per agent for multi-agent; the paper's experiments used 3 nodes of 8×H200 with one node per agent
- Research-grade framework; the AutoGen/CAMEL integrations are explicitly experimental and production readiness is undocumented
- No hosted service or Docker deployment; you must maintain the OpenRLHF/Ray/vLLM dependency stack yourself
- Published evidence covers specific model families (Qwen3, AreaL-boba, DeepCoder; DeepSeek-R1, Gemma, Llama appear in topics); effectiveness on other models is not publicly validated
How does this agent compare with similar options?
MARTI is explicitly built on the single-agent RL frameworks OpenRLHF and verl — it extends rather than replaces them: OpenRLHF/verl target single-agent policy training, while MARTI adds centralized multi-agent interaction, reward allocation, and multi-agent tree search (MARS²). The README names no other competing multi-agent RL framework.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| MARTI: Multi-Agent Reinforced Training & Inference for LLMs This agent | 40 · Major gaps | CLIFree | ★ 574 | 2mo ago | Python | — |
| AgentsMeetRL — Awesome List of Agentic Reinforcement Learning | 57 · Major gaps | Agent plugin / skillFree | ★ 1.9k | 2d ago | HTML | — |
| Hands-On Modern RL | 68 · Some gaps | CLIFree | ★ 4.9k | 16d ago | Jupyter Notebook | — |
| rLLM: Reinforcement Learning Framework for LLM Agents | 54 · Major gaps | CLIFree + model costs | ★ 5.9k | 23d ago | Python | OpenAI API |
How does FollowAgents rate this agent?
Why each dimension lost points
The execution code limits concurrency with a semaphore and passes prompts, labels, sampling parameters, and generated results through execution and a result queue, giving limited data-flow visibility and 1 point each for least privilege and transparency. The materials show no user confirmation, sensitive-data protections, or rollback mechanism, so those criteria receive 0. The dependency list includes some pinned or bounded versions but no vulnerability review or complete lock, earning 1 for dependency security. Execution involves model generation, Ray remote instances, and queue writes; these effects are visible in code but their scope is only lightly documented, earning 1. The README names foundational projects such as OpenRLHF, vLLM, and Ray and provides citation and contributor information, earning 2 for attribution.
The Agent abstraction, asynchronous execution flow, and GAIA answer-scoring code provide some evidence of internal coherence. However, execution assumes specific reset/step return keys and calls rewards.item() without visible interface validation, so self-consistency earns 1. Dependencies and an installation step are listed, but not all important dependencies are pinned and the framework relies on substantial external infrastructure, earning 1 for availability. The execution code has no visible user-readable handling for missing fields or generation failures, earning 1 for failure messages.
The README covers single- and multi-Agent setups, code and math tasks, workflows, and tree-search training, earning 2 for audience and scenarios. Capability boundaries and experimental integrations receive only brief treatment, earning 1. Example scripts identify entry points and training scenarios, but provide little guidance on when to select each mode, earning 1 for trigger precision. Hardware and model-path notes cover some environment needs, while cluster, dependency, and configuration details still rely on external guidance, earning 1 for environment fit.
The README has a table of contents, links to module documentation, and grouped examples, earning 2 for information architecture. Installation gives a requirements command but defers additional dependency setup, earning 1. Module and task names are generally clear, earning 2; examples cover several training entry points but there is no FAQ, earning 2. Experimental integrations and hardware requirements reveal some limitations, but there is no complete limitations list, earning 1. The repository includes a full MIT license, earning 3. Dated news entries provide a brief update trail but no versioned release notes, earning 1. The README identifies project leads, contributors, and contact emails, earning 2 for maintenance responsibility.
The execution code collects prompts, labels, observation tokens, rewards, scores, action ranges, and logs into a result usable by a training pipeline, earning 2. The framework combines multi-Agent workflows with reinforcement-learning training, and the README describes concrete use cases and capabilities, earning 2 for marginal value. Training requires roughly eight 80 GB GPUs, with multi-Agent training allocating nodes per Agent; cost-benefit guidance is limited, earning 1.
The README makes specific feature, hardware, and benchmark claims, but the supplied materials lack corresponding configurations, raw data, or a reproducible review trail, earning 1 for claim traceability. The README, execution code, scorer, dependency list, and license provide complementary evidence, but there is no test or documentation cross-check of execution behavior, earning 1 for cross-source corroboration. Some claims are presented as framework features or experimental results, but results remain primarily author-reported in the README, so fact and inference separation earns 1.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- This is a low-confidence review of the supplied static materials; code was not run and README experimental results were not independently verified.
- Operation depends on Ray, vLLM, models, and a GPU cluster. The README describes high hardware requirements, so deployment and cost need careful assessment.
- Execution depends on the Agent reset/step return contract and passes inputs and labels to execution instances; review data handling and error paths before using real data.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/marti)<a href="https://followagents.com/en/agents/marti"><img src="https://followagents.com/badge/marti.svg" alt="FARS review"></a>FAQ
What are the minimum hardware requirements?
Is it a deployable inference service?
Which models and RL algorithms are supported?
Can I plug in my existing AutoGen or CAMEL workflow?
Are there community projects built on it?
Thanks — we use this to improve the page.