rLLM: Reinforcement Learning Framework for LLM Agents
A unified framework for RL training of language agents, supporting any harness, any sandbox, and one-flag backend switching.
- Source repo
- rllm-org/rllm
- Stars
- ★ 5.9k
- Last updated
- 10d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 0/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- Researchers want to train large language models with RL methods like GRPO on multi-GPU clusters.
- Not a fit if
- Solo developers without multi-GPU hardware who want to train RL on a laptop
- Users wanting simple prompt calls, not the Episode/Trajectory and @rllm.rollout decorator API
- Source review
- 0/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
rLLM is an open-source framework for training language agents with reinforcement learning. It provides a unified API to run your agent on any harness (Claude Code, Codex, Terminus-2, mini-swe-agent, opencode, etc.), execute in any sandbox (Docker, Daytona, Modal, or local), and switch between training backends (verl, tinker, fireworks) with a single flag. The framework includes 60+ integrated benchmarks across math, code, QA, search, VLM, and agentic tasks, and supports multiple training methods like GRPO, REINFORCE, RLOO, SFT, and on-policy distillation. A model gateway captures token IDs and logprobs, enabling the same agent code for both evaluation and training. rLLM has been used to train state-of-the-art open-source models like DeepScaleR-1.5B, DeepCoder-14B, and DeepSWE-32B, and is adopted by academic labs and industry teams.
rLLM runs a pipeline: run your agent → collect traces → compute rewards → update the model. The agent is defined via the @rllm.rollout decorator, and the reward function via @rllm.evaluator. During training, the model gateway routes LLM calls and captures token IDs and logprobs, organizing them into Episodes (one task), Trajectories (agent run), and Steps (LLM call). The training backends include verl (distributed multi-GPU), tinker (single-machine), and fireworks (Fireworks platform). For evaluation, it provides CLI commands like rllm eval <benchmark> and rllm train <benchmark>. The framework automatically pulls and runs benchmark datasets, and supports parallel rollout collection via a workflow engine.
- Researchers want to train large language models with RL methods like GRPO on multi-GPU clusters.
- Teams need to evaluate their agents on many benchmarks (e.g., SWE-bench, AIME) without writing separate code for each.
- Developers have existing agent implementations (e.g., using OpenAI API or LangGraph) and want to add RL training without major refactoring.
- Organizations that need flexibility to switch training backends (e.g., from single-machine tinker to distributed verl) based on resource availability.
- Teams that require sandboxed execution (Docker, Modal) for safe and reproducible training runs.
How do you install or deploy this agent?
Requires Python >= 3.11. Recommended installation using uv:
# Install core CLI (tinker backend)
uv pip install "rllm @ git+https://github.com/rllm-org/rllm.git"
# For distributed multi-GPU training (verl + vLLM/SGLang)
uv pip install "rllm[verl] @ git+https://github.com/rllm-org/rllm.git"
# For Fireworks platform
uv pip install "rllm[fireworks] @ git+https://github.com/rllm-org/rllm.git"Alternatively, build from source or use Docker. See https://docs.rllm-project.com/installation for details.
How do you use this agent?
After installation, you can use the CLI or Python API.
CLI quickstart:
# Configure model provider
rllm model setup
# Evaluate on a benchmark (e.g., gsm8k)
rllm eval gsm8k
# Train on a benchmark
rllm train gsm8kPython API example:
- Define a rollout function with
@rllm.rollout. - Define an evaluator with
@rllm.evaluator. - Create an
AgentTrainerand calltrain().
See the cookbooks directory for complete examples.
What are this agent's strengths and limitations?
- Supports 10+ CLI harnesses and Harbor-compatible task directories, plus custom agents via
@rllm.rollout - Flexible sandbox options (Docker, Daytona, Modal) with snapshot and warm-pool acceleration to reduce rollout costs
- One-flag switching between training backends (verl, tinker, fireworks) with the same agent code for eval and training
- 60+ built-in benchmarks covering math, code, QA, search, VLM, and agentic tasks
- Multiple RL algorithms (GRPO, REINFORCE, RLOO, SFT, on-policy distillation)
- Proven results with models like DeepScaleR, DeepCoder, and DeepSWE
- Requires Python 3.11+, which may be restrictive for older environments
- Distributed training depends on verl and vLLM/SGLang, requiring significant GPU resources and setup
- CLI benchmark evaluation may require network access to fetch datasets
- Custom agent integration requires learning the decorator API (e.g.,
@rllm.rollout) and structuring code into Episodes/Trajectories - The Fireworks backend is a specific platform, potentially lock-in
How does this agent compare with similar options?
Compared to standalone agent frameworks (e.g., Claude Code) or RL libraries (e.g., verl), rLLM provides a higher-level abstraction that unifies harnesses, sandboxes, and backends. However, no direct competitors are mentioned in the source.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| rLLM: Reinforcement Learning Framework for LLM Agents This agent | 0 · Major gaps | CLIFree + model costs | ★ 5.9k | 10d ago | Python | OpenAI API |
| AgentsMeetRL — Awesome List of Agentic Reinforcement Learning | 29 · Major gaps | Agent plugin / skillFree | ★ 1.9k | 5d ago | HTML | — |
| HUD | 63 · Some gaps | CLIFree + model costs | ★ 314 | 3d ago | Python | OpenAI API · Claude API |
| MARTI: Multi-Agent Reinforced Training & Inference for LLMs | 36 · Major gaps | CLIFree | ★ 570 | 1mo ago | Python | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Insufficient evidence: The repository does not provide clear information on least privilege, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All criteria are unsupported, hence scored 0.
Insufficient evidence: No clear evidence for self-consistency, dependency availability, or failure messages. All criteria are unsupported, hence scored 0.
Insufficient evidence: No clear evidence for audience and scenarios, capability boundaries, trigger precision, or environment fit. All criteria are unsupported, hence scored 0.
Insufficient evidence: No clear evidence for information architecture, install notes, naming stability, examples and FAQ, known limitations, license, versioning/changelog, or maintenance responsibility. All criteria are unsupported, hence scored 0.
Insufficient evidence: No clear evidence for output usability, marginal value, or cost-benefit. All criteria are unsupported, hence scored 0.
Insufficient evidence: No clear evidence for claim traceability, cross-source corroboration, or fact-inference separation. All criteria are unsupported, hence scored 0.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- Static review based solely on provided files; no execution or independent verification performed.
- No AGENTS.md manifest provided, and publisher identity is unverified; treat as unknown.
- All criteria scored 0 due to insufficient evidence, not indicating defects but lack of assessable evidence.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v0.3.0-pre rLLM: v0.3.0-pre
Release notes excerpt
What's Changed * [feature] add support for on policy distillation by @kylemontgomery1 in https://github.com/rllm-org/rllm/pull/356 * [Bugfix] Make sure the tracking logger is now explicitly finished by trainers by @listar2000 in https://github.com/rllm-org/rllm/pull/358 * Add Geo3K Tinker training example with VLM support by @BrianChen26 in https://github.com/rllm-org/rllm/pull/357 * Support custom metric for SDK by @thwu1 in https://github.com/rllm-org/rllm/pull/362 * fix: typo in bibtex by @tongyx361 in https://github.com/rllm-org/rllm/pull/363 …View on GitHub ↗ -
v0.2.1.post1 rLLM: v0.2.1.post1
Release notes excerpt
What's Changed * update docs & add curves by @thwu1 in https://github.com/rllm-org/rllm/pull/343 * Fix import for colorful_print in agent_sdk_engine.py and agent_sdk_trainer.py by @wht0703 in https://github.com/rllm-org/rllm/pull/345 * Unblock sdk installation by overriding dependices by @wht0703 in https://github.com/rllm-org/rllm/pull/348 * [Doc] Update README and fix a few installation related issues by @listar2000 in https://github.com/rllm-org/rllm/pull/347 * fix: keyerror completion_ids by @kxfan2002 in https://github.com/rllm-org/rllm/pull/353 …View on GitHub ↗ -
v0.2.1 rLLM: v0.2.1
Release notes excerpt
rLLM v0.2.1: Tinker backend, VLM training, Eval Protocol, and SDK (preview) We are excited to release rLLM v0.2.1. This new version comes with the following exciting features: - rLLM SDK (preview): The rLLM SDK enables you to transform agents written in frameworks such as LangGraph, SmolAgent, or Strands into trainable workflows. Check out this [LangGraph RAG example](https://rllm-project.readthedocs.io/en/latest/examples/sdk_langgraph_rag/), which builds a RAG agent and trains it with the rLLM SDK. …View on GitHub ↗
FAQ
Can I use rLLM with my existing OpenAI-based agent?
@rllm.rollout. During training, the model gateway intercepts calls and captures token IDs and logprobs.What hardware is needed for training?
How can I add a custom benchmark?
rllm eval. For custom tasks, you might need to follow Harbor-compatible formats (not detailed in the README).