AReaL Async RL Platform
Asynchronous RL infrastructure for training reasoning models and tool-using agent workflows.
- Source repo
- areal-project/AReaL
- Stars
- ★ 5.9k
- Last updated
- 1d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- Insufficient evidence
At a glance
- How it runs
- Works with
- Universal · cross-platformClaude Code · OpenAI API (Partial support)
- Cost
- Free, no paid service needed
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A researcher reproducing a single-node experiment can train a math-reasoning model with the GSM8K examples for GRPO, PPO, DAPO, RLOO, or SFT.
- Not a fit if
- Teams without CUDA GPUs or large clusters
- End users wanting a ready-to-chat agent
- Teams needing fully offline, dependency-free operation
- Source review
- Insufficient evidence 8 safety controls not found
What does this agent do, and when should you use it?
AReaL is a reinforcement-learning training system for language reasoning and agent applications built around a fully asynchronous training paradigm. Its 2.0 architecture separates training, inference, agent, and weight-update services, with Hermes online RL and end-to-end SWE training examples. The repository includes workloads for math, search, customer service, coding, multi-turn tool use, and vision-language models, alongside configurations for GRPO, PPO, DAPO, DPO, SFT, and distillation. Jobs can run through a local scheduler or a Ray cluster, with Megatron, PyTorch FSDP, and PyTorch Archon training backends and SGLang or vLLM inference backends. It is training infrastructure rather than a ready-to-use conversational agent, so adopters must provide models, data, inference infrastructure, and suitable compute.
Users start training through Python scripts and YAML configurations, for example examples/math/gsm8k_rl.py with examples/math/gsm8k_grpo.yaml. That example downloads the openai/gsm8k dataset and Qwen/Qwen2-1.5B-Instruct model, then performs GRPO training according to its configuration; scheduler.type can be local or ray. Additional examples cover agent_workflow, hermes, swe, tau2, search_agent, tir, openai_agents, and camel workflows for trajectory collection, reward calculation, and model training. In AReaL 2.0, training_service, inference_service, agent_service, and weight_update services separate training, inference, agent execution, and weight updates. For black-box agent applications, the documentation states that online RL can be connected by replacing base_url; the OpenClaw example also calls for replacing base_url and api_key.
- A researcher reproducing a single-node experiment can train a math-reasoning model with the GSM8K examples for GRPO, PPO, DAPO, RLOO, or SFT.
- An ML platform team with shared storage and a multi-node GPU cluster can set scheduler.type=ray and configure cluster.n_nodes, cluster.n_gpus_per_node, and cluster.fileroot.
- An agent developer iterating on multi-turn tool use can use the Python executor and calculator scenario in examples/tir.
- A team building a software-engineering or code-repair agent can evaluate the end-to-end RL examples in examples/swe for AReaL-SWEAgent or Claude Code Agent.
- A developer connecting an existing black-box agent runtime to online reinforcement learning can replace its base_url and use the OpenClaw and OpenAI Agents SDK integration examples as starting points.
How do you install or deploy this agent?
The documented CUDA path is: git clone https://github.com/areal-project/AReaL, then enter AReaL and run pip install uv. Install a flash-attn pre-built wheel matching the Python version first; the README example uses flash_attn-2.8.3+cu128torch2.9-cp312-cp312-linux_x86_64.whl. Then run uv sync --extra cuda to install training packages and the default SGLang inference backend. For vLLM, the documented procedure copies pyproject.vllm.toml to pyproject.toml and uv.vllm.lock to uv.lock before running uv sync --extra cuda. No credentials are specified for the GSM8K quickstart; it downloads its dataset and model over the network.
How do you use this agent?
After installation, run: python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local. This is the documented first single-node training invocation. For a Ray cluster, update the YAML paths to shared storage and run the same script with cluster.n_nodes=2 cluster.n_gpus_per_node=8 cluster.fileroot=/path/to/nfs scheduler.type=ray. For an online black-box-agent workflow, replace base_url as directed by the relevant example; the OpenClaw example also requires api_key.
What are this agent's strengths and limitations?
- It uses a fully asynchronous RL design while allowing the listed algorithms to run synchronously with max_head_offpolicyness=0, enabling both execution modes in one system.
- AReaL 2.0 explicitly separates training, inference, agent, and weight-update services, which suits systems where those concerns need to evolve independently.
- It spans GRPO, PPO, DAPO, RLOO, DPO, reward modeling, SFT, and distillation, with corresponding GSM8K or HH-RLHF configurations.
- The repository documents both local and Ray scheduling plus support matrices for Megatron, PyTorch FSDP, PyTorch Archon, SGLang, and vLLM.
- Concrete training examples cover multi-turn math, search, customer service, tool use, coding agents, and VLMs.
- The documented quickstart depends on CUDA, flash-attn, and SGLang or vLLM, creating environment and version-management overhead.
- Multi-node Ray operation requires users to provision shared storage and configure cluster paths such as cluster.fileroot in YAML.
- The supplied material does not state minimum GPU, memory, cluster-size, or end-to-end cost requirements, so capacity must be validated independently.
- It is training infrastructure with examples, not a general end-user agent that can be deployed without configuration.
- Some integrations are listed only as examples, including OpenAI Agents SDK and Claude Code Agent; the supplied material does not define their full compatibility or production-support boundaries.
How does this agent compare with similar options?
The repository acknowledges ReaLHF, OpenRLHF, and VeRL as related projects but does not provide a feature-by-feature or benchmark comparison. Its stated distinction is fully asynchronous RL; the same algorithms can also run in synchronous mode by setting max_head_offpolicyness=0.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| AReaL Async RL Platform This agent | Insufficient evidence | CLIFree | ★ 5.9k | 1d ago | Python | — |
| Relax Omni-Modal RL Engine | 64 · Some gaps | CLIFree | ★ 670 | 1d ago | Python | — |
| ART Agent Reinforcement Trainer | 34 · Major gaps | Library / SDKFreemium | ★ 11k | today | Python | — |
| AgentFlow | 31 · Major gaps | CLIFree + model costs | ★ 2.1k | 9mo ago | Python | OpenAI API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows CI workflows using GCP service account keys and GHCR tokens, but no least-privilege or user confirmation mechanisms. Data flow transparency is lacking, and sensitive data handling is not described. Dependency security is not mentioned, and external effects (e.g., cloud resource consumption) are not explicit. Rollback mechanisms are absent. Source attribution is only via team name in README, unverified.
Self-consistency: README claims support for many algorithms and models, but no verification. Dependency availability: many dependencies, but version compatibility not explained. Failure messages: no error handling documentation.
Audience and scenarios: README describes various use cases, but boundaries not clear. Capability boundaries: limitations not stated. Trigger precision: no trigger mechanisms. Environment fit: supports multiple hardware, but details lacking.
Information architecture: README is well-structured, but detailed docs missing. Install notes: installation steps provided, but dependencies complex. Naming stability: version numbers clear. Examples and FAQ: examples provided, but FAQ missing. Known limitations: not mentioned. License: Apache-2.0. Versioning/changelog: not provided. Maintenance responsibility: not clear.
Output usability: no output format description. Marginal value: not assessed. Cost-benefit: no cost analysis.
Claim traceability: performance claims in README lack evidence. Cross-source corroboration: not provided. Fact-inference separation: not distinguished.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- CI workflows use GCP service account keys and GHCR tokens without least-privilege or key management details.
- Many dependencies with pinned versions, but no security audit or vulnerability scanning mentioned.
- Performance claims in README (e.g., 2.77x speedup) lack reproducible details.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
What's Changed * ci: move Docker builder to us-central1-a by @sitabulaixizawaluduo in https://github.com/areal-project/AReaL/pull/1640 * fix(trainer): run initial evaluation before first update by @harooos in https://github.com/areal-project/AReaL/pull/1636 * gov: add Le8r0nJames to codeowned scopes by @sitabulaixizawaluduo in https://github.com/areal-project/AReaL/pull/1641 * feat(workflow): support multimodal agent trajectories by @sitabulaixizawaluduo in https://github.com/areal-project/AReaL/pull/1606 …View on GitHub ↗ -
Release notes excerpt
What's Changed * chore(swe): remove unused SWE SFT dataset loader by @Le8r0nJames in https://github.com/areal-project/AReaL/pull/1467 * fix(docker): rename zeroclaw installer flag to --skip-quickstart by @sitabulaixizawaluduo in https://github.com/areal-project/AReaL/pull/1469 * fix(megatron): use dp_reshardable optimizer sharding and load optimizer state correctly by @Le8r0nJames in https://github.com/areal-project/AReaL/pull/1468 * fix(hermes): preserve singleton online reward signal by @jszzr in https://github.com/areal-project/AReaL/pull/1474 …View on GitHub ↗ -
Release notes excerpt
🚀 Key Highlights Micro-service Architecture Includes [training](https://github.com/areal-project/AReaL/tree/main/areal/v2/training_service), [inference](https://github.com/areal-project/AReaL/tree/main/areal/v2/inference_service), [agent](https://github.com/areal-project/AReaL/tree/main/areal/v2/agent_service), and [weight update](https://github.com/areal-project/AReaL/tree/main/areal/v2/weight_update) services. Hermes Agent Example …View on GitHub ↗