AReaL Async RL Platform
Asynchronous RL infrastructure for training reasoning models and tool-using agent workflows.
Per-dimension scores and reasoning
Evidence shows CI workflows using GCP service account keys and GHCR tokens, but no least-privilege or user confirmation mechanisms. Data flow transparency is lacking, and sensitive data handling is not described. Dependency security is not mentioned, and external effects (e.g., cloud resource consumption) are not explicit. Rollback mechanisms are absent. Source attribution is only via team name in README, unverified.
Self-consistency: README claims support for many algorithms and models, but no verification. Dependency availability: many dependencies, but version compatibility not explained. Failure messages: no error handling documentation.
Audience and scenarios: README describes various use cases, but boundaries not clear. Capability boundaries: limitations not stated. Trigger precision: no trigger mechanisms. Environment fit: supports multiple hardware, but details lacking.
Information architecture: README is well-structured, but detailed docs missing. Install notes: installation steps provided, but dependencies complex. Naming stability: version numbers clear. Examples and FAQ: examples provided, but FAQ missing. Known limitations: not mentioned. License: Apache-2.0. Versioning/changelog: not provided. Maintenance responsibility: not clear.
Output usability: no output format description. Marginal value: not assessed. Cost-benefit: no cost analysis.
Claim traceability: performance claims in README lack evidence. Cross-source corroboration: not provided. Fact-inference separation: not distinguished.
- CI workflows use GCP service account keys and GHCR tokens without least-privilege or key management details.
- Many dependencies with pinned versions, but no security audit or vulnerability scanning mentioned.
- Performance claims in README (e.g., 2.77x speedup) lack reproducible details.
What does this agent do, and when should you use it?
AReaL is a reinforcement-learning training system for language reasoning and agent applications built around a fully asynchronous training paradigm. Its 2.0 architecture separates training, inference, agent, and weight-update services, with Hermes online RL and end-to-end SWE training examples. The repository includes workloads for math, search, customer service, coding, multi-turn tool use, and vision-language models, alongside configurations for GRPO, PPO, DAPO, DPO, SFT, and distillation. Jobs can run through a local scheduler or a Ray cluster, with Megatron, PyTorch FSDP, and PyTorch Archon training backends and SGLang or vLLM inference backends. It is training infrastructure rather than a ready-to-use conversational agent, so adopters must provide models, data, inference infrastructure, and suitable compute.
Users start training through Python scripts and YAML configurations, for example examples/math/gsm8k_rl.py with examples/math/gsm8k_grpo.yaml. That example downloads the openai/gsm8k dataset and Qwen/Qwen2-1.5B-Instruct model, then performs GRPO training according to its configuration; scheduler.type can be local or ray. Additional examples cover agent_workflow, hermes, swe, tau2, search_agent, tir, openai_agents, and camel workflows for trajectory collection, reward calculation, and model training. In AReaL 2.0, training_service, inference_service, agent_service, and weight_update services separate training, inference, agent execution, and weight updates. For black-box agent applications, the documentation states that online RL can be connected by replacing base_url; the OpenClaw example also calls for replacing base_url and api_key.
- A researcher reproducing a single-node experiment can train a math-reasoning model with the GSM8K examples for GRPO, PPO, DAPO, RLOO, or SFT.
- An ML platform team with shared storage and a multi-node GPU cluster can set scheduler.type=ray and configure cluster.n_nodes, cluster.n_gpus_per_node, and cluster.fileroot.
- An agent developer iterating on multi-turn tool use can use the Python executor and calculator scenario in examples/tir.
- A team building a software-engineering or code-repair agent can evaluate the end-to-end RL examples in examples/swe for AReaL-SWEAgent or Claude Code Agent.
- A developer connecting an existing black-box agent runtime to online reinforcement learning can replace its base_url and use the OpenClaw and OpenAI Agents SDK integration examples as starting points.
What are this agent's strengths and limitations?
- It uses a fully asynchronous RL design while allowing the listed algorithms to run synchronously with max_head_offpolicyness=0, enabling both execution modes in one system.
- AReaL 2.0 explicitly separates training, inference, agent, and weight-update services, which suits systems where those concerns need to evolve independently.
- It spans GRPO, PPO, DAPO, RLOO, DPO, reward modeling, SFT, and distillation, with corresponding GSM8K or HH-RLHF configurations.
- The repository documents both local and Ray scheduling plus support matrices for Megatron, PyTorch FSDP, PyTorch Archon, SGLang, and vLLM.
- Concrete training examples cover multi-turn math, search, customer service, tool use, coding agents, and VLMs.
- The documented quickstart depends on CUDA, flash-attn, and SGLang or vLLM, creating environment and version-management overhead.
- Multi-node Ray operation requires users to provision shared storage and configure cluster paths such as cluster.fileroot in YAML.
- The supplied material does not state minimum GPU, memory, cluster-size, or end-to-end cost requirements, so capacity must be validated independently.
- It is training infrastructure with examples, not a general end-user agent that can be deployed without configuration.
- Some integrations are listed only as examples, including OpenAI Agents SDK and Claude Code Agent; the supplied material does not define their full compatibility or production-support boundaries.
How do you install or deploy this agent?
The documented CUDA path is: git clone https://github.com/areal-project/AReaL, then enter AReaL and run pip install uv. Install a flash-attn pre-built wheel matching the Python version first; the README example uses flash_attn-2.8.3+cu128torch2.9-cp312-cp312-linux_x86_64.whl. Then run uv sync --extra cuda to install training packages and the default SGLang inference backend. For vLLM, the documented procedure copies pyproject.vllm.toml to pyproject.toml and uv.vllm.lock to uv.lock before running uv sync --extra cuda. No credentials are specified for the GSM8K quickstart; it downloads its dataset and model over the network.
How do you use this agent?
After installation, run: python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local. This is the documented first single-node training invocation. For a Ray cluster, update the YAML paths to shared storage and run the same script with cluster.n_nodes=2 cluster.n_gpus_per_node=8 cluster.fileroot=/path/to/nfs scheduler.type=ray. For an online black-box-agent workflow, replace base_url as directed by the relevant example; the OpenClaw example also requires api_key.
How does this agent compare with similar options?
The repository acknowledges ReaLHF, OpenRLHF, and VeRL as related projects but does not provide a feature-by-feature or benchmark comparison. Its stated distinction is fully asynchronous RL; the same algorithms can also run in synchronous mode by setting max_head_offpolicyness=0.