Relax Omni-Modal RL Engine
An asynchronous distributed reinforcement-learning engine for post-training text, vision, and audio models.
Per-dimension scores and reasoning
The architecture, cross-cluster transfers, HTTP scaling, metrics, notifications, and model/dataset downloads make major external data flows reasonably visible. Dynamic engine removal is described as protected, draining, cancellable, and reversible, while client tests show abort-on-cancellation. A test specifically verifies that a session credential and response body are excluded from gateway errors. Deductions apply because there is no complete permission inventory, credential lifecycle/storage guidance, or unified confirmation policy for training, networking, telemetry, and scaling. Many dependencies are unpinned; CI tolerates requirements installation failure and substitutes a stub for the internal TransferQueue package, so supply-chain controls are not well established. Attribution names the Xiaohongshu AI Infra Team and Relax Team/Contributors, but the publisher is unverified and ownership and maintenance responsibility remain incomplete.
The README, workflows, and client tests jointly cover asynchronous polling, stable request IDs, retries, cancellation, protocol versioning, input validation, and malformed-response errors. Health monitoring and two-tier recovery are also clearly described. Deductions apply because production recovery claims are primarily README assertions without supplied implementation or tests. The key TransferQueue dependency is unavailable in CI and replaced by a stub, GPU-dependent packages are skipped, and `pip install -r requirements.txt || true` can conceal installation failures, leaving weak evidence for dependency availability in ordinary environments.
Text, vision, audio, multi-turn agent, colocated, fully asynchronous, hybrid, and cross-cluster scenarios are clearly identified with model, algorithm, hardware, and task examples. Environment interfaces, termination conditions, and execution modes provide useful boundaries. Deductions apply because some constraints are only delegated to documentation not included here; Agentic triggering and action constraints are not expressed as a complete policy, and deployment depends heavily on Ray, Megatron, SGLang, GPUs, specific model families, and an internal TransferQueue package.
The README has strong organization across installation, quick starts, architecture, algorithms, models, features, examples, contribution, citation, and licensing, with generally stable terminology. Apache-2.0 metadata matches the complete LICENSE, justifying full license credit. The news table and Versioneer establish update and version paths, but there is no supplied formal release history or compatibility policy. Installation uses a mutable `latest` image without a digest, and no centralized FAQ, troubleshooting section, or systematic limitations list is shown. Maintenance is attributed only to generic teams and Contributors; with unknown publisher identity, named contacts, support scope, and a security-reporting path are absent.
Runnable training scripts, resource requirements, expected log lines, checkpoint format, and a conversion tool make outputs practically usable for the intended audience. Unified multimodal training, asynchronous execution, elastic inference, and pluggable rewards offer substantial marginal value over manually composing these systems. Deductions apply because throughput, recovery, and production-readiness benefits lack supplied benchmarks or implementation evidence. Cost is addressed only through 8/16-GPU examples and qualitative GPU-waste claims, without resource budgets, performance measurements, or operating-cost comparisons.
Many operational claims trace to commands, paths, interface names, or tests. Protocol handling, cancellation, retries, validation, and secret-safe errors receive cross-file test support, while license metadata is corroborated by the LICENSE. Deductions apply because many architecture and performance claims occur only in the README, the supplied source coverage is narrow, and CI masks dependency failures with tolerance and stubbing. Promotional claims such as “high-performance,” “production-ready,” “maximum throughput,” and “one of the few systems” are not clearly separated from measured facts or inference.
- Do not treat the `latest` Docker image or loosely pinned Python/Node dependencies as reproducible or supply-chain verified; pin an image digest and complete dependency set and scan them before deployment.
- CI permits requirements installation to fail and replaces the key internal TransferQueue dependency with a stub. A green workflow therefore does not establish that real GPU, distributed, or cross-cluster paths work.
- The system can download external models and datasets, connect to other clusters, and send metrics or notifications to services such as WandB, ClearML, and Apprise. Review egress, credential scope, data classification, and telemetry configuration before use.
- Scaling and abort/rollback have localized evidence, but checkpoint recovery, global-restart consistency, and reversal of broader external effects are not fully demonstrated by the supplied files.
- Publisher identity is unknown; independently confirm maintenance channels, security disclosure procedures, container ownership, and long-term update responsibility before adoption.
What does this agent do, and when should you use it?
Relax is an open-source reinforcement-learning post-training framework from the Xiaohongshu AI Infra Team for multimodal large language models. Its six-layer service architecture deploys roles such as Actor, Rollout, Critic, ActorFwd, Advantages, and GenRM as Ray Serve services, while Controller, Service, and Registry handle the training loop, lifecycle, and role mappings. Megatron-LM provides the training backend, SGLang handles inference, TransferQueue streams data between components, and DCS synchronizes weights. It offers colocated synchronous, fully asynchronous, and hybrid execution modes and implements PPO, GRPO, M2PO, RLOO, REINFORCE++ variants, GSPO, SAPO, CISPO, and On-Policy Distillation. Training produces Megatron DCP checkpoints that can be converted to Hugging Face weights, and the deployment boundary is a user-operated single-node or multi-node GPU/Ray environment.
The train.py entrypoint parses CLI arguments, handles signals, connects to a Ray cluster, and starts the Controller. Rollout uses SGLang to generate samples and writes them to TransferQueue; StreamingDataLoader feeds streamed samples to the Actor, which trains through Megatron-LM, while ActorFwd, Reference, Advantages, and optional GenRM components perform forward passes, reference-policy computation, advantage estimation, and generative judge scoring. After a training step, DCS can broadcast weights over NCCL to Rollout, ActorFwd, and Reference, while --max-staleness bounds how far asynchronous samples may lag. Agentic RL runs multi-turn execute-observe-decide loops through BaseInteractionEnv.reset, step, and format_observation, masks environment observations out of the loss, and carries multimodal VLM context across turns. HTTP REST APIs can add or remove SGLang inference engines during training, and the system emits metrics, notifications, logs, and Megatron DCP checkpoints.
- A research or engineering team with eight GPUs that wants to post-train Qwen3-4B on DAPO math data using GRPO and symbolic answer verification.
- A multimodal team training Qwen3-VL for visual question answering or image reasoning with built-in or custom reward functions.
- An infrastructure team with two nodes and 16 GPUs running image-and-audio reinforcement learning for Qwen3-Omni.
- Agentic RL researchers connecting search, software-engineering, or other interactive environments to multi-turn training while excluding observations from the loss.
- Cluster operators whose Rollout phase is the bottleneck and who need to add or drain SGLang engines without stopping the training loop.
- Distributed-training teams working with MoE or very large models that require tensor, pipeline, context, and expert parallelism.
What are this agent's strengths and limitations?
- One framework covers text, vision, and audio RL, including a documented end-to-end image-plus-audio recipe for Qwen3-Omni.
- Training and inference are explicitly separated across Megatron-LM, SGLang, TransferQueue, and DCS, with synchronous, fully asynchronous, and hybrid placement options.
- The built-in algorithm set is broad and concrete: PPO, GRPO, M2PO, RLOO, two REINFORCE++ variants, GSPO, SAPO, CISPO, and On-Policy Distillation.
- Agentic training includes multi-turn interaction, loss masking, flexible termination, and accumulated visual context rather than only single-response sampling.
- Operational features include HealthManager recovery, centralized metrics, Apprise notifications, and live Rollout scaling.
- The documented starting recipes have substantial hardware requirements: eight GPUs for the text and vision examples and 16 GPUs across two nodes for the omni-modal example.
- The operational stack combines Ray Serve, Megatron-LM, SGLang, TransferQueue, DCS, NCCL/GLOO, and containerized GPU infrastructure, creating a significant deployment and debugging burden.
- The listed model families use the Megatron backend; an unlisted architecture needs rollout, training, and weight-conversion hooks through Megatron Bridge.
- Fully asynchronous execution introduces stale training samples, so adopters must tune --max-staleness and accept a throughput-versus-on-policy-accuracy tradeoff.
- Megatron DCP is the native checkpoint format, requiring an additional conversion step for Hugging Face-compatible weights.
How do you install or deploy this agent?
The documented installation path uses the official Docker image and requires NVIDIA GPUs, compatible GPU drivers, Docker, network access, and a local workspace. No API key or other credential is documented as mandatory. Run:
docker pull ghcr.io/redai-studio/relaxrl:latestdocker run -it --gpus all --ipc=host --network=host -v /path/to/your/workspace:/root ghcr.io/redai-studio/relaxrl:latest bashInside the container, run:
git clone https://github.com/redai-studio/Relax.git /root/Relaxcd /root/Relax && pip install -e .
The image includes version-matched CUDA, PyTorch, Megatron-LM, SGLang, and Ray dependencies. The project badge specifies Python 3.12.
How do you use this agent?
The smallest documented text-training example uses eight GPUs. Download its public dataset and model inside the container:
hf download --repo-type dataset zhuzilin/dapo-math-17k --local-dir /root/dapo-math-17k
hf download Qwen/Qwen3-4B --local-dir /root/Qwen3-4BStart GRPO training with:
cd /root/Relax && export EXP_DIR=/root
bash scripts/training/text/run-qwen3-4B-8xgpu.shExpected progress messages include “Finish rollout 0/200” and “training step 0/200.” Checkpoints are written in Megatron DCP format and can be converted to Hugging Face weights with scripts/tools/convert_torch_dist_to_hf_bridge.py. A vision-language recipe is available at scripts/training/multimodal/run-qwen3-vl-4B-8xgpu.sh; the documented Qwen3-Omni recipe requires 16 GPUs across two nodes and launches through scripts/entrypoint/spmd-multinode.sh.