veRL Agent Training
Multi-turn reinforcement learning training for long-horizon LLM and vision-language agents.
- Source repo
- langfengQ/verl-agent
- Stars
- ★ 2.4k
- Last updated
- 5mo ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 32/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API (Partial support)
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- An RL research team studying long-horizon text agents can train multi-step policies in ALFWorld with GiGPO, GRPO, PPO, RLOO, or DAPO.
- Not a fit if
- Individuals or small teams without multi-GPU training hardware
- Users who need a single-command, conflict-free install
- Source review
- 32/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
verl-agent is an extension of veRL for reinforcement-learning training of LLM and vision-language agents that interact with environments over multiple turns. Its rollout design builds inputs step by step rather than requiring the full interaction history to be concatenated, with customizable memory and per-step input structure. It exposes parallel Gym-style environments and group environments, whose members share an initial state at reset() for algorithms that need multiple rollouts from the same state. The repository documents GiGPO, GRPO, PPO, DAPO, GSPO, RLOO, and REINFORCE++ paths alongside ALFWorld, WebShop, Search, Sokoban, Gym Cards, and experimental AppWorld environments. Training is launched through shell scripts under examples/, with listed support for Qwen and LLaMA-family models plus a prompt-based GPT-4o agent example.
Training is launched from scripts under examples/; for example, run_alfworld.sh starts GiGPO training in ALFWorld. Environments return feedback through env.step(), while rollout_loop.py passes task parameters using envs.reset(kwargs=gen_batch.non_tensor_batch.pop('env_kwargs', None)); the agent then produces an action at each step. env_manager.py constructs step observations through build_text_obs(), and the default SimpleMemory in agent_system/memory/memory.py can be extended to retain recent steps, key events, summaries, or external knowledge. Prompts can be edited in agent_system/environments/prompts; the WebShop example asks for reasoning in <think> tags and an admissible action in <action> tags. For Search experiments, the repository also documents a local e5 retrieval service and processed Search-R1 data for task rollouts.
- An RL research team studying long-horizon text agents can train multi-step policies in ALFWorld with GiGPO, GRPO, PPO, RLOO, or DAPO.
- A researcher building an e-commerce interaction agent can install the separate WebShop environment and run its GiGPO, GRPO, PPO, RLOO, or DAPO examples.
- A vision-language research team can train on visual tasks such as Sokoban or Gym Cards using the listed Qwen2.5-VL and Qwen3-VL support.
- An engineer evaluating retrieval-based search agents can preprocess Search-R1 data, download a local index, start the retrieval server, and train in the Search environment.
- A developer testing memory strategies for interactive agents can replace SimpleMemory and control how history, summaries, and current observations are assembled per step.
How do you install or deploy this agent?
From the repository root, create the core training environment:
conda create -n verl-agent python==3.12 -y
conda activate verl-agent
pip3 install vllm==0.11.0
pip3 install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dirpip install -e .
Before a first ALFWorld run, install its environment dependencies:
pip3 install gymnasium==0.29.1
pip3 install stable-baselines3==2.6.0
pip install alfworld
alfworld-download -fThen run:
bash examples/gigpo_trainer/run_alfworld.shEnvironment requirements vary: WebShop requires Python 3.10 or lower, while Search requires a separate retriever environment, faiss-gpu, and a local index. A prompt-based GPT-4o script is supplied, but the repository does not document the required OpenAI credentials or their configuration.
How do you use this agent?
After installing the target environment, run the relevant training script from the repository root. GiGPO examples include bash examples/gigpo_trainer/run_alfworld.sh, run_webshop.sh, run_search.sh, and run_sokoban.sh; the LoRA example is bash examples/gigpo_trainer/run_alfworld_lora.sh. For multiple rollouts from a shared initial state, configure env.rollout.n in verl/trainer/config/ppo_trainer.yaml. To add an environment, create a Gym-style, multi-process environment package, define its prompts, and register it in env_manager.py following EnvironmentManagerBase.
What are this agent's strengths and limitations?
- Step-wise input construction and customizable history management suit long-horizon training where retaining the entire history is undesirable.
- Group environments share initial states at reset(), directly supporting methods such as GRPO and DAPO that use multiple trajectories from one state.
- The documented environment set spans text and multimodal tasks, including ALFWorld, WebShop, Search, Sokoban, Gym Cards, and AppWorld.
- GiGPO is supplied alongside GRPO, PPO, DAPO, GSPO, RLOO, and REINFORCE++, enabling algorithm comparisons within one framework.
- A LoRA path is included, and the documentation states that 7B models can be trained on two H100 GPUs.
- Installation is not uniform: WebShop requires Python 3.10 or lower, whereas the core installation uses Python 3.12, and the documentation recommends separate Conda environments.
- Search requires a separate local retrieval service, index download, and faiss-gpu; the retrieval service is documented as using about 6GB of GPU memory per GPU.
- AppWorld is explicitly experimental, so adopters must validate compatibility and stability for their setup.
- The GPT-4o prompt-agent script has no documented API credential, environment-variable, or configuration procedure.
- The documentation notes that GiGPO performance changed slightly after the 2025-06-03 major update; reproducing original paper results requires a version released before that update.
How does this agent compare with similar options?
Compared with veRL, verl-agent focuses its extension on multi-turn agent-environment interaction and agent RL training. Its README contrasts step-wise input construction with the full-history concatenation described for RAGEN and Search-R1: the former allows each step's observation and memory to be assembled independently, while the latter grows context with the interaction history.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| veRL Agent Training This agent | 32 · Major gaps | CLIFree + model costs | ★ 2.4k | 5mo ago | Python | — |
| RLinf RL Infrastructure | 35 · Major gaps | Library / SDKFree | ★ 5.6k | today | Python | — |
| AReaL Async RL Platform | Insufficient evidence | CLIFree | ★ 5.9k | 1d ago | Python | — |
| ART Agent Reinforcement Trainer | 34 · Major gaps | Library / SDKFreemium | ★ 11k | today | Python | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows no explicit mechanisms for least privilege, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All trust criteria are unsupported, hence score 0.
Self-consistency: README claims support for various models and environments, but requirements.txt pins transformers to 4.51.1 while README install guide does not specify, a minor inconsistency. Dependency availability: requirements.txt lists dependencies but without locked versions or checksums, and some like flash-attn require specific builds, affecting reproducibility. Failure messages: No error handling or diagnostic information provided, score 0.
Audience and scenarios: README clearly targets researchers and developers for LLM/VLM agent training, with examples for multiple environments and algorithms. Capability boundaries: README lists supported features and limitations, but not exhaustive. Trigger precision: Run scripts provided, but trigger conditions or parameters not detailed. Environment fit: Installation guides for multiple environments, but hardware requirements or compatibility details not specified.
Information architecture: README is well-structured with TOC, features, installation, examples. Install notes: Detailed steps provided, but some dependency versions not pinned. Naming stability: Project name and version consistent, but no version history. Examples and FAQ: Multiple examples and FAQ sections, but no common issues. Known limitations: Some environments marked experimental, but not comprehensive. License: Apache-2.0 provided. Versioning/changelog: News updates but no formal changelog. Maintenance responsibility: No clear maintainer or contribution guidelines.
Output usability: Training scripts and models provided, but no output format or evaluation metrics. Marginal value: New algorithm GiGPO and multiple environments offer research value. Cost-benefit: No performance benchmarks or resource requirements, making cost-benefit unclear.
Claim traceability: README cites papers and HuggingFace models, but no detailed experimental configs. Cross-source corroboration: W&B links and third-party support provided, but not independently verified. Fact-inference separation: Results and inferences distinguished, but not explicitly labeled.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- Dependency versions are not pinned, which may affect reproducibility.
- No security audit or permission management mechanisms are provided.
- Some environment installations rely on external services (e.g., Google Drive), which may be unstable.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
What's Changed * Langfeng alfworld by @langfengQ in https://github.com/langfengQ/verl-agent/pull/3 * Langfeng Merge verl 31/3/2025, including SGlang and vllm 0.8.2 by @langfengQ in https://github.com/langfengQ/verl-agent/pull/7 * Address the SuccessManager bug in the validation by @langfengQ in https://github.com/langfengQ/verl-agent/pull/8 * Update README.md by @langfengQ in https://github.com/langfengQ/verl-agent/pull/9 * Langfeng by @langfengQ in https://github.com/langfengQ/verl-agent/pull/11 …View on GitHub ↗