veRL Agent Training

What does this agent do, and when should you use it?

verl-agent is an extension of veRL for reinforcement-learning training of LLM and vision-language agents that interact with environments over multiple turns. Its rollout design builds inputs step by step rather than requiring the full interaction history to be concatenated, with customizable memory and per-step input structure. It exposes parallel Gym-style environments and group environments, whose members share an initial state at reset() for algorithms that need multiple rollouts from the same state. The repository documents GiGPO, GRPO, PPO, DAPO, GSPO, RLOO, and REINFORCE++ paths alongside ALFWorld, WebShop, Search, Sokoban, Gym Cards, and experimental AppWorld environments. Training is launched through shell scripts under examples/, with listed support for Qwen and LLaMA-family models plus a prompt-based GPT-4o agent example.

Training is launched from scripts under examples/; for example, run_alfworld.sh starts GiGPO training in ALFWorld. Environments return feedback through env.step(), while rollout_loop.py passes task parameters using envs.reset(kwargs=gen_batch.non_tensor_batch.pop('env_kwargs', None)); the agent then produces an action at each step. env_manager.py constructs step observations through build_text_obs(), and the default SimpleMemory in agent_system/memory/memory.py can be extended to retain recent steps, key events, summaries, or external knowledge. Prompts can be edited in agent_system/environments/prompts; the WebShop example asks for reasoning in <think> tags and an admissible action in <action> tags. For Search experiments, the repository also documents a local e5 retrieval service and processed Search-R1 data for task rollouts.

1. An RL research team studying long-horizon text agents can train multi-step policies in ALFWorld with GiGPO, GRPO, PPO, RLOO, or DAPO.
2. A researcher building an e-commerce interaction agent can install the separate WebShop environment and run its GiGPO, GRPO, PPO, RLOO, or DAPO examples.
3. A vision-language research team can train on visual tasks such as Sokoban or Gym Cards using the listed Qwen2.5-VL and Qwen3-VL support.
4. An engineer evaluating retrieval-based search agents can preprocess Search-R1 data, download a local index, start the retrieval server, and train in the Search environment.
5. A developer testing memory strategies for interactive agents can replace SimpleMemory and control how history, summaries, and current observations are assembled per step.

What are this agent's strengths and limitations?

How do you install or deploy this agent?

From the repository root, create the core training environment:

conda create -n verl-agent python==3.12 -y
conda activate verl-agent
pip3 install vllm==0.11.0
pip3 install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir

pip install -e .

Before a first ALFWorld run, install its environment dependencies:

pip3 install gymnasium==0.29.1
pip3 install stable-baselines3==2.6.0
pip install alfworld
alfworld-download -f

Then run:

bash examples/gigpo_trainer/run_alfworld.sh

Environment requirements vary: WebShop requires Python 3.10 or lower, while Search requires a separate retriever environment, faiss-gpu, and a local index. A prompt-based GPT-4o script is supplied, but the repository does not document the required OpenAI credentials or their configuration.

How do you use this agent?

After installing the target environment, run the relevant training script from the repository root. GiGPO examples include bash examples/gigpo_trainer/run_alfworld.sh, run_webshop.sh, run_search.sh, and run_sokoban.sh; the LoRA example is bash examples/gigpo_trainer/run_alfworld_lora.sh. For multiple rollouts from a shared initial state, configure env.rollout.n in verl/trainer/config/ppo_trainer.yaml. To add an environment, create a Gym-style, multi-process environment package, define its prompts, and register it in env_manager.py following EnvironmentManagerBase.

How does this agent compare with similar options?

FAQ