AgentLab

What does this agent do, and when should you use it?

AgentLab is a research framework for developing and evaluating web agents across benchmarks supported by BrowserGym. Its Study and make_study interfaces organize runs on benchmarks including MiniWoB, WebArena, WorkArena, VisualWebArena, AssistantBench, and OSWorld. Ray can execute large experiment sets in parallel, while the unified LLM API supports OpenAI, Azure OpenAI, OpenRouter, and self-hosted TGI paths. Results are available through ExpResult, load_result_df, and the AgentXray interface for traces, screenshots, actions, and summaries. The project explicitly positions itself as infrastructure for web-agent research rather than a consumer product.

A developer prepares the selected benchmark, creates a Study with make_study(benchmark=..., agent_args=..., comment=...), and runs it with study.run(n_jobs=...). Each agent-on-task execution is a job; the Ray backend can terminate jobs that exceed a configured timeout and supports large parallel runs. Existing studies can be reopened with Study.load(), unfinished or errored tasks can be located with find_incomplete(include_errors=True), and then rerun. For analysis, inspect_results.load_result_df() gathers experiment summaries into a dataframe, while bgym.ExpResult exposes detailed screenshots and steps_info actions. The agentlab-xray command opens a Gradio interface over experiments in AGENTLAB_EXP_ROOT, allowing users to select an experiment, agent, task, and seed to inspect a trace.

1. A web-agent researcher comparing several AgentArgs configurations and seeds on MiniWoB can run the matrix as Studies and consolidate the outcomes.
2. An evaluation team running WebArena or VisualWebArena can use the Ray backend for parallel work while accounting for task dependencies and timed-out jobs.
3. A developer implementing a BrowserGym agent can use MostBasicAgent as a reference and implement the AgentArgs API plus the extended bgym.AbstractAgentArgs.
4. A researcher investigating a failed web task can use AgentXray to inspect the selected task and seed through screenshots, actions, and profiling steps.
5. A team reproducing a reported experiment can retain benchmark, package, and commit information in Study and use ReproducibilityAgent to rerun actions on the same task seeds.

How do you install or deploy this agent?

Python 3.11 or 3.12 is required.

pip install agentlab
playwright install

Prepare the chosen benchmark according to its setup instructions. For OpenAI-backed models, set credentials; OpenRouter or Azure OpenAI variables can be configured when applicable:

export AGENTLAB_EXP_ROOT=<experiment-results-directory>
export OPENAI_API_KEY=<your-openai-api-key>

How do you use this agent?

With default OpenAI credentials configured, start the web assistant with:

agentlab-assistant --start_url https://www.google.com

Run a first experiment:

from agentlab.agents.generic_agent import AGENT_4o_MINI
from agentlab.experiments.study import make_study

study = make_study(
    benchmark="miniwob",
    agent_args=[AGENT_4o_MINI],
    comment="My first study",
)
study.run(n_jobs=5)

Inspect completed or ongoing experiments with:

agentlab-xray

What are this agent's strengths and limitations?

How does this agent compare with similar options?

Key facts side by side with the most closely related agents.

How does FollowAgents rate this agent?

FAQ