FollowAgents AI AGENTS DIRECTORY
Home / All Agents / AgentLab
Dev & Engineering browsergymweb-agent-evaluationplaywrightraywebarenaexperiment-reproducibility

AgentLab

A BrowserGym-based framework for building, running, evaluating, and reproducing web-agent experiments.

View on GitHub ↗ Quick start ↓
Source repo
ServiceNow/AgentLab
Stars
★ 664
Last updated
1mo ago
License
NOASSERTION
Primary language
Python
FA score
63/100 · Some gaps

At a glance

How it runs
Library / SDKCLIFramework
Works with
Universal · cross-platformOpenAI API
Cost
Free software; you pay for model usage
Setup effort
High · needs real infrastructure
You'll need
Python 3.11 or 3.12PlaywrightShell / CLINetwork accessLocal filesystem
Typical use
A web-agent researcher comparing several AgentArgs configurations and seeds on MiniWoB can run the matrix as Studies and consolidate the outcomes.
Not a fit if
  • Consumers wanting a ready-to-use product
  • Teams unwilling to set up Docker, self-hosted services, or benchmark environments
Source review
63/100 · Some gaps

What does this agent do, and when should you use it?

AgentLab is a research framework for developing and evaluating web agents across benchmarks supported by BrowserGym. Its Study and make_study interfaces organize runs on benchmarks including MiniWoB, WebArena, WorkArena, VisualWebArena, AssistantBench, and OSWorld. Ray can execute large experiment sets in parallel, while the unified LLM API supports OpenAI, Azure OpenAI, OpenRouter, and self-hosted TGI paths. Results are available through ExpResult, load_result_df, and the AgentXray interface for traces, screenshots, actions, and summaries. The project explicitly positions itself as infrastructure for web-agent research rather than a consumer product.

A developer prepares the selected benchmark, creates a Study with make_study(benchmark=..., agent_args=..., comment=...), and runs it with study.run(n_jobs=...). Each agent-on-task execution is a job; the Ray backend can terminate jobs that exceed a configured timeout and supports large parallel runs. Existing studies can be reopened with Study.load(), unfinished or errored tasks can be located with find_incomplete(include_errors=True), and then rerun. For analysis, inspect_results.load_result_df() gathers experiment summaries into a dataframe, while bgym.ExpResult exposes detailed screenshots and steps_info actions. The agentlab-xray command opens a Gradio interface over experiments in AGENTLAB_EXP_ROOT, allowing users to select an experiment, agent, task, and seed to inspect a trace.

  1. A web-agent researcher comparing several AgentArgs configurations and seeds on MiniWoB can run the matrix as Studies and consolidate the outcomes.
  2. An evaluation team running WebArena or VisualWebArena can use the Ray backend for parallel work while accounting for task dependencies and timed-out jobs.
  3. A developer implementing a BrowserGym agent can use MostBasicAgent as a reference and implement the AgentArgs API plus the extended bgym.AbstractAgentArgs.
  4. A researcher investigating a failed web task can use AgentXray to inspect the selected task and seed through screenshots, actions, and profiling steps.
  5. A team reproducing a reported experiment can retain benchmark, package, and commit information in Study and use ReproducibilityAgent to rerun actions on the same task seeds.

How do you install or deploy this agent?

Python 3.11 or 3.12 is required.

pip install agentlab
playwright install

Prepare the chosen benchmark according to its setup instructions. For OpenAI-backed models, set credentials; OpenRouter or Azure OpenAI variables can be configured when applicable:

export AGENTLAB_EXP_ROOT=<experiment-results-directory>
export OPENAI_API_KEY=<your-openai-api-key>

How do you use this agent?

With default OpenAI credentials configured, start the web assistant with:

agentlab-assistant --start_url https://www.google.com

Run a first experiment:

from agentlab.agents.generic_agent import AGENT_4o_MINI
from agentlab.experiments.study import make_study

study = make_study(
    benchmark="miniwob",
    agent_args=[AGENT_4o_MINI],
    comment="My first study",
)
study.run(n_jobs=5)

Inspect completed or ongoing experiments with:

agentlab-xray

What are this agent's strengths and limitations?

Pros
  • It puts multiple BrowserGym-supported web benchmarks into one Study workflow instead of requiring separate experiment organization and result structures per benchmark.
  • Its Ray backend includes timeout termination for stuck jobs, which is useful for research matrices with many tasks, seeds, or ablations.
  • It provides one documented LLM API path for OpenAI, Azure OpenAI, OpenRouter, and self-hosted TGI.
  • It combines dataframe-level summaries and detailed ExpResult access with AgentXray's interactive trace inspection.
  • Study records benchmark, package, and commit information, and ReproducibilityAgent supports examining rerun differences.
Limitations
  • It is explicitly not a consumer product; adopters must prepare benchmark environments, model credentials, and experiment infrastructure themselves.
  • Some benchmarks require self-hosted services, Docker, static files, or the live web, creating additional environment and operational overhead.
  • WebArena and VisualWebArena tasks can affect instance state; instances are reset before evaluation, and cross-instance evaluations are not currently supported.
  • Model API changes, live websites, regional differences, and stochasticity can affect reproducibility; temperature 0 only reduces most stochasticity.
  • AgentXray depends on Gradio, and the documentation notes possible display issues that may require refreshing and reselecting the experiment.

How does this agent compare with similar options?

BrowserGym is the benchmark ecosystem AgentLab builds on, not a separate substitute: BrowserGym supplies supported benchmarks, while AgentLab handles agent configuration, experiment execution, parallelism, result analysis, and reproducibility records.

Key facts side by side with the most closely related agents.

Agent Source review Form / cost Stars Updated Language Full support on
AgentLab This agent 63 · Some gaps Library / SDKFree + model costs ★ 664 1mo ago Python OpenAI API
BrowserGym Web-Agent Lab 52 · Major gaps Library / SDKFree + model costs ★ 1.4k 11d ago Python OpenAI API
Open Operator Evals 44 · Major gaps CLIFree + model costs ★ 47 1y ago Python —
Ouroboros 77 · Good Desktop appFree + model costs ★ 1.6k today Python —

How does FollowAgents rate this agent?

FollowAgents source review · FARS-2.1
Some gaps
63/ 100 5-point scale 3.2 / 5
Trust 11/29
Reliability 8/14
Adaptability 14/18
Convention 13/18
Effectiveness 12/13
Verifiability 5/8
Why each dimension lost points
Trust11 / 29 · 1.9/5

The README identifies AgentLab as a research framework rather than a consumer product and advises caution. Setup and usage examples require LLM API keys, and the UI Assistant warns that use is at the user's cost and risk. The environment-variable notes disclose that AgentXRay can open a public tunnel and that the Ray dashboard can bind to a public address, but the supplied evidence does not establish comprehensive least-privilege controls, data-retention or secret-handling practices, or pre-action confirmation. Results can be rerun and errored tasks relaunched, but recovery from side effects is not described. The LICENSE, citations, and author metadata provide some attribution evidence; publisher identity remains unverified and earns no credit by itself.

Reliability8 / 14 · 2.9/5

The README gives concrete guidance on Python versions, installation, benchmark setup, job timeouts, parallelism, and factors affecting reproducibility. Supplied tests cover generic-agent result records and parse retries, GAIA environment construction, and CI is configured for formatting and docstring checks. Dependency ranges and optional groups are listed, but the dependency set is broad and some entries have only lower bounds; no lock or security-review evidence is supplied, so dependency availability receives only partial credit. The documentation explains some hangs and agent-failure handling, but does not establish clear diagnosis for every failure path.

Adaptability14 / 18 · 3.9/5

The README clearly targets web-agent research and evaluation, listing benchmarks, experiment scale, analysis, and extension entry points across hosted, self-hosted, and live-web scenarios. It states concrete limits, including lack of multi-instance evaluation, slow resets for some benchmarks, and a beta reproducibility agent; runtime and permission boundaries are less fully described. The commands and API examples are reasonably specific for common tasks, and environment guidance covers Python, Playwright, environment variables, and parallel resource use, but gives limited deployment and security configuration advice.

Convention13 / 18 · 3.6/5

The README is organized into setup, experiments, result analysis, leaderboard, agent implementation, and reproducibility sections, with installation steps, code samples, a benchmark table, and documented limitations. Both LICENSE and pyproject identify Apache-2.0, supporting full license credit. The project derives versions from VCS and defines a fallback, but the supplied material contains no changelog or release policy. Authors appear in package metadata, while maintenance ownership, support channels, and update commitments are unclear, so maintenance and version-evolution scores are low.

Effectiveness12 / 13 · 4.6/5

The framework offers directly useful multi-benchmark experiment orchestration, parallel execution, result aggregation, trace visualization, agent extension, and reproducibility records. The examples make these capabilities actionable for research users, and the value over managing benchmarks and experiments manually is clear. The README also identifies costs: benchmark setup, potentially large job counts, paid LLM use, slow resets, and resource needs. This supports a positive but not maximal cost-benefit score; the evidence does not present it as a low-cost turnkey product.

Verifiability5 / 8 · 3.1/5

Most README claims about features and use have corresponding clues in the supplied pyproject, main.py, tests, or CI configuration; the test excerpts support some parse-retry and result-record behavior. Many benchmark and feature claims are not independently corroborated within this packet, and a CI configuration does not show that tests passed. The documentation labels a beta feature and describes observed issues and reproducibility factors, but some status statements are subjective or time-sensitive. This is a static review and does not treat them as runtime verification.

Risks and how to mitigate them
  • Prepare the benchmark environment and configure the relevant model-service credentials before running. LLM calls may incur charges, and experiments may affect live websites or benchmark state.
  • The README says AgentXRay can open a public tunnel and the Ray dashboard can be configured for public access. Check network exposure and data handling before deployment; the supplied material does not describe complete secret protection or side-effect recovery.
  • This is a research framework; some features are beta, and reproducibility can be affected by dependency versions, model updates, and live-site changes.
Evidence confidence: Low Reviewed Nov 07, 2026 Reviewed revision 8488c564eea0
Review evidence README.mdmain.pyLICENSEpyproject.tomlworkflows/code_format.ymlworkflows/darglint.ymlagents/test_agent.pyagents/test_gaia_agent.pyagents/test_generic_prompt.py
See the full review method →
Maintain this project? Add the FARS badge to your README

The badge shows the current review score and verdict and links back to this page. It refreshes about once an hour after a re-review.

FARS

[![FARS review](https://followagents.com/badge/agentlab.svg)](https://followagents.com/en/agents/agentlab)
<a href="https://followagents.com/en/agents/agentlab"><img src="https://followagents.com/badge/agentlab.svg" alt="FARS review"></a>

Recent releases

Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.

  1. v0.4.2
    Release notes excerpt
    What's Changed * WebArena-Verified by @NicolasAG in https://github.com/ServiceNow/AgentLab/pull/319 * LiteLLMChatModel for GenericAgent by @patricebechard in https://github.com/ServiceNow/AgentLab/pull/325 * Add traces section to README by @amanjaiswal73892 in https://github.com/ServiceNow/AgentLab/pull/326 New Contributors * @NicolasAG made their first contribution in https://github.com/ServiceNow/AgentLab/pull/319 Full Changelog: https://github.com/ServiceNow/AgentLab/compare/v0.4.1...v0.4.2
    View on GitHub ↗
  2. v0.4.1
    Release notes excerpt
    What's Changed * moved this post_hoc_get_action to private codebase by @optimass in https://github.com/ServiceNow/AgentLab/pull/218 * Update repro o1 o3 mini by @jardinetsouffleton in https://github.com/ServiceNow/AgentLab/pull/216 * Add VLLMChatModel support to chat API by @jardinetsouffleton in https://github.com/ServiceNow/AgentLab/pull/220 * Add missing agent imports by @jardinetsouffleton in https://github.com/ServiceNow/AgentLab/pull/223 * adding log_prob option for chat models by @TLSDC in https://github.com/ServiceNow/AgentLab/pull/219 …
    View on GitHub ↗
  3. v0.4.0
    Release notes excerpt
    What's Changed * chore: update __init__.py by @eltociear in https://github.com/ServiceNow/AgentLab/pull/181 * parallel study evaluation by @recursix in https://github.com/ServiceNow/AgentLab/pull/180 * Refactor HuggingFace model initialization to include base model name … by @jardinetsouffleton in https://github.com/ServiceNow/AgentLab/pull/190 * limiting to python 3.11 and above by @ThibaultLSDC in https://github.com/ServiceNow/AgentLab/pull/194 * Implement parallel processing for studies using ProcessPoolExecutor a… by @recursix in https://github.com/ServiceNow/AgentLab/pull/195 …
    View on GitHub ↗

FAQ

Will running AgentLab incur model costs?
Yes. The web assistant is offered at the user's own cost and risk; OpenAI, OpenRouter, and Azure OpenAI runs require their respective API credentials, and cost depends on the provider and experiment scale.
Can I use it without OpenAI?
Yes. The documented unified LLM API paths include OpenRouter, Azure OpenAI, and self-hosted TGI.
Does it guarantee exact reproducibility?
No. It records versions and commit information and includes reproduction tooling, but model API changes, live sites, regional variation, and stochasticity can still change results.
What happens when a task hangs?
The Ray parallel backend can terminate jobs that exceed a configured timeout. For debugging, the documentation recommends n_jobs=1 with VSCode breakpoints.
What license is confirmed for the repository?
The supplied repository metadata lists NOASSERTION, so no specific license terms can be confirmed from that metadata.
Was this page helpful?
View on GitHub ↗ Install ↓