BrowserGym Web-Agent Lab
A Gymnasium-based environment for building and evaluating web-task automation agents.
- Source repo
- ServiceNow/BrowserGym
- Stars
- ★ 1.4k
- Last updated
- 11d ago
- License
- NOASSERTION
- Primary language
- Python
- FA score
- 52/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A web-agent researcher wants to compare an in-house policy across MiniWoB, WebArena, and WorkArena using one Gymnasium execution loop.
- Not a fit if
- Teams wanting a turnkey consumer automation product, not research infrastructure
- Users expecting one command to run every benchmark without per-benchmark setup
- Source review
- 52/100 · Major gaps 1 safety controls not found
What does this agent do, and when should you use it?
BrowserGym is a Gym environment framework for web-agent research, not a consumer automation product. It registers open-ended and benchmark tasks with Gymnasium, so an agent runs through a reset/step loop that receives observations, rewards, termination signals, and task information. Its default benchmark coverage includes MiniWoB, WebArena, WorkArena, AssistantBench, WebLINX, OpenApps, and TimeWarp, although individual benchmarks can require separate setup. The core interface is designed for teams to implement their own agents, while the repository also includes a demo agent backed by OpenAI. It fits research and engineering teams that need common task interfaces and benchmark entry points, rather than a turnkey governed automation platform.
After installation, importing modules such as browsergym.core, browsergym.miniwob, or browsergym.webarena registers their tasks as Gymnasium environments. An open-ended browser task is created with gym.make("browsergym/openended", task_kwargs={"start_url": "https://www.google.com/"}, wait_for_user_message=True); env.reset() returns obs and info, then the agent supplies actions through env.step(action) and receives obs, reward, terminated, truncated, and info before calling env.close(). Benchmark environments can be started with IDs such as browsergym/miniwob.choose-list, browsergym/workarena.servicenow.order-ipad-pro, and browsergym/webarena.310. The demo entry point, demo_agent/run_demo.py, runs openended, miniwob, workarena, webarena, or visualwebarena tasks and exposes model-name and screenshot options.
- A web-agent researcher wants to compare an in-house policy across MiniWoB, WebArena, and WorkArena using one Gymnasium execution loop.
- An engineer building an open-ended browser assistant needs a task that begins at a chosen start_url and waits for user input after each agent chat message.
- A ServiceNow-focused team wants to exercise knowledge-work browser tasks such as browsergym/workarena.servicenow.order-ipad-pro.
- A multimodal-agent developer wants to run the VisualWebArena demo and enable use_screenshot for a vision-language model.
- An experimentation team wants to use AgentLab alongside BrowserGym benchmarks to run agents and collect and analyze traces.
How do you install or deploy this agent?
Recommended installation:
pip install browsergym
playwright install chromiumFor only the open-ended core task, install browsergym-core; benchmark-specific packages include browsergym-miniwob, browsergym-webarena, and browsergym-workarena. Each benchmark has additional setup requirements. For the demo agent, run pip install -r demo_agent/requirements.txt, then playwright install chromium; its OpenAI backend requires OPENAI_API_KEY.
How do you use this agent?
Import gymnasium as gym and browsergym.core, then create an environment with gym.make("browsergym/openended", task_kwargs={"start_url": "https://www.google.com/"}, wait_for_user_message=True). Call env.reset(), drive your agent in an env.step(action) loop until terminated or truncated is true, and release the environment with env.close(). To validate the demo flow after setting OPENAI_API_KEY, run:
python demo_agent/run_demo.py --task_name openended --start_url https://www.google.comWhat are this agent's strengths and limitations?
- It uses one Gymnasium reset/step interface for open-ended browsing and multiple web-agent benchmarks, making agent loops reusable.
- Its documented default coverage spans MiniWoB, WebArena, WebArenaVerified, VisualWebArena, WorkArena, AssistantBench, WebLINX, OpenApps, and TimeWarp.
- Teams can implement their own action policy or run the included OpenAI-backed demo agent, with model-name and screenshot options.
- New web benchmarks can be designed by inheriting AbstractBrowserTask.
- The repository explicitly positions BrowserGym as research infrastructure rather than a consumer product, so production adoption requires separate security, operations, and workflow controls.
- Playwright Chromium is a runtime dependency, and several benchmarks have their own additional setup.
- The documented demo agent uses an OpenAI backend and requires an OPENAI_API_KEY.
- There is no documented one-command deployment for every benchmark; environment and data preparation remain benchmark-specific.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| BrowserGym Web-Agent Lab This agent | 52 · Major gaps | Library / SDKFree + model costs | ★ 1.4k | 11d ago | Python | OpenAI API |
| Notte Web Automation | 64 · Some gaps | Library / SDKFreemium | ★ 2.1k | today | Python | — |
| Playwright Browser Automation MCP | 65 · Some gaps | MCP serverFree | ★ 39k | 1d ago | TypeScript | Codex · Claude Code |
| assay | 81 · Good | CLIFree | ★ 110 | 2d ago | Python | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
BrowserGym exposes an executable browser environment; the demo includes navigation, tab, and click actions and mentions a broader PythonActionSet. This supports research use but documents limited permission boundaries, so least privilege scores 1. The README example enables wait_for_user_message, but no general confirmation for consequential actions or undo/recovery mechanism is shown, so confirmation is 1 and rollback 0. The demo sends chat, URLs, accessibility tree, DOM, and optionally screenshots to the OpenAI API and requires an API key. This gives partial data-flow visibility, but no sensitive-data filtering or retention guidance is shown, so transparency and sensitive-data handling are 1. The packet gives install steps for Playwright and other packages but no dependency audit or vulnerability mitigation evidence, so dependency security is 1. Browser actions can affect external sites without documented impact controls, so external effects is 1. The LICENSE states Apache-2.0 and the README provides citations and project identification, supporting attribution at 2.
The README has setup and usage examples for multiple environments, but the VisualWebArena example imports webarena and the AssistantBench example imports workarena and lists a workarena prefix; these mismatches reduce self-consistency to 1. Setup names Playwright Chromium and benchmark-specific prerequisites, but many benchmarks rely on external services, data, or accounts, and the sample shows no lock strategy, so dependency availability is 1. The demo raises on unexpected chat roles and gives a ValueError when neither HTML nor accessibility input is enabled. Error handling is partially supported, but user recovery guidance is limited, so failure messages is 1. This is a static review; the project was not run.
The README explicitly positions the project as a web-agent research framework rather than a consumer product and lists open-ended tasks, benchmarks, interaction, and experiment scenarios, earning 3 for audience and scenarios. The AbstractBrowserTask extension point, optional DOM/AXTree/screenshot inputs, and configurable action subsets support adaptation; however, the PythonActionSet example is commented out and boundaries are not fully described, so capability boundaries is 2. Task setup, entry points, and API-key prerequisites are mostly identifiable, but mismatched README imports and prefixes lower trigger precision to 2. The Gymnasium interface, Playwright, and task registration fit the framework purpose, while benchmark-specific external setup adds friction, so environment fit is 2.
The README separates setup, usage, demo, ecosystem, and citation, with benchmark-specific setup pointers, supporting information architecture at 2. Installation includes pip choices, browser installation, and local development steps, but external benchmark configuration is distributed across other documents not fully present in the sample, so install notes is 2. Package names, task IDs, and split packages are visible, but example mismatches weaken naming stability, scoring 1. Multiple task snippets, demo commands, and a --help pointer provide examples; there is no FAQ in the packet and some snippets have incorrect imports, so examples and FAQ is 2. The README warns that this is not a consumer product and requires caution, and points out benchmark setup requirements, so known limitations is 2. The repository LICENSE gives Apache-2.0, scoring 3. Dynamic versioning and a tag-based release workflow are shown, but no changelog is in the packet, so versioning/changelog is 1. Publisher verification is absent; the README identifies the ServiceNow project and contribution path but gives no clear maintainer or support responsibility, so maintenance responsibility is 1.
BrowserGym offers a unified Gym interface, registered tasks, action spaces, and browser observations; the examples help researchers integrate it, so output usability is 2. Its broad benchmark integrations, task extension base class, and research framework role show clear additional value over a single-purpose script, scoring marginal value 3. Users need Chromium and may need benchmark services, datasets, or accounts; setup explains some of this, but no cost assessment or lighter-weight risk controls are shown, so cost-benefit is 1.
README setup, interface, ecosystem, and citation claims have examples; the CI workflow names build and test steps, and the demo source shows observation fields and API calls, supporting traceability at 2. README, demo, license, release, and CI files corroborate the framework role, license, and usage to some extent, but the CI sample is truncated and some benchmark examples conflict, so cross-source corroboration is 2. The material partly separates concrete instructions from illustrative claims, but assertions such as ease of designing benchmarks and ecosystem capability lack quantitative support in the sample, so fact/inference separation is 1.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- This is a low-confidence static review; code was not run and tests were not independently verified.
- The demo sends page DOM, URLs, chat content, and optional screenshots to the OpenAI API; assess data exposure before using real accounts or sensitive pages.
- The README has import or task-prefix mismatches in its VisualWebArena and AssistantBench examples; consult the benchmark-specific setup documents for external prerequisites.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/browsergym)<a href="https://followagents.com/en/agents/browsergym"><img src="https://followagents.com/badge/browsergym.svg" alt="FARS review"></a>Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
What's Changed * Add "BrowserGym Ecosystem" paper in the readme by @xhluca in https://github.com/ServiceNow/BrowserGym/pull/363 * Add WebArena Lite by @amanjaiswal73892 in https://github.com/ServiceNow/BrowserGym/pull/368 * update webarenalite requirement and toml by @amanjaiswal73892 in https://github.com/ServiceNow/BrowserGym/pull/369 * Version bump dev by @amanjaiswal73892 in https://github.com/ServiceNow/BrowserGym/pull/371 * Webarenalite build fix by @amanjaiswal73892 in https://github.com/ServiceNow/BrowserGym/pull/374 …View on GitHub ↗ -
Release notes excerpt
What's Changed * Miniwob zoom by @recursix in https://github.com/ServiceNow/BrowserGym/pull/352 * prevent lxml to go to version 6 by @recursix in https://github.com/ServiceNow/BrowserGym/pull/356 * Fix_bbox_scale by @recursix in https://github.com/ServiceNow/BrowserGym/pull/357 * Make miniwob installation automatic. by @amanjaiswal73892 in https://github.com/ServiceNow/BrowserGym/pull/359 * Fill->type to trigger drop down by @recursix in https://github.com/ServiceNow/BrowserGym/pull/358 New Contributors …View on GitHub ↗ -
Release notes excerpt
What's Changed * Some changes in preparation for agentlab's new ToolUseAgent and the new APIs by @TLSDC in https://github.com/ServiceNow/BrowserGym/pull/340 * Miniwob zoom by @recursix in https://github.com/ServiceNow/BrowserGym/pull/351 Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.13.4...v0.14.1View on GitHub ↗
FAQ
Can BrowserGym directly automate my production web workflow?
Is OpenAI required?
Does installing browsergym make every benchmark immediately runnable?
How do I create a custom web benchmark?
Thanks — we use this to improve the page.