BrowserGym Web-Agent Lab
A Gymnasium-based environment for building and evaluating web-task automation agents.
- Source repo
- ServiceNow/BrowserGym
- Stars
- ★ 1.4k
- Last updated
- 1d ago
- License
- NOASSERTION
- Primary language
- Python
- FA score
- 54/100 · Major gaps
At a glance
- Works with
- Universal · cross-platformOpenAI API
- You'll need
- Typical use
- A web-agent researcher wants to compare an in-house policy across MiniWoB, WebArena, and WorkArena using one Gymnasium execution loop.
- Main limitation
- The repository explicitly positions BrowserGym as research infrastructure rather than a consumer product, so production adoption requires separate security, operations, and workflow controls.
- Source review
- 54/100 · Major gaps 1 safety controls not found
What does this agent do, and when should you use it?
BrowserGym is a Gym environment framework for web-agent research, not a consumer automation product. It registers open-ended and benchmark tasks with Gymnasium, so an agent runs through a reset/step loop that receives observations, rewards, termination signals, and task information. Its default benchmark coverage includes MiniWoB, WebArena, WorkArena, AssistantBench, WebLINX, OpenApps, and TimeWarp, although individual benchmarks can require separate setup. The core interface is designed for teams to implement their own agents, while the repository also includes a demo agent backed by OpenAI. It fits research and engineering teams that need common task interfaces and benchmark entry points, rather than a turnkey governed automation platform.
After installation, importing modules such as browsergym.core, browsergym.miniwob, or browsergym.webarena registers their tasks as Gymnasium environments. An open-ended browser task is created with gym.make("browsergym/openended", task_kwargs={"start_url": "https://www.google.com/"}, wait_for_user_message=True); env.reset() returns obs and info, then the agent supplies actions through env.step(action) and receives obs, reward, terminated, truncated, and info before calling env.close(). Benchmark environments can be started with IDs such as browsergym/miniwob.choose-list, browsergym/workarena.servicenow.order-ipad-pro, and browsergym/webarena.310. The demo entry point, demo_agent/run_demo.py, runs openended, miniwob, workarena, webarena, or visualwebarena tasks and exposes model-name and screenshot options.
- A web-agent researcher wants to compare an in-house policy across MiniWoB, WebArena, and WorkArena using one Gymnasium execution loop.
- An engineer building an open-ended browser assistant needs a task that begins at a chosen start_url and waits for user input after each agent chat message.
- A ServiceNow-focused team wants to exercise knowledge-work browser tasks such as browsergym/workarena.servicenow.order-ipad-pro.
- A multimodal-agent developer wants to run the VisualWebArena demo and enable use_screenshot for a vision-language model.
- An experimentation team wants to use AgentLab alongside BrowserGym benchmarks to run agents and collect and analyze traces.
How do you install or deploy this agent?
Recommended installation:
pip install browsergym
playwright install chromiumFor only the open-ended core task, install browsergym-core; benchmark-specific packages include browsergym-miniwob, browsergym-webarena, and browsergym-workarena. Each benchmark has additional setup requirements. For the demo agent, run pip install -r demo_agent/requirements.txt, then playwright install chromium; its OpenAI backend requires OPENAI_API_KEY.
How do you use this agent?
Import gymnasium as gym and browsergym.core, then create an environment with gym.make("browsergym/openended", task_kwargs={"start_url": "https://www.google.com/"}, wait_for_user_message=True). Call env.reset(), drive your agent in an env.step(action) loop until terminated or truncated is true, and release the environment with env.close(). To validate the demo flow after setting OPENAI_API_KEY, run:
python demo_agent/run_demo.py --task_name openended --start_url https://www.google.comWhat are this agent's strengths and limitations?
- It uses one Gymnasium reset/step interface for open-ended browsing and multiple web-agent benchmarks, making agent loops reusable.
- Its documented default coverage spans MiniWoB, WebArena, WebArenaVerified, VisualWebArena, WorkArena, AssistantBench, WebLINX, OpenApps, and TimeWarp.
- Teams can implement their own action policy or run the included OpenAI-backed demo agent, with model-name and screenshot options.
- New web benchmarks can be designed by inheriting AbstractBrowserTask.
- The repository explicitly positions BrowserGym as research infrastructure rather than a consumer product, so production adoption requires separate security, operations, and workflow controls.
- Playwright Chromium is a runtime dependency, and several benchmarks have their own additional setup.
- The documented demo agent uses an OpenAI backend and requires an OPENAI_API_KEY.
- There is no documented one-command deployment for every benchmark; environment and data preparation remain benchmark-specific.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| BrowserGym Web-Agent Lab This agent | 54 · Major gaps | ★ 1.4k | 1d ago | Python | OpenAI API |
| Notte Web Automation | 44 · Major gaps | ★ 2k | 1d ago | Python | — |
| CamoFox Browser Server | 72 · Some gaps | ★ 396 | 1mo ago | JavaScript | Claude Code |
| Webwright Browser Coding Agent | 66 · Some gaps | ★ 6k | 1mo ago | Python | Codex · Claude Code · OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: project provides browser automation environment, but does not explicitly implement least privilege; user confirmation exists (e.g., wait_for_user_message) but not enforced; data flow transparency limited, no detailed data collection and transmission; sensitive data handling not explicit; dependency security not audited; external effects (e.g., browser actions) not explicitly restricted; rollback not mentioned; source attribution clear (ServiceNow). Deductions: lack of security design documentation and permission control details.
Evidence shows: project structure consistent, README and code examples align; dependency availability good, CI tests present; failure messages reflected in tests (e.g., last_action_error). Deductions: no detailed error handling documentation.
Evidence shows: targeted at researchers, provides multiple benchmark scenarios; capability boundaries clear (e.g., openended tasks); trigger precision via task IDs; environment fit good (supports multiple browsers and OS). Deductions: no detailed environment configuration documentation.
Evidence shows: information architecture clear, README and docs; install notes detailed; naming stable (e.g., browsergym.*); examples and FAQ provided; known limitations warned in README; license Apache 2.0; versioning via GitHub Actions release; maintenance responsibility clear (ServiceNow). Deductions: missing detailed changelog.
Evidence shows: output usability high, multiple APIs and examples; marginal value high, multiple benchmarks; cost-benefit reasonable, open source free. Deductions: no performance benchmark data.
Evidence shows: claims traceable, paper and citations; cross-source verification (e.g., GitHub Actions tests); fact-inference separation (e.g., warnings in README). Deductions: no independent third-party verification.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Project is a research framework, not a consumer product, use with caution.
- No security audit or permission control details provided, recommend security assessment before production use.
- Depends on external services (e.g., OpenAI API), be aware of data privacy and costs.