WebArena: Realistic Web Environment for Agents
A self-hostable, realistic web environment for building and evaluating autonomous agents.
- Source repo
- web-arena-x/webarena
- Stars
- ★ 1.6k
- Last updated
- 10mo ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 39/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- Researchers evaluating the performance of autonomous agents on realistic web navigation tasks.
- Not a fit if
- Teams that want a quick trial without self-hosting multiple sites and databases
- Users who cannot supply an OpenAI API key or pay per-call costs
- Source review
- 39/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
WebArena is a self-hosted web environment that provides realistic and diverse websites (e-commerce, forums, GitLab, maps) to test autonomous agents. It includes standardized interfaces for agent interaction, a benchmark suite for evaluation, and tools to reproduce paper results. The environment supports browser-based interaction via Playwright, with observation states like accessibility trees. Integration with the AgentLab framework enhances parallel experiment support. It supports multiple prompting strategies and models (e.g., GPT-3.5) and provides an extensible evaluation framework.
WebArena provides a browser-based environment where agents can perform actions (e.g., click elements) and observe states (e.g., accessibility tree). It includes the ScriptBrowserEnv class for environment resetting and action execution. Provides run.py evaluation script, with config generation, auto-login, and result recording. Agents operate via ID-based actions and can be customized with predefined or custom prompts. The environment supports 812 tasks across shopping, forums, GitLab, and maps.
- Researchers evaluating the performance of autonomous agents on realistic web navigation tasks.
- Developers building web automation agents based on accessibility observations and actions.
- Engineers testing agent robustness on shopping, forum, and code hosting platforms in a controlled environment.
- Teams comparing different prompting strategies and models on the same benchmark.
- Educators demonstrating web agent behavior in a teaching environment.
How do you install or deploy this agent?
Requires Python 3.10+. Install: conda create -n webarena python=3.10; conda activate webarena; pip install -r requirements.txt; playwright install; pip install -e . Optional dev install: pip install -e ".[dev]" and mypy --install-types --non-interactive browser_env agents evaluation_harness.
How do you use this agent?
First, set environment variables (e.g., SHOPPING, REDDIT) to point to self-hosted sites. Then run python scripts/generate_test_data.py to generate config files, python browser_env/auto_login.py to obtain auto-login cookies, and set OPENAI_API_KEY. Launch evaluation: python run.py --instruction_path agent/prompts/jsons/p_cot_id_actree_2s.json --test_start_idx 0 --test_end_idx 1 --model gpt-3.5-turbo --result_dir <your_result_dir>.
What are this agent's strengths and limitations?
- Provides a realistic, diverse set of web tasks and a standard evaluation framework.
- Built on Python/Playwright, easy to customize and extend.
- Supports multiple observation spaces (accessibility tree) and action spaces.
- Requires self-hosting all test websites (e.g., shopping, forums), which is complex and time-consuming.
- Evaluation depends on external APIs (e.g., OpenAI), requiring keys and potential costs.
- Codebase maintenance has slowed; recommend switching to AgentLab for experiments.
How does this agent compare with similar options?
Compared to VisualWebArena, WebArena focuses on text and accessibility trees, while VisualWebArena emphasizes visual perception. AgentLab offers enhanced parallel experiments and a unified benchmark.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| WebArena: Realistic Web Environment for Agents This agent | 39 · Major gaps | CLIFree + model costs | ★ 1.6k | 10mo ago | Python | — |
| Auto Browser | 62 · Some gaps | Self-hosted serviceFree + model costs | ★ 908 | today | Python | OpenAI API · Claude API |
| Browser Use | 47 · Major gaps | Hosted serviceFreemium | ★ 118k | 1d ago | Python | Codex · Claude Code · OpenAI API · Claude API |
| Obscura Headless Browser | 75 · Good | CLIFree | ★ 29k | 3d ago | Rust | Claude.ai |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows no explicit mechanisms for least privilege, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. README only mentions needing OPENAI_API_KEY but does not specify its usage scope or protection. Dependency list includes old versions (e.g., openai==0.27.0, transformers==4.33.2) without security audit or mitigation. Therefore all trust criteria score 0.
Self-consistency: Code and tests show consistent environment interface, tests cover various actions, but no detailed failure messages. Dependency availability: Fixed versions but no lock file or mirror, may change over time. Failure messages: Tests only assert success, no error info on failure. Thus self-consistency scores 2, dependency availability and failure messages score 1.
Audience and scenarios: README clearly targets researchers and developers, provides multiple usage scenarios. Capability boundaries: Documentation explains environment for evaluation but does not explicitly limit. Trigger precision: Action definitions are clear, tests cover various actions. Environment fit: Provides Docker and AMI, but not all platforms. Thus each scores 2.
Information architecture: README well-structured with install, quickstart, evaluation sections. Install notes: Detailed installation steps. Naming stability: Version tags exist but no changelog. Examples and FAQ: Provides minimal_example.py and resources page, but no FAQ. Known limitations: Not explicitly listed. License: Apache-2.0 complete. Versioning and changelog: Release tags but no detailed changelog. Maintenance responsibility: No explicit maintainer. Thus information architecture, install, naming, examples score 2, known limitations, versioning, maintenance score 1.
Output usability: Environment provides standard interface, output parseable. Marginal value: As research benchmark, provides unique value. Cost-benefit: Requires self-hosting, high cost, but documentation provides AMI to reduce deployment. Thus output usability and marginal value score 2, cost-benefit score 1.
Claim traceability: README cites paper and leaderboard, verifiable. Cross-source corroboration: Paper and leaderboard exist, but no independent verification. Fact-inference separation: Documentation distinguishes facts and recommendations, but not explicitly. Thus claim traceability scores 2, others score 1.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- Dependency versions are old and may have known vulnerabilities; consider upgrading or security audit.
- No data flow transparency; be cautious with OPENAI_API_KEY usage.
- Self-hosting the environment is resource-intensive; assess requirements.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
What's Changed * Fix homepage url bugs. Add instructions on setting it up by @shuyanzhou in https://github.com/web-arena-x/webarena/pull/28 * Fix Mangeto base URL redirect problem by @shuyanzhou in https://github.com/web-arena-x/webarena/pull/29 * 34 os agnostic select all by @oootttyyy in https://github.com/web-arena-x/webarena/pull/43 * Improve Evaluation Annotation by @shuyanzhou in https://github.com/web-arena-x/webarena/pull/44 * Fix evaluation annotation for example 301, 302 by @shuyanzhou in https://github.com/web-arena-x/webarena/pull/47 …View on GitHub ↗ -
Release notes excerpt
The first release of WebArena See our [README](https://github.com/web-arena-x/webarena) for details! Thanks @shuyanzhou @lwaekfjlk @optimass @urialon Full Changelog: https://github.com/web-arena-x/webarena/commits/latestView on GitHub ↗
FAQ
What dependencies are required to run evaluation?
How do I self-host the test websites?
environment_docker/ to pre-install all sites. Alternatively, use the provided Amazon Machine Image.Does it support multiple languages?
Will the environment reset after evaluation?
environment_docker/README.md.