WebsiteBench
A benchmark where AI agents reverse-engineer real websites through browser exploration and get scored.
- Source repo
- TIGER-AI-Lab/WebsiteBench
- Stars
- ★ 84
- Last updated
- 1d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 43/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platform
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- Research groups comparing how different browser-operating agents handle realistic website reconstruction.
- Not a fit if
- Teams that need to ship a real website quickly
- Environments without Docker or a model API key
- Source review
- 43/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
WebsiteBench is an agent benchmark for full-stack website reconstruction: given a production-level site that the agent can only explore through a browser, it must produce a full-stack version matching the original's appearance and behavior. It ships 70 website tasks and 46,916 hidden test cases, distributed via Harbor Hub and Hugging Face and downloadable through the websitebench PyPI package. Execution relies on Docker and Harbor (the README pins harbor==0.25.0); you launch runs with harbor run --dataset websitebench/WebsiteBench@v1 and must allow specific environment hosts such as snapshot.debian.org, github.com, nodejs.org and registry.npmjs.org. Each task contains 302 to 2,016 hidden cases; a case passes only when its functional checks pass and every visual checkpoint reaches a normalized SSIM of 0.99, and a task's reward is the share of passing cases, written to jobs/<job>/<trial>/verifier/reward.txt. Benchmark code, task definitions and the verifier are Apache-2.0, while the reference websites reproduce third-party content for non-commercial research only.
The loop has three stages: the agent explores the target production site in a browser, reverse-engineers a full-stack implementation, and the benchmark's verifier scores the result. Tasks run inside Docker through harbor run --dataset websitebench/WebsiteBench@v1, or a single task via --task websitebench/medium@v1; --agent codex --model '<model>' selects the agent and model, and repeated --allow-environment-host flags whitelist the hosts needed during builds. Task data is fetched after pip install websitebench by running websitebench download --task medium. Each hidden case combines functional checks with visual checkpoints judged at a normalized SSIM of 0.99; rewards land in the trial's verifier/reward.txt. Running with --agent oracle applies the reference solution so you can confirm a task is solvable before spending model budget.
- Research groups comparing how different browser-operating agents handle realistic website reconstruction.
- Model developers regression-testing full-stack code generation and UI fidelity against one hidden test set before release.
- Benchmark maintainers who need reproducible commands and a public leaderboard to publish results.
- Teams validating a new or edited task definition with --agent oracle before it enters a scoring run.
- Evaluators who want to run all 70 tasks on cloud machines as described in docs/cloud.md.
How do you install or deploy this agent?
Install Docker first, then Harbor and the websitebench package. The README pins Harbor to 0.25.0:
uv tool install harbor==0.25.0
pip install websitebenchYou also need a model-provider API key, for example:
export OPENAI_API_KEY='<key>'How do you use this agent?
Download the tasks and run an agent over all 70 tasks:
websitebench download --task medium
harbor run --dataset websitebench/WebsiteBench@v1 \
--agent codex --model '<model>' --n-concurrent 1 \
--allow-environment-host snapshot.debian.org \
--allow-environment-host raw.githubusercontent.com \
--allow-environment-host github.com \
--allow-environment-host nodejs.org \
--allow-environment-host registry.npmjs.orgTo check one task with its reference solution, swap the dataset flag for a task:
harbor run --task websitebench/medium@v1 --agent oracleRead each trial's reward from its job directory:
cat jobs/<job>/<trial>/verifier/reward.txtWhat are this agent's strengths and limitations?
- 70 production-level websites and 46,916 hidden test cases give enough signal to separate agents by capability.
- Scoring combines functional checks with visual checkpoints at a 0.99 SSIM threshold, not just a render-success check.
- The --agent oracle reference solution lets you self-check a task before running a full model evaluation.
- Tasks are obtainable through the PyPI package and Harbor Hub, and rewards are written to a predictable file path.
- Requires Docker plus a pinned Harbor 0.25.0 install, adding setup work beyond a single command.
- Network access must be opened to snapshot.debian.org, nodejs.org, registry.npmjs.org and similar hosts.
- A model API key is mandatory, and running all 70 tasks incurs real model-call costs.
- Reference websites reproduce third-party sites and are licensed for non-commercial research only.
- The README does not list provider adapters, so agents or models outside its example need their own verification.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| WebsiteBench This agent | 43 · Major gaps | CLIFree + model costs | ★ 84 | 1d ago | Python | — |
| Open Operator Evals | 29 · Major gaps | CLIFree + model costs | ★ 47 | 1y ago | Python | — |
| SWE-Bench Pro | 55 · Major gaps | CLIFree | ★ 557 | 1mo ago | Python | — |
| Chrome DevTools MCP | 79 · Some gaps | MCP serverFree | ★ 55k | 1d ago | TypeScript | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
least_privilege: CI declares permissions: contents: read, publishing uses PyPI Trusted Publishing without stored tokens, and the mailpit container runs with network_mode: none and external requests disabled — good practice. However the README's harbor run requires --allow-environment-host for several external hosts with no sandbox boundary explanation, so 1. user_confirmation: README instructs exporting OPENAI_API_KEY and running directly with no confirmation or prompt mechanism, 0. data_flow_transparency: only states tasks come from Harbor Hub/Hugging Face and must be downloaded; no description of data flows or key usage scope, 1. sensitive_data_handling: tasks include sensitive scenarios like 1password import, but no credential-handling guidance is visible; only mailpit isolation hints at care, 1. dependency_security: harbor pinned exactly to 0.25.0, mailpit image pinned by sha256 digest, publish workflow actions pinned to commit SHAs — strong; huggingface-hub only has a lower bound, so 2. external_effects: runs pull external hosts, download datasets, and build containers; README does not describe impact scope or cleanup, 1. rollback: no rollback, cleanup, or recovery mechanism described, 0. source_attribution: LICENSE and NOTICE present, README attributes third-party site content and restricts to non-commercial research, citation provided, 2.
README install/run/scoring descriptions align with pyproject dependencies and CI steps (harbor==0.25.0, websitebench CLI, reward.txt path); no visible contradictions, 2. dependency_availability: harbor==0.25.0 and huggingface-hub>=1.0 are on PyPI, but Python is constrained to >=3.12,<3.15 and the external Harbor Hub dataset availability is assumed with no offline/mirror option, 1. failure_messages: CI validates release tag against version and emits ::error::, but user-facing runtime failure messages and common error handling are absent from visible files, 1.
README targets researchers and agent developers with single-task, full-suite, and cloud scenarios, 2. capability_boundaries: describes task scale and scoring but not agent capability boundaries, unsupported cases, or scope limits, 1. trigger_precision: CLI commands and --task/--dataset flags are fairly clear, but full parameter documentation and invalid-input handling are missing, 1. environment_fit: requires Docker, Harbor, a specific Python range, and multiple allowed hosts; environment fit is demanding with limited guidance, 1.
README has clear navigation, badges, sections, and links to docs, 2. install_notes: pip install, uv tool install harbor, and Docker prerequisites are given, 2. naming_stability: package name websitebench matches the CLI, but version 0.0.1 and Beta status with no stability commitment, 1. examples_and_faq: run examples exist but no FAQ, sample output, or troubleshooting, 1. known_limitations: only third-party content non-commercial note and canary GUID; no systematic limitations list, 1. license: full Apache-2.0 text, pyproject declares license and license-files, README distinguishes code vs content licensing, 3. versioning_changelog: version 0.0.1 with tag/version check in publish workflow, but no CHANGELOG or versioning policy, 1. maintenance_responsibility: authored as WebsiteBench Authors with no named maintainers, governance, or contact; publisher identity unverified, 1.
reward.txt output and explicit task/test-case scale make results directly comparable across agents, 2. marginal_value: 70 websites and 46,916 hidden test cases provide clear incremental value for evaluating web-rebuild agents, 2. cost_benefit: requires Docker, Harbor, external host allowances, and many concurrent tasks; no resource or cost estimates provided, 1.
claim_traceability: README claims about task counts, test-case counts, and scoring thresholds are traceable to task definitions and docs, but visible files do not show concrete task content or verifier details, 1. cross_source_corroboration: README, pyproject, CI, and task docker-compose corroborate dependency versions and workflow, but no independent third-party corroboration, 1. fact_inference_separation: docs separate code license from third-party content license and flag the canary GUID, but some capability claims (e.g., scoring strictness) lack supporting evidence, 1.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Runs require allowing multiple external hosts and downloading datasets; README does not describe sandbox boundaries or cleanup, posing external side-effect risk.
- README instructs setting OPENAI_API_KEY and running directly with no confirmation or key-usage scope.
- Tasks include sensitive scenarios like 1password, but no credential-handling or data-flow guidance is visible.
- Version 0.0.1 marked Beta with no CHANGELOG or stability commitment; interfaces may change.
- Publisher identity is unverified and maintenance responsibility/contact is unclear.
- High run cost (Docker, Harbor, many concurrent tasks) with no resource or cost estimates.