ClawBench Browser Agent Benchmark
Evaluate browser agents on real-world online tasks with recorded, interceptable browser sessions.
What does this agent do, and when should you use it?
ClawBench is an open-source benchmark for evaluating AI browser agents on everyday online tasks such as food ordering, travel booking, job applications, and email management. It provides 153 V1 tasks and 130 V2 tasks across 144 websites, plus task definitions, rubrics, metadata, a 20-task V1 Lite subset, and a 19-task Claw-Eval suite. Each run starts Chromium in an isolated Docker or Podman environment and captures browser activity through CDP. The resulting artifact bundle includes video or a remote replay URL, screenshots, browser actions, HTTP traffic, agent messages, and interception data for later review. Its CLI exposes interactive, single-task, batch, rescore, and Harbor-adaptation workflows through commands including `clawbench`, `clawbench-run`, and `clawbench-batch`.
A user selects a case from test-cases/v1/, test-cases/v2/, test-cases/v1-lite/, or test-cases/claw-eval/, and ClawBench starts its containerized Chromium runtime. The agent drives the browser while a FastAPI server records CDP Fetch, Runtime, and Page events; critical outbound requests can be intercepted using url_pattern, method, and optional body or parameter criteria. Runs are written under test-output/ with recording.mp4 or a replay URL in run-meta.json, screenshots/, actions.jsonl, requests.jsonl, agent-messages.jsonl, and interception.json. Batch output can be evaluated later with clawbench-rescore, which compares agent trajectories with human reference trajectories and produces evidence-backed PASS/FAIL decisions.
- A browser-agent team comparing models or harnesses on end-to-end tasks performed on live consumer websites.
- A research group downloading V1 or V2 task definitions and full execution traces to inspect or reproduce browser-agent evaluation.
- A model-platform engineer using `clawbench-batch` to run a configured model across the complete V2 corpus and collect batch summaries.
- An evaluation engineer creating a human reference trajectory through `--human` before reviewing agent behavior against it.
- A Harbor Framework user converting V2 cases into a local Harbor dataset and connecting an external agent through CDP.
- A researcher watching an active single-task session through the noVNC URL emitted by a run.
What are this agent's strengths and limitations?
- Executes everyday workflows on live websites rather than self-hosted replicas, while intercepting matched irreversible requests such as checkout, form submission, and email sending.
- Captures five synchronized evidence layers—video, screenshots, browser actions, HTTP traffic, and agent messages—for inspection and re-grading.
- Combines runnable tasks, task and trace datasets, CLI evaluation, human-reference mode, and Harbor dataset export instead of providing only a leaderboard.
- Documents multiple harnesses, including Codex, Claude Code, OpenClaw, browser-use, Hermes, and Pi, with Docker or Podman container support.
- Adoption requires Python 3.11+, uv, and Docker or Podman; the first run builds a container image that includes Chromium and related components.
- Because tasks use live websites, changes in site flows, access, login behavior, or availability can affect runs and reproducibility.
- Consistent task-correctness judgement requires credentials and a compatible endpoint for `deepseek-v4-pro`.
- For payment-wall tasks using a deliberately non-matching interception pattern, sessions run until timeout instead of yielding a mechanically intercepted request.
How do you install or deploy this agent?
For standard installation, run uv tool install clawbench-eval. The documented alternatives are pipx install clawbench-eval and python -m pip install clawbench-eval. Requirements are Python 3.11+, uv, and Docker or Podman. On first launch, clawbench creates local templates under models/; configure the model there. For task judgement, configure deepseek-v4-pro in models/models.yaml with api_key, base_url, and api_type: openai-completions. From a source checkout, first run cp models/models.example.yaml models/models.yaml and edit that file.
How do you use this agent?
Launch the interactive interface with clawbench, or uv run clawbench from a source checkout. Run one task with uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6. Run the V2 corpus with clawbench-batch --models your-model --cases-suite v2 --all-cases. Add --no-judge when collecting traces without scoring, then rescore with clawbench-rescore ./my-run --judge-model deepseek-v4-pro --rubric both. To produce a human reference run, use uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats --human.
How does this agent compare with similar options?
ClawBench positions itself around end-to-end browser tasks on live consumer websites. Unlike the self-hosted environments of WebArena, VisualWebArena, and TheAgentCompany, it uses real sites; unlike Mind2Web, it executes dynamic online tasks rather than relying on static traces; compared with WebVoyager, it emphasizes interception-based grading and five-layer run recordings.