ClawBench Browser Agent Benchmark
Evaluate browser agents on real-world online tasks with recorded, interceptable browser sessions.
Per-dimension scores and reasoning
Evidence shows: project uses Docker isolation and request interceptor, indicating least-privilege awareness; but user confirmation mechanism not specified, data flow transparency only partial (e.g., actions.jsonl recorded), sensitive data handling not detailed, dependency security lacks lock files or vulnerability scanning, external effects (e.g., actions on live sites) have interceptor but no explicit user consent, rollback not mentioned, source attribution has author info but unverified. Deductions: lack of user confirmation, rollback, insufficient evidence for sensitive data handling and dependency security.
Evidence shows: project has tests (e.g., test_answer_judge.py) and CI workflows, indicating some consistency; dependencies listed in pyproject.toml but no lock file, availability not fully guaranteed; failure messages not detailed in provided files. Deductions: missing dependency lock, insufficient evidence for failure messages.
Evidence shows: project targets AI agent developers, provides multiple scenarios (V1/V2 tasks), capability boundaries via task definitions and harness configs, trigger precision not explicit, environment fit supports Docker and Browserbase. Deductions: trigger precision not explicit, environment fit details limited.
Evidence shows: README well-structured, install notes detailed, naming stable (clawbench commands), examples and FAQ present, known limitations not explicit, license Apache-2.0, CHANGELOG and versioning present, maintenance responsibility has author info but unverified. Deductions: known limitations not explicit, maintenance responsibility unverified.
Evidence shows: outputs include task results and scoring, marginal value (benchmark), cost-benefit not detailed. Deductions: insufficient evidence for cost-benefit.
Evidence shows: README claims have arXiv links and datasets, but no specific evidence files; cross-source corroboration limited, fact-inference separation not explicit. Deductions: insufficient cross-source corroboration and fact-inference separation.
- No dependency lock file provided, supply chain risk needs attention.
- User confirmation and rollback mechanisms absent, caution when operating on live sites.
- Sensitive data handling details insufficient, needs review.
What does this agent do, and when should you use it?
ClawBench is an open-source benchmark for evaluating AI browser agents on everyday online tasks such as food ordering, travel booking, job applications, and email management. It provides 153 V1 tasks and 130 V2 tasks across 144 websites, plus task definitions, rubrics, metadata, a 20-task V1 Lite subset, and a 19-task Claw-Eval suite. Each run starts Chromium in an isolated Docker or Podman environment and captures browser activity through CDP. The resulting artifact bundle includes video or a remote replay URL, screenshots, browser actions, HTTP traffic, agent messages, and interception data for later review. Its CLI exposes interactive, single-task, batch, rescore, and Harbor-adaptation workflows through commands including `clawbench`, `clawbench-run`, and `clawbench-batch`.
A user selects a case from test-cases/v1/, test-cases/v2/, test-cases/v1-lite/, or test-cases/claw-eval/, and ClawBench starts its containerized Chromium runtime. The agent drives the browser while a FastAPI server records CDP Fetch, Runtime, and Page events; critical outbound requests can be intercepted using url_pattern, method, and optional body or parameter criteria. Runs are written under test-output/ with recording.mp4 or a replay URL in run-meta.json, screenshots/, actions.jsonl, requests.jsonl, agent-messages.jsonl, and interception.json. Batch output can be evaluated later with clawbench-rescore, which compares agent trajectories with human reference trajectories and produces evidence-backed PASS/FAIL decisions.
- A browser-agent team comparing models or harnesses on end-to-end tasks performed on live consumer websites.
- A research group downloading V1 or V2 task definitions and full execution traces to inspect or reproduce browser-agent evaluation.
- A model-platform engineer using
clawbench-batchto run a configured model across the complete V2 corpus and collect batch summaries. - An evaluation engineer creating a human reference trajectory through
--humanbefore reviewing agent behavior against it. - A Harbor Framework user converting V2 cases into a local Harbor dataset and connecting an external agent through CDP.
- A researcher watching an active single-task session through the noVNC URL emitted by a run.
What are this agent's strengths and limitations?
- Executes everyday workflows on live websites rather than self-hosted replicas, while intercepting matched irreversible requests such as checkout, form submission, and email sending.
- Captures five synchronized evidence layers—video, screenshots, browser actions, HTTP traffic, and agent messages—for inspection and re-grading.
- Combines runnable tasks, task and trace datasets, CLI evaluation, human-reference mode, and Harbor dataset export instead of providing only a leaderboard.
- Documents multiple harnesses, including Codex, Claude Code, OpenClaw, browser-use, Hermes, and Pi, with Docker or Podman container support.
- Adoption requires Python 3.11+, uv, and Docker or Podman; the first run builds a container image that includes Chromium and related components.
- Because tasks use live websites, changes in site flows, access, login behavior, or availability can affect runs and reproducibility.
- Consistent task-correctness judgement requires credentials and a compatible endpoint for
deepseek-v4-pro. - For payment-wall tasks using a deliberately non-matching interception pattern, sessions run until timeout instead of yielding a mechanically intercepted request.
How do you install or deploy this agent?
For standard installation, run uv tool install clawbench-eval. The documented alternatives are pipx install clawbench-eval and python -m pip install clawbench-eval. Requirements are Python 3.11+, uv, and Docker or Podman. On first launch, clawbench creates local templates under models/; configure the model there. For task judgement, configure deepseek-v4-pro in models/models.yaml with api_key, base_url, and api_type: openai-completions. From a source checkout, first run cp models/models.example.yaml models/models.yaml and edit that file.
How do you use this agent?
Launch the interactive interface with clawbench, or uv run clawbench from a source checkout. Run one task with uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6. Run the V2 corpus with clawbench-batch --models your-model --cases-suite v2 --all-cases. Add --no-judge when collecting traces without scoring, then rescore with clawbench-rescore ./my-run --judge-model deepseek-v4-pro --rubric both. To produce a human reference run, use uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats --human.
How does this agent compare with similar options?
ClawBench positions itself around end-to-end browser tasks on live consumer websites. Unlike the self-hosted environments of WebArena, VisualWebArena, and TheAgentCompany, it uses real sites; unlike Mind2Web, it executes dynamic online tasks rather than relying on static traces; compared with WebVoyager, it emphasizes interception-based grading and five-layer run recordings.
FAQ
Will a run actually place an order, make a payment, or send an email?
What credentials are needed for evaluation?
models/models.yaml. The documented judgement setup also requires a deepseek-v4-pro API key, base URL, and OpenAI-completions API type for consistent correctness decisions.What does local execution require?
CONTAINER_ENGINE explicitly.Can I validate scoring without rerunning an agent?
clawbench-rescore, or use clawbench-reproduce to check a leaderboard model against the documented tolerance.