Open Operator Evals
A repeatable benchmark for measuring open web operators across real browsing tasks.
- Source repo
- nottelabs/open-operator-evals
- Stars
- ★ 47
- Last updated
- 1y ago
- Primary language
- Python
- FA score
- 44/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changes
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A browser-agent team runs WebVoyager30 repeatedly before a release to check whether success rates and task times remain stable.
- Not a fit if
- Teams needing full ~600-task WebVoyager coverage
- Teams wanting documented install or arbitrary agent adapters
- Source review
- 44/100 · Major gaps 3 safety controls not found
What does this agent do, and when should you use it?
Open Operator Evals is an open-source benchmark repository for evaluating web operators, not an end-user web automation agent. It uses WebVoyager30, a 30-task subset drawn from the roughly 600-task WebVoyager dataset and spanning 15 websites. The repository reports repeated runs for Notte, Browser-Use, and Convergence using agent self-reports, GPT-4 LLM evaluation, time per task, and task reliability. Evaluations are launched through the Python module eval.run, and outputs are organized by dataset, provider, timestamp, and task, with JSON results and summary.webp replay summaries. It fits engineering and research teams that need to inspect browser-agent performance or reproduce supplied configurations, but the supplied material does not document a general installation flow or an adapter interface for arbitrary agents.
The benchmark takes configuration input from files such as configs/notte.headless.gemini and can be run with cat configs/notte.headless.gemini | uv run python -m eval.run. It runs providers against WebVoyager30; the documented evaluation limit is six minutes or 20 steps per task, with eight attempts used to measure variance. Agents return a success status and an answer, while the benchmark aggregates self-reported outcomes and applies WebVoyager’s GPT-4 evaluation prompt to judge completion. It writes outputs in dataset/provider/timestamp/task_name, where a task directory can contain results.json, results_no_screenshot.json, and summary.webp for results and replay inspection.
- A browser-agent team runs WebVoyager30 repeatedly before a release to check whether success rates and task times remain stable.
- A researcher audits the recorded results for Notte, Browser-Use, and Convergence under the versions and reasoning configurations named in the repository.
- An agent-quality engineer opens
results_no_screenshot.jsonandsummary.webpto investigate a failed task trajectory. - An evaluator dealing with high-variance web tasks uses eight attempts per task and task reliability instead of drawing conclusions from a single run.
- A team compares agent self-reports with GPT-4 judgments through Alignment and Mismatch to identify overconfident outcomes.
How do you install or deploy this agent?
The supplied material does not document cloning, dependency installation, locked dependencies, environment variables, or credential setup, so a complete verified installation procedure cannot be provided. It explicitly states Python 3.11 and uses uv; the documented invocation is cat configs/notte.headless.gemini | uv run python -m eval.run. Credentials and configuration requirements for the individual providers are not specified.
How do you use this agent?
In an environment with Python 3.11, uv, network access, and permission to write repository output files, select a supplied configuration and run cat configs/notte.headless.gemini | uv run python -m eval.run. Inspect output under dataset/provider/timestamp/task_name; use results.json or results_no_screenshot.json for results and summary.webp for the replay summary. The material gives this concrete command for Notte only, and does not explicitly document commands or credential requirements for the other providers.
What are this agent's strengths and limitations?
- Uses WebVoyager30 and eight attempts per task, directly addressing the high variance of browser-agent evaluations.
- Measures self-reporting, GPT-4 evaluation, elapsed time, reliability, Alignment, and Mismatch, separating task completion from self-assessment quality.
- Keeps task-level JSON results and
summary.webpfiles, allowing reviewers to inspect trajectories instead of relying only on aggregate scores. - Publishes per-run breakdowns for Notte, Browser-Use, and Convergence with named versions or commit and reasoning configurations.
- The documented benchmark covers only the 30-task, 15-site WebVoyager30 subset, so it does not establish performance on the full roughly 600-task dataset or other web environments.
- Installation, dependency resolution, credentials, and onboarding of an additional provider are not documented in the supplied material.
- Results can be affected by changing web content, nondeterministic models, network behavior, cookie-consent popups, CAPTCHAs, and bot detection.
- The README estimates roughly $20 for Browser-Use with GPT-4o under its WebVoyager30 eight-attempt setup; that estimate is not a general operating cost.
How does this agent compare with similar options?
The repository compares three recorded configurations: Notte v1.3.3 with gemini/gemini-2.0-flash, Browser-Use v0.1.40 with openai/gpt-4o, and Convergence Proxy-lite at commit a4389c5. In the README’s aggregate table, their LLM Evaluation scores are 79.0%, 60.2%, and 31.4%, with reported average task times of 47s, 113s, and 83s respectively. These are repository-reported repeated WebVoyager30 results, not evidence for every version, model, network environment, or task set.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Open Operator Evals This agent | 44 · Major gaps | CLIFree + model costs | ★ 47 | 1y ago | Python | — |
| WebsiteBench | 61 · Some gaps | CLIFree + model costs | ★ 124 | 1d ago | Python | — |
| OpenBrowser | 51 · Major gaps | CLIFreemium | ★ 279 | 1mo ago | Python | Codex · Claude Code · Claude.ai · OpenAI API · Claude API |
| Webwright Browser Coding Agent | 53 · Major gaps | CLIFree + model costs | ★ 6.1k | 12d ago | Python | Codex · Claude Code · OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
The README identifies browser agents, an LLM evaluator, and web tasks, and points to inspectable trajectories and result files, giving partial visibility into data flows (data_flow_transparency=1) and the scope of external effects (external_effects=1). It gives no permissions, confirmation, sensitive-data, or rollback guidance, so those criteria score 0–1. requirements.txt pins versions but provides no security review or mitigation evidence (dependency_security=1). The README names evaluated projects, versions, and the evaluation prompt (source_attribution=2); publisher identity remains unknown.
The README specifies task counts, retries, time and step limits, and environment; requirements.txt pins dependency versions. This supports partial consistency and dependency availability (self_consistency=1, dependency_availability=2). There is no failure handling or troubleshooting guidance. Also, claims such as “objective” evaluation need qualification because the judge is an LLM, so failure_messages=1.
The README defines the use as web-agent evaluation and describes WebVoyager, a 30-task subset, multiple websites, and repeated runs, making its audience and trigger scope fairly clear (audience_and_scenarios=2, trigger_precision=2). Supported agent interfaces, capability boundaries, and failure conditions are not fully specified (capability_boundaries=1). Python version, Mac M1, headless mode, a run command, and cost estimates provide useful but partial environment guidance (environment_fit=2).
The README has clear sections for metrics, breakdowns, configuration, reproduction, costs, and replays (information_architecture=2). It gives a command and output layout, but installation instructions are thin (install_notes=1). Terminology and provider names are mostly stable (naming_stability=2); metric definitions, reproduction instructions, and replay examples are useful (examples_and_faq=2). It discloses changing web content, variance, CAPTCHA, and costs (known_limitations=2). License metadata is unknown and no license is shown (license=0). Evaluated project versions are named, but no project release history or changelog is provided (versioning_changelog=1). A hiring link does not establish maintenance responsibility or an update path (maintenance_responsibility=1).
The benchmark table, per-run results, and replay paths make outputs readable and inspectable (output_usability=2). Repeated runs, LLM judging, and public trajectories add useful comparative value for web-agent evaluation (marginal_value=2). The README states the subset size, eight runs, and estimated costs, providing a practical cost-benefit account, but gives no independent evidence that the subset is representative or the estimates are accurate (cost_benefit=2).
Specific scores, per-run values, configuration, a run command, and replay paths make many claims traceable (claim_traceability=2). Tables, prose, and per-run data partially corroborate one another, but all are within the same README and lack independent verification (cross_source_corroboration=1). Metric definitions and limitations are described; claims that the approach is “more statistically reliable” exceed the statistical argument shown, so fact and inference are separated reasonably but not fully (fact_inference_separation=2).
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- The packet provides no license, full installation steps, maintenance ownership, or sensitive-data handling guidance.
- Performance and representativeness claims rely mainly on the same README benchmark material; the claim of greater statistical reliability is not fully supported by statistical analysis.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/open-operator-evals)<a href="https://followagents.com/en/agents/open-operator-evals"><img src="https://followagents.com/badge/open-operator-evals.svg" alt="FARS review"></a>FAQ
Is this a browser agent I can use to complete my own web tasks?
How long and how much does an evaluation cost?
Why does the benchmark run a task multiple times?
What failures should I expect during reproduction?
Thanks — we use this to improve the page.