Open Operator Evals
A repeatable benchmark for measuring open web operators across real browsing tasks.
- Source repo
- nottelabs/open-operator-evals
- Stars
- ★ 47
- Last updated
- 1y ago
- Primary language
- Python
- FA score
- 29/100 · Major gaps
At a glance
- Works with
- Portable with changes
- You'll need
- Typical use
- A browser-agent team runs WebVoyager30 repeatedly before a release to check whether success rates and task times remain stable.
- Main limitation
- The documented benchmark covers only the 30-task, 15-site WebVoyager30 subset, so it does not establish performance on the full roughly 600-task dataset or other web environments.
- Source review
- 29/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
Open Operator Evals is an open-source benchmark repository for evaluating web operators, not an end-user web automation agent. It uses WebVoyager30, a 30-task subset drawn from the roughly 600-task WebVoyager dataset and spanning 15 websites. The repository reports repeated runs for Notte, Browser-Use, and Convergence using agent self-reports, GPT-4 LLM evaluation, time per task, and task reliability. Evaluations are launched through the Python module `eval.run`, and outputs are organized by dataset, provider, timestamp, and task, with JSON results and `summary.webp` replay summaries. It fits engineering and research teams that need to inspect browser-agent performance or reproduce supplied configurations, but the supplied material does not document a general installation flow or an adapter interface for arbitrary agents.
The benchmark takes configuration input from files such as configs/notte.headless.gemini and can be run with cat configs/notte.headless.gemini | uv run python -m eval.run. It runs providers against WebVoyager30; the documented evaluation limit is six minutes or 20 steps per task, with eight attempts used to measure variance. Agents return a success status and an answer, while the benchmark aggregates self-reported outcomes and applies WebVoyager’s GPT-4 evaluation prompt to judge completion. It writes outputs in dataset/provider/timestamp/task_name, where a task directory can contain results.json, results_no_screenshot.json, and summary.webp for results and replay inspection.
- A browser-agent team runs WebVoyager30 repeatedly before a release to check whether success rates and task times remain stable.
- A researcher audits the recorded results for Notte, Browser-Use, and Convergence under the versions and reasoning configurations named in the repository.
- An agent-quality engineer opens
results_no_screenshot.jsonandsummary.webpto investigate a failed task trajectory. - An evaluator dealing with high-variance web tasks uses eight attempts per task and task reliability instead of drawing conclusions from a single run.
- A team compares agent self-reports with GPT-4 judgments through Alignment and Mismatch to identify overconfident outcomes.
How do you install or deploy this agent?
The supplied material does not document cloning, dependency installation, locked dependencies, environment variables, or credential setup, so a complete verified installation procedure cannot be provided. It explicitly states Python 3.11 and uses uv; the documented invocation is cat configs/notte.headless.gemini | uv run python -m eval.run. Credentials and configuration requirements for the individual providers are not specified.
How do you use this agent?
In an environment with Python 3.11, uv, network access, and permission to write repository output files, select a supplied configuration and run cat configs/notte.headless.gemini | uv run python -m eval.run. Inspect output under dataset/provider/timestamp/task_name; use results.json or results_no_screenshot.json for results and summary.webp for the replay summary. The material gives this concrete command for Notte only, and does not explicitly document commands or credential requirements for the other providers.
What are this agent's strengths and limitations?
- Uses WebVoyager30 and eight attempts per task, directly addressing the high variance of browser-agent evaluations.
- Measures self-reporting, GPT-4 evaluation, elapsed time, reliability, Alignment, and Mismatch, separating task completion from self-assessment quality.
- Keeps task-level JSON results and
summary.webpfiles, allowing reviewers to inspect trajectories instead of relying only on aggregate scores. - Publishes per-run breakdowns for Notte, Browser-Use, and Convergence with named versions or commit and reasoning configurations.
- The documented benchmark covers only the 30-task, 15-site WebVoyager30 subset, so it does not establish performance on the full roughly 600-task dataset or other web environments.
- Installation, dependency resolution, credentials, and onboarding of an additional provider are not documented in the supplied material.
- Results can be affected by changing web content, nondeterministic models, network behavior, cookie-consent popups, CAPTCHAs, and bot detection.
- The README estimates roughly $20 for Browser-Use with GPT-4o under its WebVoyager30 eight-attempt setup; that estimate is not a general operating cost.
How does this agent compare with similar options?
The repository compares three recorded configurations: Notte v1.3.3 with gemini/gemini-2.0-flash, Browser-Use v0.1.40 with openai/gpt-4o, and Convergence Proxy-lite at commit a4389c5. In the README’s aggregate table, their LLM Evaluation scores are 79.0%, 60.2%, and 31.4%, with reported average task times of 47s, 113s, and 83s respectively. These are repository-reported repeated WebVoyager30 results, not evidence for every version, model, network environment, or task set.
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| Open Operator Evals This agent | 29 · Major gaps | ★ 47 | 1y ago | Python | — |
| Webwright Browser Coding Agent | 66 · Some gaps | ★ 6k | 1mo ago | Python | Codex · Claude Code · OpenAI API · Claude API |
| Chrome DevTools MCP | 79 · Some gaps | ★ 53k | today | TypeScript | Codex · Claude Code |
| browser-control | 72 · Some gaps | ★ 3.1k | 28d ago | Rust | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows the repository provides no documentation on permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All trust criteria are absent, hence score 0.
Self-consistency: README repeats the same result tables multiple times, but no raw data or code verification is provided, only partial support. Dependency availability: requirements.txt pins versions, but no explanation of dependency availability or maintenance, only partial support. Failure messages: No documentation of failure handling or error messages, score 0.
Audience and scenarios: Clearly targets researchers and developers, provides benchmark scenarios, score 2. Capability boundaries: Only describes benchmark scope, not agent capability limits, score 1. Trigger precision: No trigger mechanism or precise instructions provided, score 0. Environment fit: Mentions running on Macbook M1 and Python 3.11, but no other environment support, score 1.
Information architecture: README is well-structured with tables, sections, and links, score 2. Install notes: Provides run command but not full installation steps, score 1. Naming stability: Version numbers are clear, but no naming conventions, score 1. Examples and FAQ: Provides example tasks and replay links, but no FAQ, score 2. Known limitations: Mentions high variance and CAPTCHA issues, but not systematically, score 1. License: No license information, score 0. Versioning and changelog: Mentions versions but no changelog, score 1. Maintenance responsibility: No clear maintainer or contribution guidelines, score 1.
Output usability: Provides result tables and replay files, easy to use, score 2. Marginal value: Provides independent evaluation comparing different agents, valuable, score 2. Cost benefit: Provides cost estimates showing low cost, score 2.
Claim traceability: Provides result files and replay links, but not raw data, score 2. Cross-source corroboration: Mentions Browser-Use's blog, but no other independent sources, score 1. Fact-inference separation: Distinguishes self-report and LLM evaluation, but not clearly separating facts and inferences, score 1.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- The repository lacks a license, so legal risks should be assessed before use.
- Evaluation results may be affected by environmental factors (e.g., IP location), so reproduction should be cautious.
- The dependency list is large and pinned, which may contain security vulnerabilities; regular updates are needed.