assay
Agent-native QA CLI that opens a local web page in Chromium, exercises every control it finds, and reports self-contradictions — deterministically, with no tests to write and no LLM.
- Source repo
- awss1i/assay
- Stars
- ★ 110
- Last updated
- 1d ago
- License
- MIT
- Primary language
- Python
- FA score
- 61/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformCodex · Claude Code
- Cost
- Free, no paid service needed
- Setup effort
- Low · running in minutes
- You'll need
- Typical use
- A coding agent in Claude Code or Codex finishes a static page; the skill or plugin runs assay automatically so obvious self-contradictions are caught before reporting back.
- Not a fit if
- Teams that need to verify business-logic intent, not just self-contradiction
- Test scenarios that require crawling multiple pages or logging in
- Users expecting it to run npm install or your build step
- Source review
- 61/100 · Some gaps
What does this agent do, and when should you use it?
assay is a Python CLI that performs end-to-end quality checks on a local web page without any hand-written test cases. It serves the target folder on localhost, opens the page in Chromium, waits for controls to settle, then enumerates every rendered control, including clickable divs, drag handles and editable regions. From those controls it plans checks — pressing buttons once and twice, typing empty, ordinary, very long and HTML-laden text, changing dropdowns, drawing on canvases, filling and submitting forms — and runs each one in a fresh tab, comparing the page before and after. Instead of guessing intent, it reports where the page contradicts itself, such as throwing an error, Undo failing on the first press, an empty list counted as 1, or NaN appearing. No API key or model is involved, so runs are repeatable and free. Beyond the CLI, it ships a skill and a hook-based plugin for fifteen coding harnesses, plus a Python API for passing acceptance criteria.
On invocation, assay starts a local HTTP server for the folder, loads the page in Chromium and waits until the rendered controls stop changing. It then lists every interactive element the browser exposes, including clickable divs, drag handles and editable regions, and plans checks from them: pressing every button once and twice, typing empty, ordinary, very long and HTML-laden text into fields, changing every dropdown, drawing on canvases, and filling in and submitting forms. Each check runs in a fresh tab and compares the page state before and after the action. Findings are limited to self-contradictions — an error thrown, Undo doing nothing on the first press but working on the second, an empty list showing a count of 1, a total rising when an item is added but not falling when removed, NaN where a number should be, or nothing on the page responding to any control. Repeats collapse into messages like "same finding as C003". Output can be a short --one-line summary for scripts and agents or an HTML report with screenshots via --report; the exit code is non-zero when anything fails, and a Python API accepts acceptance criteria for intent checks.
- A coding agent in Claude Code or Codex finishes a static page; the skill or plugin runs assay automatically so obvious self-contradictions are caught before reporting back.
- A frontend developer points assay at a built HTML folder for a fast smoke check without authoring any Playwright specs.
- A CI script calls assay --one-line and fails the job on a non-zero exit code when the page throws errors or stops responding.
- A solo developer inheriting an AI-generated small web app uses assay to surface dead buttons, wrong empty-state counts and similar defects.
- A team that needs deterministic, repeatable checks without an API key or model call uses assay as a fixed step in its workflow.
How do you install or deploy this agent?
Requires Python 3.10+; Chromium is downloaded on the first run.
pip install assay-uiIf you don't have Python 3.10+, uv can fetch an interpreter:
uv tool install assay-uiOr install into an isolated environment with pipx:
pipx install assay-uiIf pip isn't found, use python3 -m pip install assay-ui. To work on assay itself:
git clone https://github.com/awss1i/assay.git && cd assay
pip install -e ".[dev]"How do you use this agent?
Point the CLI at a folder containing index.html:
assay ./my-appWrite an HTML report with screenshots as well:
assay ./my-app --report out.htmlGet a short summary for scripts and agents:
assay ./my-app --one-lineThe exit code is non-zero if anything failed. In Claude Code you can add the plugin so a hook runs the check after every turn that changed a page:
/plugin marketplace add awss1i/assay
/plugin install assay@assayFor the other supported harnesses, install the skill or plugin following the harness setup docs.
What are this agent's strengths and limitations?
- No test authoring: checks are planned from the controls actually rendered on the page.
- No LLM, API key or reference screenshot, so results are identical run to run and free.
- Reports only self-contradictions (NaN, wrong counts, broken Undo) and flagged none of 205 working benchmark pages.
- Collapses repeated root causes into "same finding as Cxxx" instead of repeating the message.
- Offers a CLI, an HTML report with screenshots, a --one-line summary and a Python API for acceptance criteria.
- Ships a skill for fifteen coding harnesses and a hook-based plugin for automatic post-turn checks.
- Cannot judge intent: a page that runs but computes a total wrongly usually passes.
- Checks one page at a time from a local folder; it doesn't crawl or log in, and the page keeps internet access during the run.
- Doesn't run npm install or your build — you must supply built output.
- A finished game with no restart path can be falsely reported as nothing responding.
- Found only 12 of 50 planted bugs in its own in-sample benchmark.
- First run needs network access for the Chromium download, and page checks take seconds to a minute.
How does this agent compare with similar options?
The README positions assay against three alternatives: Playwright and Cypress require someone to write tests; Percy and Chromatic require a reference baseline screenshot; asking a model to review the code reads source rather than running it. assay's claim is that it needs no prepared tests, no baseline, and actually runs the whole page.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| assay This agent | 61 · Some gaps | CLIFree | ★ 110 | 1d ago | Python | Codex · Claude Code |
| Webcmd | 63 · Some gaps | CLIFree + model costs | ★ 2.8k | 18d ago | TypeScript | Codex · Claude Code |
| agent-qa | 49 · Major gaps | CLIFree + model costs | ★ 856 | today | TypeScript | Codex · Claude Code |
| Playwright Browser Automation MCP | 60 · Some gaps | MCP serverFree | ★ 39k | 1d ago | TypeScript | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
SECURITY.md and tests/test_browser.py show loopback-only serving, hidden-file and out-of-folder symlink refusal, which is solid least privilege; however the browser runs without sandbox when it cannot start (warning only), and pages can reach the internet and exfiltrate served files, leaving sensitive-data handling to user discipline, so those are not full marks. The plugin hook runs assay automatically each turn rather than per-run confirmation, capping user_confirmation at 1. Dependency is only playwright>=1.40 with no pin or lock, so dependency_security is thin. Rollback beyond exit codes and no-disk-write is undocumented. Attribution is only the author name awss1i with an unverified publisher, hence 1.
Tests cover CLI targeting, missing pages, missing browser, and sandbox-off paths with specific readable messages, so failure_messages earns 3. README, usage, and tests are mutually consistent, giving self_consistency 2. Dependency availability relies on first-run Chromium download with no offline or pinned-version story, so 1.
README states limits clearly (no intent checking, web pages only, no build, finished-game false positives), so capability_boundaries is 3. Scenarios for coding agents and 15 harnesses are clear, but trigger precision depends on hooks and --if-page behavior with limited evidence, so 2. Environment fit covers Python 3.10-3.14 and pip/uv/pipx but needs Chromium and network, so 2.
Docs are well structured, multilingual, with complete install notes; known limitations are thorough and the license is full MIT text, so 3. Version appears only as 0.3.1 in pyproject and the changelog merely points to releases with no embedded record, so 1. No API stability promise, so naming_stability 1. Maintenance responsibility points only to Issues/Discussions with an unverified publisher, so 1.
Outputs include exit codes, --json, --one-line, HTML report with screenshots, so output_usability is 2. Marginal value is running pages without writing tests, but the benchmark shows only 12/50 planted bugs found and is in-sample, so 2. No LLM cost and seconds-to-a-minute per page make cost_benefit reasonable at 2.
Benchmark numbers are claimed to be written by scoring scripts, but the provided files contain no bench scripts or results, so claim_traceability is 1. Cross-source corroboration is limited to README/usage consistency with no independent source, so 1. Fact and inference are reasonably separated, with in-sample and limits explicitly flagged, so 2.
- Chromium runs without sandbox when it cannot start, printing only a warning; risk rises under root or in containers.
- Checked pages can reach the internet and exfiltrate files served from the folder; keep secrets and .env out of the checked directory.
- The plugin hook runs assay automatically each turn rather than on explicit confirmation; verify it does not misfire or clutter agent conversations.
- Dependency is only playwright>=1.40 with no pin, so build reproducibility and supply-chain evidence are weak.
- The planted-bug benchmark finds only 12/50 and is in-sample, so real detection ability may be overstated.
- Publisher identity is unverified, leaving maintenance responsibility and update path unclear.