Hercules Testing Agent
World's first open-source testing agent enabling UI, API, security, accessibility, and visual validations without code.
- Source repo
- test-zeus-ai/testzeus-hercules
- Stars
- ★ 1.2k
- Last updated
- 5d ago
- License
- AGPL-3.0
- Primary language
- Python
- FA score
- 48/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changes
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- QA engineers who want to write end-to-end tests in plain Gherkin without coding
- Not a fit if
- Teams unwilling to pay per-use LLM API costs
- Users who need a GUI instead of a terminal
- Source review
- 48/100 · Major gaps 1 safety controls not found
What does this agent do, and when should you use it?
Hercules is an open-source testing agent that turns Gherkin feature files into automated end-to-end tests. Built on a LangGraph state machine, it coordinates planner, executor, and assertion nodes to route actions to navigation agents for browser, API, security, SQL, MCP, and more. Users write tests in Gherkin; Hercules executes them and produces JUnit XML and HTML reports, along with video recordings, screenshots, and network logs as proof. It offers multiple deployment paths: PyPI package, Docker image, or from source. Additionally, Hercules can run as an MCP server, exposing tools like generate_gherkin and run_test to MCP clients.
It reads Gherkin feature files, parses them into strict JSON plans, and uses a PlannerAgent to decide the next step. The executor routes to the appropriate navigation agent—such as browser_nav_agent, api_nav_agent, sec_nav_agent—which bind LangChain StructuredTools (open_url, click, bulk_enter_text) to interact with browsers or APIs. It integrates security scanning via Nuclei and accessibility checks per WCAG standards. After execution, it generates JUnit XML and HTML reports, and saves artifacts like screenshots, videos, and network logs. It also supports MCP servers for tool discovery and can act as an MCP server itself.
- QA engineers who want to write end-to-end tests in plain Gherkin without coding
- DevOps teams integrating Hercules into CI/CD pipelines using the Docker image for automated regression testing
- Security testers who need to run vulnerability scans (Nuclei) from Gherkin test cases
- Accessibility specialists verifying WCAG 2.0/2.1/2.2 compliance at A, AA, AAA levels
- Teams that want to extend functionality with custom Python scripts or community tools via ADDITIONAL_TOOL_DIRS
- Organizations needing remote browser execution (BrowserStack, LambdaTest, BrowserBase) for scalable testing
How do you install or deploy this agent?
Install Python 3.11–3.13, then run: pip install testzeus-hercules; next, run playwright install --with-deps to set up browsers. Alternatively, use Docker: docker pull testzeus/hercules:latest. For source-based installation, clone the repository, install UV, and run make setup-uv && make install. You'll need to configure LLM environment variables (LLM_MODEL_NAME, LLM_MODEL_API_KEY, LLM_MODEL_BASE_URL) or optionally use an agents_llm_config.json for per-agent model routing.
How do you use this agent?
Create a Gherkin feature file (e.g., opt/input/test.feature). Set environment variables like LLM_MODEL_NAME=gpt-4o and LLM_MODEL_API_KEY=your-key. Run testzeus-hercules --project-base=opt or specify --input-file, --output-path, and --test-data-path. After execution, check the output directory for JUnit XML and HTML reports. For Docker, run: docker run --env-file=.env -v ./opt:/testzeus-hercules/opt --rm -it testzeus/hercules:latest. Results appear in the mounted opt folder.
What are this agent's strengths and limitations?
- No-code testing: write tests in Gherkin, no programming required
- Multi-type coverage: UI, API, security (Nuclei), accessibility, and visual checks
- Flexible deployment: PyPI package, Docker, or source
- Rich execution evidence: videos, screenshots, network logs
- MCP integration: can connect to MCP servers and act as one
- Auto-healing capabilities reduce maintenance
- Requires LLM API access, incurring cost (approx. $0.20 per complex use case)
- Configuration overhead: model keys and environment variables needed
- Playwright browser installation required (playwright install)
- Windows users need extra setup script (hercules_windows_setup.ps1)
- Telemetry enabled by default, but can be disabled
How does this agent compare with similar options?
Hercules differentiates from traditional test automation frameworks (e.g., Selenium, Cypress) and LLM-driven approaches by being open-source, code-free, and covering multiple test types. Specific comparisons to other agents are not detailed in the source.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Hercules Testing Agent This agent | 48 · Major gaps | CLIFree + model costs | ★ 1.2k | 5d ago | Python | — |
| Emdash Skills | 53 · Major gaps | Agent plugin / skillFree + model costs | ★ 24 | 8d ago | TypeScript | Codex · Claude Code |
| Agentic Architectures | 54 · Major gaps | Library / SDKFree + model costs | ★ 4.6k | 4mo ago | Jupyter Notebook | OpenAI API · Claude API |
| Camofox Browser | 52 · Major gaps | Self-hosted serviceFree | ★ 12k | 14d ago | JavaScript | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: README mentions telemetry enabled by default but does not explain how to fully disable; environment variables control browser, video, etc., but least privilege principle is not explicit; user confirmation mechanism (e.g., AUTO_MODE env var) exists but not detailed; data flow transparency is limited, telemetry data flow not detailed; sensitive data handling not explicit; dependency security has some measures (e.g., override-dependencies in pyproject.toml) but no full vulnerability scan report; external effects (e.g., browser actions) are described but no rollback mechanism; source attribution has author info but publisher unverified. Deductions: lack of explicit least privilege design, opaque user confirmation, unclear data flow, unclear sensitive data handling, insufficient dependency security evidence, no rollback, unverified publisher.
Evidence shows: README and pyproject.toml are mostly consistent, but there are minor inconsistencies (e.g., classifiers include Python 3.10 while requires-python is >=3.11); dependency list is complete but no lock file details; failure messages have basic description (e.g., logs and reports) but no detailed error handling docs. Deductions: minor self-consistency issues, dependency availability not fully verified, failure messages not detailed enough.
Evidence shows: README provides multiple installation methods (PyPI, Docker, source) and multiple use cases (UI, API, security, visual testing), audience is clear; capability boundaries are described (e.g., supported models and browsers) but not detailed; trigger precision is described (e.g., Gherkin steps) but no detailed trigger condition docs; environment fit is described (e.g., Windows, Docker, remote browsers) but not all environments detailed. Deductions: capability boundaries and trigger precision docs not detailed enough.
Evidence shows: README has clear structure with installation, usage, configuration, output sections; install notes are detailed with multiple methods; naming is stable (e.g., testzeus-hercules command); examples and FAQ provided (video tutorials); known limitations not explicitly listed; license is AGPL-3.0 but full text not provided (only partial); versioning and changelog not provided; maintenance responsibility has author info but no explicit maintenance strategy. Deductions: known limitations, versioning/changelog missing, license text incomplete.
Evidence shows: Output includes JUnit XML, HTML reports, screenshots, videos, etc., high usability; marginal value is high as it provides automated testing; cost-benefit not detailed but multiple installation methods provided. Deductions: cost-benefit analysis insufficient.
Evidence shows: Claims in README (e.g., 'world's first open-source testing agent') lack independent verification; cross-source verification limited, relying mainly on README and some code; facts and inferences not clearly separated. Deductions: claims lack traceability, cross-source verification insufficient, facts and inferences mixed.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Telemetry is enabled by default; verify if it can be fully disabled and where data goes.
- Publisher identity is unverified; assess supply chain risks carefully.
- Dependency security measures are limited; check for known vulnerabilities.
- No rollback mechanism; be cautious of external effects during test execution.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
Shahal Thayyil (1): feat(telemetry): capture folder paths and document telemetry collection (#148) Shriyansh Agnihotri (1): release: version 1.0.2 🚀 rmudukanagoudra (2): chore: Update packages, fix formatting and remove mem agent (#145) release: version 1.0.1 🚀 (#147)View on GitHub ↗ -
Release notes excerpt
Shahal Thayyil (1): release: version 1.0.0 🚀 (#142) rmudukanagoudra (2): chore: add litellm integration to readme (#144) release: version 1.0.1 🚀View on GitHub ↗ -
1.0.0 TestZeus Hercules 1.0.0
Release notes excerpt
Hercules 1.0.0 moves the agent runtime from AG2 to LangGraph and introduces a more explicit, dependable execution model for AI-driven end-to-end testing. The public workflow remains the same—provide Gherkin and receive test reports—while planning, helper execution, tool calling, and assertions now run through a structured state graph. Highlights LangGraph orchestration - Replaced the AG2 group-chat runtime with a LangGraph StateGraph composed of explicit planner, executor, and assertion nodes. …View on GitHub ↗