Argus Agent Red Team
Attack your AI agent before someone else does with 266 bundled red-team probes over tools, content and requests.
- Source repo
- gy15901580825/Argus
- Stars
- ★ 216
- Last updated
- today
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 74/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A security team screening a chatbot or tool-using agent before release and uploading the SARIF into GitHub Code Scanning.
- Not a fit if
- Teams that want every verdict decided without any cloud LLM judge
- Users validating macOS-only skill behaviour (scan-skill runs in a Linux container)
- Teams wanting an inline runtime guard for production traffic
- Website
- gy15901580825.github.io/Argus ↗ set on GitHub
- Maintenance
- Commits in the last 30 dayslast commit 2026-11-20latest release v1.19.0 (2026-11-19)+1.0 ★/day over 7 days
- Source review
- 74/100 · Some gaps
What does this agent do, and when should you use it?
Argus is an open-source red-team testing framework for AI agents, distributed on PyPI as argus-probe, with a bundled library of 266 adversarial probes mapped to OWASP LLM Top 10, MITRE ATLAS and NIST AI RMF. It treats the target agent as a black box and reaches it through 15 target adapters such as openai_compat, anthropic_native, mcp_agent, browser_use, a2a_agent and vllm_server. Verdicts come either from an LLM judge with per-probe rubrics or from deterministic checks on recorded browser requests, MCP tool calls and mock x402 payments. Beyond the probe library it ships three focused subcommands: mcp-trap for running your agent against a hostile MCP server, scan-mcp for auditing an MCP server's tools, and scan-skill for sandboxing a Claude Code skill before installation. Findings are emitted as SARIF 2.1.0, JUnit XML or HTML, ready for GitHub Code Scanning or a CI gate via the argus-probe-action GitHub Action. Local mode runs everything in-process, so the CLI does not need a self-hosted server.
Argus loads YAML probes from argus/probes/, each with prompts, an optional conversation flag, severity, mappings (atlas, owasp_llm, owasp_agentic, nist_ai_rmf, eu_ai_act) and a judge.rubric_path. At run time it picks an adapter based on the target JSON's kind, sends the attack prompts, and judges replies against rubrics in argus/rubrics/ using a fast model (Anthropic Haiku 4.5 by default) with a strong-model re-judge (Sonnet 4.6) when the first pass reports high/critical with confidence ≥ 0.7; OpenAI, litellm-backed providers and TypeSafe Jev are also supported. The mcp_agent probes, most payment probes and several garak wrappers skip the judge and assert against recorded MCP interactions or payment testbed results. Reports come out as SARIF 2.1.0, JUnit XML or HTML with per-standard coverage and, for probes with captured evidence, an evidence section next to the verdict. Companion commands include mcp-trap (hostile MCP server scenarios), scan-mcp (read-only tool calls against an MCP server), scan-skill (runs a Claude Code skill in a throwaway container and records file reads, commands and outbound requests), and draft-target (drafts a target spec from a local OpenAPI file).
- A security team screening a chatbot or tool-using agent before release and uploading the SARIF into GitHub Code Scanning.
- Developers building on MCP who want to check whether their agent falls for tool poisoning, rug pulls, tool shadowing or credential lures with mcp-trap.
- Payment-agent teams using the payment_agent adapter to see whether the agent pays a swapped payee, splits payments under an approval limit or pays twice on retry.
- Anyone about to install a third-party Claude Code skill who wants scan-skill to observe decoy-secret reads, suspicious commands and outbound requests in a sandbox.
- DevSecOps teams wiring argus-probe-action into CI so every commit triggers a local scan and fails the job on critical findings or errored probes.
- Security researchers who need findings classified against OWASP LLM Top 10, MITRE ATLAS and NIST AI RMF for audit records.
Key setup facts
Pulled from the install and usage notes below. The project README has the current values.
- Install / run
pipx install argus-probepip install argus-probepip install
- Environment variables
ANTHROPIC_API_KEYOPENAI_API_KEYAGENT_API_KEYARGUS_API_URLARGUS_API_TOKEN- Config files
argus-target.json- Runtime
- Python 3.11+
- Install notes cover
- Docker
How do you install or deploy this agent?
Requires Python 3.11+. Prefer pipx to avoid clashing with the unrelated argus package on PyPI:
pipx install argus-probe
# or
pip install argus-probeNo model key is needed for deterministic probes; set an Anthropic or OpenAI key to run judged probes:
export ANTHROPIC_API_KEY=<your key>
# or
export OPENAI_API_KEY=<your key>scan-skill additionally needs Docker and ANTHROPIC_API_KEY, and browser probes need Playwright with Chromium. To install from source, clone the repo and run pip install ..
How do you use this agent?
One-minute trial with no API key, using a bundled vulnerable MCP demo agent:
argus-probe quickstartThis writes argus-quickstart.html. For your own agent, write a target JSON such as:
{
"kind": "openai_compat",
"endpoint_url": "https://api.your-agent.example.com/v1/chat/completions",
"model": "your-agent-prod-v3",
"api_key": "${AGENT_API_KEY}"
}Run selected probes in-process and emit SARIF:
argus-probe run --local --target argus-target.json \
--probes owasp_01_prompt_injection_basic,syscard_crescendo \
--format sarif --out argus.sarifList probes that fit the target kind:
argus-probe list-probes --target argus-target.jsonPoint your agent at a hostile MCP server:
argus-probe mcp-trapScan a skill before installing it:
argus-probe scan-skill ./path/to/skillWire it into GitHub Actions with the official Action and a fail-on threshold:
- uses: actions/checkout@v4
- uses: gy15901580825/argus-probe-action@v2
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
with:
target: argus-target.json
fail-on: criticalWithout --local the CLI is a client of a server implementing /api/v1/redteam and needs ARGUS_API_URL and ARGUS_API_TOKEN; the full self-hosted stack lives at the v0.3-full tag.
What are this agent's strengths and limitations?
- 266 bundled probes with OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF and OWASP Agentic Top 10 mappings, so findings trace back to specific framework entries.
- MCP, payment and browser probes prefer deterministic verdicts from recorded calls and requests rather than a model reading the reply, reducing judging noise.
- Local --local mode needs no server, and quickstart runs end-to-end without an API key, so first contact is cheap.
- Native SARIF 2.1.0 and JUnit XML output plus a GitHub Action with Code Scanning upload fit naturally into devsecops pipelines.
- Three focused subcommands, mcp-trap, scan-mcp and scan-skill, cover MCP agents, MCP servers and Claude Code skills respectively.
- Judged probes depend on cloud models and incur spend; daily and per-run caps exist but you still need Anthropic, OpenAI or a litellm-compatible key.
- scan-skill runs only inside a throwaway Linux container, so macOS-only skills do nothing there, and a clean report is not proof of safety.
- The full self-hosted server stack (REST API, web UI, database migrations, Helm charts) has been removed from the current repo and is only available at the v0.3-full tag.
- mcp_agent probes default to a local testbed and need extra configuration for real agents, and scanning a stdio MCP server requires --on-host.
- An unrelated argus package on PyPI installs the same top-level name, so pipx or a dedicated virtualenv is advised.
How does this agent compare with similar options?
The README names NVIDIA's garak directly: Argus bundles 99 wrappers restating garak probe prompts, and garak is not a dependency. It also recommends running NVIDIA SkillSpector or Cisco skill-scanner alongside scan-skill for static review of skill text.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Argus Agent Red Team This agent | 74 · Some gaps | CLIFree + model costs | ★ 216 | today | Python | OpenAI API · Claude API |
| Agent Audit | 60 · Some gaps | CLIFree | ★ 246 | 2mo ago | Python | — |
| Superagent SDK | 46 · Major gaps | Library / SDKFreemium | ★ 6.9k | 2mo ago | TypeScript | Claude Code · OpenAI API · Claude API |
| SkillSpector Security Scanner | 75 · Good | CLIFree + model costs | ★ 27k | today | Python | Codex · Claude Code · OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence: SECURITY.md states credentials are read only from the operator's environment and used only against the operator's own target, and treats credential leakage as a vulnerability; README discloses that a cloud judge sends attack prompts and target replies to a third party, with cost caps and predictive abort; pyproject pins litellm to a narrow reviewed range with a documented 2026-10-21 malicious-release incident, plus CVE floors for jinja2/filelock — solid dependency security. Deductions: scan-skill installs and executes untrusted skills in a throwaway container and scan-mcp calls tools by default (stdio needs --on-host); these external effects are documented but lack a unified confirmation/rollback path. No rollback or undo guidance exists beyond checking out the v0.3-full tag, so rollback scores 1. Attribution names garak and the Semia paper but no NOTICE content or per-item provenance list is shown, so 2.
Evidence: CLAUDE.md documents an import-linter boundary contract; CI covers tests, install, each testbed, image builds and probe schema validation; failure semantics are concrete in README (401/timeout becomes error not pass, empty reply is error, unreachable registry is error). Deductions: README and quickstart are in tension about server mode (the repo no longer ships a server; use the v0.3-full tag), and client_agent must match the server release or every browser probe errors, showing brittle version coupling; degradation paths for optional deps (mcp, playwright, Docker) are documented but not evidenced in code, so no criterion earns full marks.
Evidence: the README Scope section enumerates 15 target adapters and per-category counts of 266 probes, and has an explicit 'What Argus does not do' list (no model training, no patching, not a production runtime gate), so capability boundaries are well handled; audience and scenarios (security teams, CI, payment/MCP/browser agents) are broad. Deductions: trigger precision relies on probe names matching target kind, the README admits judge misjudgment (small models miss jailbreaks, Azure content filters block judge calls) and deep-mode traffic is not fully predictable; environment fit requires the target to reach 127.0.0.1 or a self-deployed hosted trap for browser probes, with extra config for containers/remote hosts.
Evidence: install notes are thorough (pipx/pip, Python 3.11+, warning about the unrelated PyPI 'argus' package, optional extras); examples and FAQ are rich (quickstart, target JSON, GitHub Action YAML, scan-skill samples); known limitations are unusually specific (a clean scan is not proof of safety, macOS-only skills do nothing in the Linux container, judge-accuracy doc); LICENSE is full Apache-2.0 with license-files declared. Deductions: information architecture is strained — the README is very long and duplicates docs/, and CLAUDE.md and README disagree on server availability; naming is confusable (argus-probe dist vs top-level argus package vs separate argus-probe-action repo); versioning has only a link to GitHub releases with no in-repo CHANGELOG; maintenance responsibility scores 1 because the publisher is unverified and only a contact email is given.
Evidence: output usability is strong — SARIF 2.1.0 uploads to GitHub Code Scanning, JUnit gates CI, HTML for humans, with per-standard coverage and evidence sections; the CI workflow shows the fail-on gate tripping with exit code 3. Deductions: marginal value is partly overlapping — 99 of 266 probes are garak wrappers and many verdicts come from an LLM judge, so it overlaps existing garak/LLM-eval tooling, and the README itself says it does not replace an eval harness; cost-benefit is bounded but not cheap (--probes all estimated ~$5.32 for a chat target, scan-skill caps of $1–4.75 per task) and depends on a third-party judge API.
Evidence: claim traceability is decent — the probe mapping table is generated rather than hand-maintained, SECURITY.md records the exact litellm review method and what it did not verify (publisher identity, no PEP 740 attestations), and RESULTS.md and judge-accuracy.md provide measured data. Deductions: cross-source corroboration requires the reader to sum the Scope category counts to reach 266, and the 12 quickstart probes only indirectly match the 12 SARIF results asserted in CI; fact/inference separation is muddied because the README mixes shipped, in-engine-but-unexposed, and no-longer-shipped states, leaving the reader to sort fact from plan.
- scan-skill installs and executes untrusted skills in a container, and scan-mcp calls target tools by default (stdio needs --on-host); obtain written authorization before running against production or third-party systems.
- A cloud judge sends attack prompts and target replies to a third-party LLM provider; do not scan systems whose replies contain data you may not send out.
- The repo no longer ships the server stack; server mode requires deploying the v0.3-full tag yourself, and client_agent must match the server release or every browser probe errors.
- The publisher is not verified by the FollowAgents registry; maintenance responsibility and update path rest on a single contact email, so assess supply chain before production adoption.
- litellm is pinned to 1.108.3–1.108.4, and SECURITY.md itself says publisher identity and PEP 740 attestations were not verified, so dependency safety depends on continued manual review.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v1.19.0 argus-probe 1.19.0
Release notes excerpt
argus-probe 1.19.0 and argus-drivers-google 1.19.0. 320 probes, up from 313. Added: MCP gateway and proxy testing (#344-#361) A new mcp_gateway target tests the gateway or proxy that sits between MCP clients and upstream MCP servers. Argus hosts passive upstream fixtures: - a hostile upstream MCP server; - an honest authorization server; - a foreign origin. …View on GitHub ↗ -
v1.18.0 argus-probe 1.18.0
Release notes excerpt
argus-probe 1.18.0 and argus-drivers-google 1.18.0. 313 probes (was 295). Eighteen new probes cover attack scenarios from 2025-2026 security research. Each one is scored from what a trap, testbed or tool ledger records, with no LLM judge. Each one has a control that an honest agent passes, and is skipped when the agent never reached the attack. Added: browser probes without instruction text …View on GitHub ↗ -
v1.17.1 argus-probe 1.17.1
Release notes excerpt
argus-probe 1.17.1 and argus-drivers-google 1.17.1. 295 probes (unchanged). Fixed: Gmail seed insertion (#330) - With seed_mode: insert, the Google driver posted the seed message to a path the Gmail API does not have, so every run failed with HTTP 404 and the finding became error ("scenario dependency failed"). It now posts to users/me/messages, the documented insert endpoint. The request body and internalDateSource=receivedTime are unchanged. …View on GitHub ↗
FAQ
Do I need a model API key to run anything?
What does a run cost?
Does Argus patch or protect my agent?
Where does my data go?
Can it run in GitHub Actions?
Is Argus Agent Red Team open source?
Is Argus Agent Red Team safe to use?
Is Argus Agent Red Team still maintained?
Who makes Argus Agent Red Team?
What is Argus Agent Red Team's official website?
Thanks — we use this to improve the page.