Rogue — AI Agent Evaluator & Red Team Platform
Stress-test your AI agents before attackers do, with automated evaluation and red teaming.
Evidence shows: CLI supports auth types (API_KEY, BEARER_TOKEN, BASIC_AUTH) but least privilege not explicit; user confirmation missing (no interactive confirmation step); data flow transparency partially present (README describes architecture and protocols but not detailed data flow); sensitive data handling uses SecretStr and credential validation but storage/transmission security not specified; dependency security has version pins and CI build but no vulnerability scanning mentioned; external effects include network requests and file outputs but side effects not clarified; rollback not mentioned; source attribution has copyright and license but publisher unverified. Deductions: missing user confirmation, rollback, insufficient data flow and sensitive data details.
Evidence shows: README and pyproject.toml consistent, test files exist but no test results; dependencies partially pinned but availability not assured; failure messages not documented. Deductions: dependency availability and failure messages lack evidence.
Evidence shows: README identifies target users (developers and security teams) and scenarios (automatic evaluation and red teaming); capability boundaries via scan types and protocols; trigger precision via CLI options and config; environment fit supports multiple protocols and Python versions. Deductions: limitations and boundary conditions not explicit.
Evidence shows: README well-structured with quick start and docs links; install notes via uvx; naming stable (rogue-ai); examples and FAQ links present; known limitations not explicit; license is MIT with Commons Clause but metadata NOASSERTION; versioning and changelog absent; maintenance responsibility via security policy but not explicit. Deductions: license metadata inconsistency, missing versioning and changelog.
Evidence shows: output reports support Markdown, CSV, JSON, high usability; marginal value in providing automated evaluation and red teaming; cost-benefit not explicit but many dependencies. Deductions: cost-benefit lacks evidence.
Evidence shows: README claims features but no test results or independent verification; cross-source corroboration limited; facts and inferences not clearly separated. Deductions: claims lack traceability.
- License metadata is NOASSERTION but LICENSE.md is MIT with Commons Clause; verify actual license.
- Publisher unverified, identity unknown; evaluate cautiously.
- Many dependencies with some pinned versions; check supply chain security.
- No user confirmation mechanism; may execute external actions automatically.
- No rollback mechanism; operations may be irreversible.
What does this agent do, and when should you use it?
Rogue is a platform for evaluating and red-teaming AI agents, featuring two modes: automatic evaluation and red teaming. Automatic evaluation verifies agent behavior against business policies, while red teaming simulates adversarial attacks (75+ vulnerabilities, 12 security categories, 20+ attack techniques). It operates on a client-server architecture with a TUI (Go + Bubble Tea) and CLI, supporting protocols A2A, MCP, and direct Python function calls. Users can launch via uvx and configure a judge LLM to evaluate agent interactions. Risk scoring is CVSS-based, and scans are reproducible with random seeds. The license is proprietary; free for personal and internal use, commercial hosting requires a license.
Rogue reads business context and scenario files, connects to the target agent via CLI/TUI (e.g., --evaluated-agent-url), and uses a judge LLM (e.g., openai/gpt-4o-mini) to evaluate conversations. In red team mode, it executes attacks across categories (encoding, social engineering, injection, semantic, technical), logs vulnerabilities, and computes CVSS risk scores (0-10). Supports --random-seed for reproducibility. It generates reports in Markdown, CSV, and JSON. Example CLI invocation: 'uvx rogue-ai cli --protocol python --python-entrypoint-file ./my_agent.py --judge-llm openai/gpt-4o-mini'. TUI provides real-time monitoring.
- DevOps engineer integrates Rogue CLI into CI/CD pipeline for regression testing of agent policies.
- Security auditor performs red teaming against AI systems using OWASP and MITRE frameworks.
- Developer quickly tests a custom Python agent using --protocol python without setting up a server.
- Product team validates that new agent versions comply with business policies before release.
- Compliance team generates vulnerability reports aligned with GDPR and EU AI Act.
What are this agent's strengths and limitations?
- Dual functionality: automatic evaluation and red teaming in one tool.
- Supports multiple protocols (A2A, MCP, Python), easing integration with different agent types.
- CVSS-based risk scoring and reproducible scans with random seeds.
- TUI provides real-time conversation monitoring and multiple report export formats.
- Proprietary license; commercial hosting requires payment, personal/internal use is free but limited.
- Requires external LLM API keys (OpenAI/Anthropic/Google), incurring API costs.
- No support for local models; relies on platform APIs.
- Full red team scan can be time-consuming (30-45 minutes).
How do you install or deploy this agent?
Requires Python 3.10+ and uvx (install from Astral). Run 'uvx rogue-ai' for TUI or 'uvx rogue-ai cli' for CLI; uvx handles installation. Configure LLM API keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, or GOOGLE_API_KEY) as environment variables.
How do you use this agent?
- Start the target agent (e.g., sample t-shirt store: 'uvx rogue-ai --example=tshirt_store'). 2. In TUI, set Agent URL (e.g., http://localhost:10001) and mode (Automatic Evaluation or Red Teaming). 3. For CLI: run 'uvx rogue-ai cli --evaluated-agent-url http://localhost:10001 --judge-llm openai/gpt-4o-mini --business-context-file ./.rogue/business_context.md'. 4. Review generated reports (default output in current directory).