Agentic Security
Open-source vulnerability scanner for LLMs and agent workflows, detecting jailbreaks, fuzzing, and multimodal attacks.
Evidence shows: The tool requires users to provide API keys (e.g., Bearer tokens) and access external LLM APIs, but does not clearly state least privilege or user confirmation mechanisms. Data flow transparency is partial: README explains HTTP request format and dataset loading, but not how data is processed or stored. Sensitive data handling: no mention of how API keys or user data are handled. Dependency security: pyproject.toml lists dependencies but no vulnerability scanning or version pinning. External effects: tool sends requests to external APIs but no user confirmation or limits. Rollback: not mentioned. Source attribution: author and license are clear, but publisher is unverified. Deductions: lack of detail on least privilege, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, and source attribution.
Evidence shows: README and config examples are consistent, but there are inconsistencies such as version 1.0.0 in pyproject.toml vs 0.0.x in SECURITY.md. Dependency availability: dependency list is complete but no lock file. Failure messages: log output examples show errors and warnings, but no detailed error handling documentation. Deductions: version inconsistency and missing dependency lock.
Evidence shows: README provides multiple use cases (CLI, CI, API integration) and explains capability boundaries (e.g., text, image, audio). Trigger precision: provides config examples and CLI options. Environment fit: supports Python 3.12+ and provides Docker workflow. Deductions: not all environment limitations are detailed.
Evidence shows: Information architecture is clear with README, docs links, and examples. Install notes are simple. Naming stability: version 1.0.0 in pyproject.toml but SECURITY.md supports 0.0.x, inconsistent. Examples and FAQ: multiple examples but no FAQ. Known limitations: not explicitly listed. License: Apache-2.0, complete. Versioning/changelog: not provided. Maintenance responsibility: author is clear but no maintenance plan. Deductions: version inconsistency, no FAQ, no known limitations, no changelog.
Evidence shows: Output is tabular and easy to read. Marginal value: provides various attack simulation features but no comparison with other tools. Cost-benefit: no cost estimates or performance benchmarks. Deductions: lack of cost-benefit analysis.
Evidence shows: Claims in README (e.g., feature list) are not backed by specific evidence or test results. Cross-source: no external validation. Fact-inference separation: not clearly distinguished. Deductions: claims lack traceability.
- The tool requires users to provide API keys, but does not explain how to securely store or handle them, posing a leak risk.
- Version numbers are inconsistent (1.0.0 in pyproject.toml vs 0.0.x in SECURITY.md), which may cause confusion.
- Dependencies are not pinned, potentially introducing incompatibilities or vulnerabilities.
- No known limitations or FAQ are provided, so users may misunderstand the tool's boundaries.
What does this agent do, and when should you use it?
Agentic Security is an open-source vulnerability scanner designed for LLMs and agent workflows, helping developers, researchers, and security teams proactively identify and mitigate risks in AI systems. It supports multimodal attack probing (text, image, audio), multi-step jailbreak simulation, comprehensive fuzzing, API integration and stress testing, and reinforcement learning-based adaptive attacks. The project is licensed under Apache 2.0, installable via pip, and can be launched via CLI with a web UI. Scan results are presented as reports and can be integrated into CI/CD pipelines (e.g., GitHub Actions) for automated security scanning. The tool is compatible with any LLM API, as users can configure the target model via a custom HTTP specification.
Agentic Security reads a custom LLM API specification (HTTP request template) and executes a set of predefined attack vectors (prompts) to test the target LLM's robustness. It loads prompt sets from Hugging Face datasets or local CSV files, runs scans through the fuzzer module (e.g., perform_single_shot_scan), calculates failure rates, and generates reports with pass/fail status. Users can run scans in CI/CD via CLI commands like agentic_security ci. It provides a /v1/self-probe endpoint for testing and supports image and audio modality probing. The tool also allows custom dataset loading and dynamic dataset mutations (e.g., stenography, noise injection).
- Security engineers integrate agentic_security ci into CI/CD pipelines to ensure LLM security before each deployment.
- Researchers use multi-step jailbreak simulation to test the defense mechanisms of the latest LLMs and evaluate their resilience to complex attacks.
- Developers use fuzzing features to discover unexpected behaviors or edge cases in LLM applications by sending random inputs.
- Red team members leverage RL-based attacks to generate adaptive probes that simulate real-world adaptive threats.
- QA teams use stress testing to verify LLM APIs' stability and security responses under high load.
What are this agent's strengths and limitations?
- Supports multimodal attacks (text, image, audio), covering a broader attack surface.
- Includes multi-step jailbreak and RL-based attacks, enabling simulation of complex attack scenarios.
- Easy integration into CI/CD workflows with CLI and configuration support.
- Customizable datasets and dynamic mutations provide high extensibility.
- Requires user-provided LLM API credentials (e.g., API keys), posing a security risk.
- Integration of third-party tools (e.g., Garak, InspectAI) is listed as planned and may not be fully implemented yet.
- Scan performance depends on network latency and API rate limits; large-scale scans may be time-consuming.
- Relies on external Hugging Face datasets; if they are removed or changed, stability may be affected.
How do you install or deploy this agent?
Install via pip: pip install agentic_security. Requires Python 3.9+.
How do you use this agent?
After installation, run agentic_security to start the web UI (default port 8718). Alternatively, specify port and host with agentic_security --port=PORT --host=HOST. Initialize configuration with agentic_security init to generate agesec.toml. Edit this file to set the llmSpec field (HTTP request template with Authorization and Body) and specify dataset modules. Then run agentic_security ci to perform security scans; results are output with pass/fail status. For API integration, refer to the /v1/self-probe endpoint example.