RAMPART
A pytest-native safety and security testing framework that brings adversarial and benign-failure coverage to agentic AI apps in your existing test workflow.
Evidence shows: pyrit pinned to a git commit (6dc8b94), SHA-pinned CI actions, permissions: {} with per-job least privilege, persist-credentials: false, scheduled CodeQL, and bounded dependency ranges — supporting dependency_security and least_privilege at the engineering-config level (2 each). Not full marks because the core runtime code (adapters, evaluators, manifest handling) is absent from the evidence; the agent product's own permission model, confirmation flows, data flow, sensitive-data handling, external effects, and rollback cannot be verified, and the README says nothing about them — those criteria score 1 (only suggestive placeholders like AppManifest and ObservabilityLevel exist). Source attribution is clear: Microsoft AI Red Team authorship, consistent LICENSE and copyright headers (2).
Evidence shows: MockSession/MockAdapter raise descriptive ValueErrors on empty input and document contracts (cycling behavior, nested sequence semantics) — failure_messages 2. Self-consistency: markers, coverage config, and CI matrix (3.11–3.14) match requires-python (2). Deductions: pyrit as a git dependency makes availability hinge on an upstream commit; no runtime code beyond fixtures is available to assess failure paths, so dependency_availability is 1.
Evidence shows: clear audience (developers, pytest users, AI red teamers), keywords, classifiers, pytest11 entry point, asyncio auto mode, OS-independent claim — supporting environment_fit (2) and audience_and_scenarios (2). Deductions: the README names no supported agent frameworks or concrete scenario coverage, so capability_boundaries is undocumented; harm/trial/slow markers exist but no examples demonstrate trigger semantics, so trigger_precision is 1.
Evidence shows: complete MIT LICENSE consistent with pyproject, copyright headers, and the ruff copyright regex (license 3). Versioning via hatch-vcs, PyPI badges, and an honest 'Alpha' classifier (versioning_changelog 2 — no CHANGELOG file, so not full marks). Stable naming (RAMPART/rampart consistent, 2). Maintenance: Microsoft attribution, Issues link, active CI and scheduled CodeQL (2). Deductions: the README contains only a positioning statement and trademark notice — no install steps, no examples, no FAQ, no known-limitations section — so information_architecture, install_notes, examples_and_faq, and known_limitations all score 1.
Evidence shows: as a pytest-native framework the integration cost for pytest users is low (entry point, asyncio auto, markers), and the marginal value — turning AI red-team findings into asserted, CI-runnable tests — is well-founded (2). Coverage fail_under=80 and ruff ALL show quality gates. Deductions: no usage examples, no description of report output or result readability, so output_usability is 1; no statement of token/API cost of running attack suites, so cost_benefit stops at 2.
Evidence shows: pyproject and CI corroborate each other (version scheme, tests/unit, lint toolchain), and README claims have partial traceable anchors in markers and fixtures (fact_inference_separation 2 — the Alpha classifier honestly separates maturity claims). Deductions: the README's core claims (adversarial attacks, benign failures, harm categories, evaluation-driven assertions) have no code or documentation backing in the supplied files, so claim_traceability is 1; OpenSSF Scorecard and PyPI badges are external references that cannot be cross-verified within the provided files, so cross_source_corroboration is 1.
- The supplied files contain no runtime code: adapters, evaluators, and manifest loading cannot be statically reviewed; scores largely reflect engineering configuration and test fixtures.
- pyrit is pinned via a git commit; build availability depends on long-term accessibility of that upstream commit.
- The README lacks install steps, usage examples, known limitations, and data-flow/privacy statements; consult the docs site before adoption.
- The project self-identifies as Alpha; APIs and marker semantics may change — avoid unreviewed reliance in production pipelines.
- No tests were executed; this is a purely static review with low confidence.
What does this agent do, and when should you use it?
RAMPART (Risk Assessment & Measurement Platform for Agentic Red Teaming) is Microsoft's open-source safety and security testing framework for agentic AI applications. It runs natively under pytest, letting developers author and execute security tests for AI agents inside their existing pytest workflow. The framework covers adversarial attacks, benign failures, and a broad range of harm categories, judged via evaluation-driven assertions. It is MIT-licensed, published on PyPI, and ships with OpenSSF Scorecard and GitHub CI badges. It suits engineering teams that want to systematically verify agent safety within the software development lifecycle.
RAMPART lets developers define safety and security tests for AI agents as pytest test cases. It reads those definitions, runs scenarios covering adversarial attacks, benign failures, and a range of harm categories, and produces pass/fail verdicts on agent behavior through evaluation-driven assertions. Execution is embedded in pytest's existing run and reporting mechanics, so security tests can be triggered alongside regular test suites.
- An AI application engineering team adding safety regression tests for agents in CI to keep new prompt-injection risks out of production
- A security red team systematically probing agent resilience with structured adversarial attack cases
- A platform team detecting benign failures of agents on normal inputs, such as wrongful refusals and hallucinations
- A compliance or risk-assessment group measuring agent behavior by harm category
- A pytest-using team folding agent safety testing into its existing suite with minimal learning overhead
What are this agent's strengths and limitations?
- pytest-native design reuses your existing pytest workflow, plugins, and CI integration, keeping learning cost low
- Covers adversarial attacks, benign failures, and a broad set of harm categories, giving a comprehensive assessment surface
- Evaluation-driven assertions make safety tests decidable and regression-friendly
- MIT license plus a PyPI release make evaluation and adoption straightforward in enterprise projects
- The README is very sparse — no usage examples, configuration docs, or integration guidance — raising evaluation cost
- Maintained by Microsoft; modified versions must comply with Microsoft's Trademark & Brand Guidelines
- Testing agents requires network access and a reachable target system, so test stability depends on the system under test
- No repository topics or additional guidance, making the ecosystem and integration scope hard to gauge quickly
How do you install or deploy this agent?
The repository publishes to PyPI (https://pypi.org/project/RAMPART/), so installation is presumably: pip install RAMPART. The README does not spell out the exact install command, required Python version, or dependency list; verify precise installation instructions on the PyPI page or repository docs before adopting.
How do you use this agent?
Because the framework is pytest-native, the basic usage is to author safety tests against your AI agent as pytest test cases using evaluation-driven assertions, then run them with pytest. The README provides no concrete example code, CLI invocation, or configuration format; consult the repository's tests and docs for working examples before adopting.