Giskard
Evaluate, red-team, and generate tests for LLM-powered and multi-turn agent systems.
What does this agent do, and when should you use it?
Giskard is an open-source Python library for testing and evaluating agentic systems, with a modular, lightweight, async-first v3 architecture. Its primary user-facing packages are `giskard-checks` for evaluations and tests, and `giskard-scan` for vulnerability scanning plus RAG or quality evaluation; both are marked Beta. It accepts synchronous or asynchronous `(inputs) -> outputs` callables as Targets and runs single-turn or multi-turn evaluations through Scenarios, Checks, and Suites. `giskard-checks` produces scenario run reports, while `giskard-scan` generates adversarial suites from an agent description and performs vulnerability or knowledge-base quality evaluation. It is a Python library embedded in application code rather than a hosted chat product; LLM judges and scan generators require a provider SDK and matching API key.
A Scenario calls the system under test with .interact(inputs=..., outputs=target), applies checks to outputs or traces with .check(...), then runs through await scenario.run() and can print results with result.print_report(). Built-in evaluation types include string matching, comparisons, regex, semantic similarity, and LLM judges such as Groundedness, Conformity, and LLMJudge. vulnerability_scan(target=..., description=..., languages=...) accepts an async target and plain-language agent description, automatically generating adversarial testing across prompt injection, harmful content, stereotypes, misinformation, and more. Teams can extend generation with custom ScenarioGenerator instances passed to generate_suite or registered on vulnerability_suite_generator_registry. For RAG, v3 exposes a quality_scan and KnowledgeBase path; legacy v2 generate_testset generates questions, reference answers, and context from a knowledge base.
- A Python team iterating on a customer-support agent can place multi-turn `Scenario` evaluations in its test workflow to catch behavioral regressions after changes.
- A team operating a RAG question-answering system can use `Groundedness` to assess whether answers are based on supplied retrieved context.
- An agent developer preparing for release can run `vulnerability_scan` to generate adversarial tests for prompt injection and harmful-content risks from a product description.
- A security team with internal attack patterns can implement custom `ScenarioGenerator` instances and pass them to `generate_suite` for additional scan coverage.
- A user currently relying on v2 RAGET or LLM Scan can assess a move to v3 `quality_scan` and `vulnerability_scan`.
What are this agent's strengths and limitations?
- V3 separates evaluations, scanning, shared core services, LLM routing, and workflow orchestration into focused packages, avoiding scan dependencies for basic checks.
- The Scenario API supports sync or async Targets, traces, and multi-turn interactions for testing non-deterministic agent behavior.
- `giskard-scan` can generate adversarial tests from a plain-language agent description across prompt injection, jailbreak, harmful-content, and related areas.
- It combines conventional assertions with LLM-based evaluation through `Groundedness`, `Conformity`, and `LLMJudge`.
- Both `giskard-checks` and `giskard-scan` are marked Beta in the README, so adopters should expect the implementation to continue evolving.
- LLM judges and scan generators require provider SDKs, matching API keys, and network access; this is not a fully offline evaluation path.
- The runtime requires Python 3.12+, which may require an upgrade for older Python environments.
- V2 is no longer actively maintained; its automated tabular/ML scan remains v2-only and is not planned for v3.
How do you install or deploy this agent?
Python 3.12+ is required. Install the base package with pip install giskard, or install checks alone with pip install giskard-checks. For scanning, run pip install "giskard[scan]" or pip install giskard-scan. LLM judges and scan generators need a provider extra, for example pip install "giskard[openai]", plus the matching API key; the README also lists extras such as anthropic. To disable optional aggregated telemetry before importing Giskard, run export DO_NOT_TRACK=1 or export GISKARD_TELEMETRY_DISABLED=1.
How do you use this agent?
For a minimal evaluation, define def get_answer(inputs: str) -> str, build Scenario("test_france_capital").interact(inputs="What is the capital of France?", outputs=get_answer).check(Groundedness(name="answer is grounded", context="...")), then call result = await scenario.run() and result.print_report() inside async def main(). For a minimal scan, define async def my_agent(inputs: str) -> str and call await vulnerability_scan(target=my_agent, description="A customer support chatbot for an e-commerce platform.", languages=["en"]). Groundedness defaults to openai/gpt-4o-mini, so that example needs the relevant provider extra and API key.
How does this agent compare with similar options?
Compared with the no-longer-actively-maintained Giskard v2, v3 replaces LLM Scan with giskard-scan's vulnerability_scan and moves the RAGET path to quality_scan with KnowledgeBase. However, v2's automated tabular/ML scan based on giskard.Model and giskard.Dataset for performance, bias, and robustness detection, plus the giskard.testing ML suite and Giskard Hub, are not planned for v3.