Giskard

What does this agent do, and when should you use it?

Giskard is an open-source Python library for testing and evaluating agentic systems, with a modular, lightweight, async-first v3 architecture. Its primary user-facing packages are `giskard-checks` for evaluations and tests, and `giskard-scan` for vulnerability scanning plus RAG or quality evaluation; both are marked Beta. It accepts synchronous or asynchronous `(inputs) -> outputs` callables as Targets and runs single-turn or multi-turn evaluations through Scenarios, Checks, and Suites. `giskard-checks` produces scenario run reports, while `giskard-scan` generates adversarial suites from an agent description and performs vulnerability or knowledge-base quality evaluation. It is a Python library embedded in application code rather than a hosted chat product; LLM judges and scan generators require a provider SDK and matching API key.

A Scenario calls the system under test with .interact(inputs=..., outputs=target), applies checks to outputs or traces with .check(...), then runs through await scenario.run() and can print results with result.print_report(). Built-in evaluation types include string matching, comparisons, regex, semantic similarity, and LLM judges such as Groundedness, Conformity, and LLMJudge. vulnerability_scan(target=..., description=..., languages=...) accepts an async target and plain-language agent description, automatically generating adversarial testing across prompt injection, harmful content, stereotypes, misinformation, and more. Teams can extend generation with custom ScenarioGenerator instances passed to generate_suite or registered on vulnerability_suite_generator_registry. For RAG, v3 exposes a quality_scan and KnowledgeBase path; legacy v2 generate_testset generates questions, reference answers, and context from a knowledge base.

1. A Python team iterating on a customer-support agent can place multi-turn Scenario evaluations in its test workflow to catch behavioral regressions after changes.
2. A team operating a RAG question-answering system can use Groundedness to assess whether answers are based on supplied retrieved context.
3. An agent developer preparing for release can run vulnerability_scan to generate adversarial tests for prompt injection and harmful-content risks from a product description.
4. A security team with internal attack patterns can implement custom ScenarioGenerator instances and pass them to generate_suite for additional scan coverage.
5. A user currently relying on v2 RAGET or LLM Scan can assess a move to v3 quality_scan and vulnerability_scan.

What are this agent's strengths and limitations?

How do you install or deploy this agent?

Python 3.12+ is required. Install the base package with pip install giskard, or install checks alone with pip install giskard-checks. For scanning, run pip install "giskard[scan]" or pip install giskard-scan. LLM judges and scan generators need a provider extra, for example pip install "giskard[openai]", plus the matching API key; the README also lists extras such as anthropic. To disable optional aggregated telemetry before importing Giskard, run export DO_NOT_TRACK=1 or export GISKARD_TELEMETRY_DISABLED=1.

How do you use this agent?

For a minimal evaluation, define def get_answer(inputs: str) -> str, build Scenario("test_france_capital").interact(inputs="What is the capital of France?", outputs=get_answer).check(Groundedness(name="answer is grounded", context="...")), then call result = await scenario.run() and result.print_report() inside async def main(). For a minimal scan, define async def my_agent(inputs: str) -> str and call await vulnerability_scan(target=my_agent, description="A customer support chatbot for an e-commerce platform.", languages=["en"]). Groundedness defaults to openai/gpt-4o-mini, so that example needs the relevant provider extra and API key.

How does this agent compare with similar options?

FAQ