Giskard
Evaluate, red-team, and generate tests for LLM-powered and multi-turn agent systems.
- Source repo
- Giskard-AI/giskard-oss
- Stars
- ★ 5.9k
- Last updated
- 4d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 47/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Low · running in minutes
- You'll need
- Typical use
- A Python team iterating on a customer-support agent can place multi-turn
Scenarioevaluations in its test workflow to catch behavioral regressions after changes. - Not a fit if
- Environments stuck on Python 3.11 or older that cannot upgrade
- Teams needing stable, non-Beta APIs for production use
- Source review
- 47/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
Giskard is an open-source Python library for testing and evaluating agentic systems, with a modular, lightweight, async-first v3 architecture. Its primary user-facing packages are giskard-checks for evaluations and tests, and giskard-scan for vulnerability scanning plus RAG or quality evaluation; both are marked Beta. It accepts synchronous or asynchronous (inputs) -> outputs callables as Targets and runs single-turn or multi-turn evaluations through Scenarios, Checks, and Suites. giskard-checks produces scenario run reports, while giskard-scan generates adversarial suites from an agent description and performs vulnerability or knowledge-base quality evaluation. It is a Python library embedded in application code rather than a hosted chat product; LLM judges and scan generators require a provider SDK and matching API key.
A Scenario calls the system under test with .interact(inputs=..., outputs=target), applies checks to outputs or traces with .check(...), then runs through await scenario.run() and can print results with result.print_report(). Built-in evaluation types include string matching, comparisons, regex, semantic similarity, and LLM judges such as Groundedness, Conformity, and LLMJudge. vulnerability_scan(target=..., description=..., languages=...) accepts an async target and plain-language agent description, automatically generating adversarial testing across prompt injection, harmful content, stereotypes, misinformation, and more. Teams can extend generation with custom ScenarioGenerator instances passed to generate_suite or registered on vulnerability_suite_generator_registry. For RAG, v3 exposes a quality_scan and KnowledgeBase path; legacy v2 generate_testset generates questions, reference answers, and context from a knowledge base.
- A Python team iterating on a customer-support agent can place multi-turn
Scenarioevaluations in its test workflow to catch behavioral regressions after changes. - A team operating a RAG question-answering system can use
Groundednessto assess whether answers are based on supplied retrieved context. - An agent developer preparing for release can run
vulnerability_scanto generate adversarial tests for prompt injection and harmful-content risks from a product description. - A security team with internal attack patterns can implement custom
ScenarioGeneratorinstances and pass them togenerate_suitefor additional scan coverage. - A user currently relying on v2 RAGET or LLM Scan can assess a move to v3
quality_scanandvulnerability_scan.
How do you install or deploy this agent?
Python 3.12+ is required. Install the base package with pip install giskard, or install checks alone with pip install giskard-checks. For scanning, run pip install "giskard[scan]" or pip install giskard-scan. LLM judges and scan generators need a provider extra, for example pip install "giskard[openai]", plus the matching API key; the README also lists extras such as anthropic. To disable optional aggregated telemetry before importing Giskard, run export DO_NOT_TRACK=1 or export GISKARD_TELEMETRY_DISABLED=1.
How do you use this agent?
For a minimal evaluation, define def get_answer(inputs: str) -> str, build Scenario("test_france_capital").interact(inputs="What is the capital of France?", outputs=get_answer).check(Groundedness(name="answer is grounded", context="...")), then call result = await scenario.run() and result.print_report() inside async def main(). For a minimal scan, define async def my_agent(inputs: str) -> str and call await vulnerability_scan(target=my_agent, description="A customer support chatbot for an e-commerce platform.", languages=["en"]). Groundedness defaults to openai/gpt-4o-mini, so that example needs the relevant provider extra and API key.
What are this agent's strengths and limitations?
- V3 separates evaluations, scanning, shared core services, LLM routing, and workflow orchestration into focused packages, avoiding scan dependencies for basic checks.
- The Scenario API supports sync or async Targets, traces, and multi-turn interactions for testing non-deterministic agent behavior.
giskard-scancan generate adversarial tests from a plain-language agent description across prompt injection, jailbreak, harmful-content, and related areas.- It combines conventional assertions with LLM-based evaluation through
Groundedness,Conformity, andLLMJudge.
- Both
giskard-checksandgiskard-scanare marked Beta in the README, so adopters should expect the implementation to continue evolving. - LLM judges and scan generators require provider SDKs, matching API keys, and network access; this is not a fully offline evaluation path.
- The runtime requires Python 3.12+, which may require an upgrade for older Python environments.
- V2 is no longer actively maintained; its automated tabular/ML scan remains v2-only and is not planned for v3.
How does this agent compare with similar options?
Compared with the no-longer-actively-maintained Giskard v2, v3 replaces LLM Scan with giskard-scan's vulnerability_scan and moves the RAGET path to quality_scan with KnowledgeBase. However, v2's automated tabular/ML scan based on giskard.Model and giskard.Dataset for performance, bias, and robustness detection, plus the giskard.testing ML suite and Giskard Hub, are not planned for v3.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Giskard This agent | 47 · Major gaps | Library / SDKFree + model costs | ★ 5.9k | 4d ago | Python | OpenAI API · Claude API |
| DeepEval | 53 · Major gaps | CLIFree + model costs | ★ 19k | 1d ago | Python | OpenAI API · Claude API |
| AI Eval Skills | 52 · Major gaps | Agent plugin / skill | ★ 1.7k | 1mo ago | — | — |
| Agent Learning Kit | 38 · Major gaps | CLIFree + model costs | ★ 128 | 2d ago | Python | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence: telemetry can be disabled (DO_NOT_TRACK/GISKARD_TELEMETRY_DISABLED) and claims no prompts or outputs are sent; CI workflows use least privilege (permissions: {} or contents: read) and pin action versions; integration tests have authorization gating for external PRs (authorize job) and 'safe for build' label. Deductions: no user confirmation mechanism found; data flow transparency partial (telemetry explained but not detailed); sensitive data handling not explicit (e.g., API key handling); dependency security only mentioned via pip-audit in dev group, not shown in CI; external effects (e.g., network calls) not clearly stated; rollback mechanism not mentioned; source attribution only via author field and copyright notice, not verified.
Evidence: version numbers consistent across README and pyproject.toml (3.0.0b1), dependency declarations clear (giskard-checks>=1.0.2b6,<2), CI tests cover multiple Python versions and packages. Deductions: failure messages not explicitly documented (e.g., error handling guide); dependency availability not fully verified (e.g., external provider SDKs not all tested in CI).
Evidence: README clearly identifies target audience (developers, researchers), provides multiple use cases (regression testing, RAG validation, safety rules, multi-turn agents); capability boundaries clear through module separation (checks, scan, core, llm, agents); trigger precision clear via Scenario and Check concepts; environment fit via Python 3.12+ and optional extras for multiple providers. Deductions: no detailed configuration guides for different environments (e.g., Docker, cloud deployment).
Evidence: README structure clear, includes installation, quickstart, concept explanations; install notes detailed (pip install and extras); naming stable (giskard-checks, giskard-scan, etc.); examples and FAQ via quickstart and documentation links; known limitations mentioned in README (v2 no longer maintained, v3 beta); license Apache-2.0; version number in pyproject.toml, but no CHANGELOG file found; maintenance responsibility via CONTRIBUTING.md and community links. Deductions: naming stability not fully verified (v3 rewrite may change API); version changelog not provided.
Evidence: output usability via result.print_report() and examples in docs; marginal value via automated scanning and test generation; cost-benefit not explicitly discussed (e.g., resource consumption, API costs). Deductions: cost-benefit not quantified.
Evidence: claims in README (e.g., features, architecture) supported by documentation links; CI tests provide partial verification; but no independent third-party verification or cross-source corroboration found. Deductions: claim traceability partial (docs links but no specific test results); cross-source corroboration insufficient; fact-inference separation not clear.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- This repository is in beta, API may change, use with caution in production.
- Telemetry is disableable but may be enabled by default; set environment variables before import.
- Depends on external LLM providers; manage API keys and costs yourself.
- No explicit rollback mechanism found; backup before upgrading.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
giskard-agents/v1.0.6 giskard-agents v1.0.6
Release notes excerpt
What's Changed * fix(deps): bump certifi, filelock and setuptools to clear pip-audit advisories by @kevinmessiaen in https://github.com/Giskard-AI/giskard-oss/pull/2898 * [Security] Exclude litellm compromised version by @pierlj in https://github.com/Giskard-AI/giskard-oss/pull/2896 Full Changelog: https://github.com/Giskard-AI/giskard-oss/compare/giskard-agents/v1.0.5...giskard-agents/v1.0.6View on GitHub ↗ -
giskard-agents/v1.0.5 giskard-agents v1.0.5
Release notes excerpt
Full Changelog: https://github.com/Giskard-AI/giskard-oss/compare/giskard-llm/v1.0.3...giskard-agents/v1.0.5View on GitHub ↗ -
giskard-llm/v1.0.3 giskard-llm v1.0.3
Release notes excerpt
Full Changelog: https://github.com/Giskard-AI/giskard-oss/compare/giskard-core/v1.0.3...giskard-llm/v1.0.3View on GitHub ↗
FAQ
Must Giskard call an external model?
Groundedness, Conformity, LLMJudge, and scan generators need a provider extra and API key.What kinds of targets can it test?
(inputs) -> outputs callable, optionally with a trace. The README explicitly lists LLMs, black-box agents, and multi-step pipelines.Does it send prompts or model outputs in telemetry?
giskard-core aggregated telemetry as optional and states that no prompts or outputs are sent. It can be disabled before import with DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1.What are the boundaries of a v2 migration?
giskard.testing ML tests, and Giskard Hub remain v2-only and are not planned for v3.