Strands Evals SDK
Evaluate, diagnose, and improve AI agents and LLM applications.
- Source repo
- strands-agents/evals
- Stars
- ★ 251
- Last updated
- today
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 64/100 · Some gaps
At a glance
- How it runs
- Works with
- Portable with changesCodex · Claude Code (Partial support)
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- An agent engineering team can validate an experiment JSON file in CI, run its regression suite, and use the documented
--fail-onexit-code rules as a release gate. - Not a fit if
- Teams on non-Strands runtimes, which need adapter code
- Teams wanting zero-config setup without provider credentials or tracing
- Source review
- 64/100 · Some gaps 1 safety controls not found
What does this agent do, and when should you use it?
Strands Evals SDK is a Python framework for assessing AI agents and LLM applications across final outputs, tool trajectories, multi-agent interactions, skills, multimodal responses, and complete sessions. Its main abstractions include Case, Experiment, EvaluationReport, and a collection of output-, trace-, tool-, session-, and multimodal evaluators, with an extension point for custom Evaluator implementations. Evaluation workloads run through a user-supplied Python callable or agent factory and may consume direct return values, raw message trajectories, or Session objects mapped from OpenTelemetry traces. The package also includes conversation and tool simulators, automated experiment generation, failure detection, root-cause analysis, deterministic chaos injection, and JSON experiment serialization. Teams can integrate it through the Python API or the strands-evals CLI for running, validating, reporting, diagnosing, and generating experiments; it is an evaluation SDK and command-line tool, not a standalone hosted agent.
A user defines inputs, expected outputs, expected trajectories, and metadata as Case objects, then passes them with evaluators such as OutputEvaluator, TrajectoryEvaluator, HelpfulnessEvaluator, or GoalSuccessRateEvaluator to Experiment. Experiment.run_evaluations() invokes the supplied task function and produces an EvaluationReport that can be displayed or serialized. For behavioral analysis, evaluators can inspect raw message lists or spans captured by StrandsEvalsTelemetry and converted into a Session by StrandsInMemorySessionMapper. ActorSimulator drives adaptive, goal-oriented multi-turn conversations, while ToolSimulator uses an LLM to return Pydantic-schema-validated tool results and can maintain shared state across related tools. ExperimentGenerator synthesizes test cases from context; detect_failures, analyze_root_cause, and diagnose_session identify session failures and recommend fixes. ChaosPlugin injects timeouts, network errors, execution or validation failures, and response corruption through Strands plugin hooks, while experiment files can be saved as JSON and validated, executed, diagnosed, or rendered through the CLI.
- An agent engineering team can validate an experiment JSON file in CI, run its regression suite, and use the documented
--fail-onexit-code rules as a release gate. - Engineers building tool-using assistants can combine TrajectoryEvaluator, ToolSelectionAccuracyEvaluator, and ToolParameterAccuracyEvaluator to inspect tool choice, order, and arguments.
- A customer-service or task-oriented dialogue team can use ActorSimulator to create adaptive multi-turn users and score the sessions with HelpfulnessEvaluator and GoalSuccessRateEvaluator.
- Developers without test APIs, databases, or devices can replace real execution with ToolSimulator responses that follow output schemas and share simulated state.
- Reliability engineers can inject tool timeouts, network failures, and corrupted responses with ChaosPlugin, then assess communication, partial completion, and recovery behavior.
- Safety and quality teams can apply multimodal judges, skill-selection checks, session diagnosis, or the included adversarial red-team strategies to targeted risks.
How do you install or deploy this agent?
Python 3.10 or newer is required. Install the published package with:
pip install strands-agents-evalsFor source development, create a virtual environment and install the base project or its optional test and development dependencies:
python -m venv .venv
source .venv/bin/activate
pip install -e .pip install -e ".[test]"pip install -e ".[test,dev]"On Windows, activate the environment with .venv\Scripts\activate. The examples import strands.Agent, but the supplied material does not document provider credentials, environment variables, or a default model setup. Adopters must configure those separately for their chosen model and Strands runtime.
How do you use this agent?
A minimal Python workflow creates a Case, configures an OutputEvaluator, runs an Experiment, and displays the report:
from strands import Agent
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluator
test_cases = [
Case[str, str](
name="knowledge-1",
input="What is the capital of France?",
expected_output="The capital of France is Paris.",
metadata={"category": "knowledge"}
)
]
evaluators = [
OutputEvaluator(
rubric="""
Evaluate based on:
1. Accuracy - Is the information correct?
2. Completeness - Does it fully answer the question?
3. Clarity - Is it easy to understand?
Score 1.0 if all criteria are met excellently.
Score 0.5 if some criteria are partially met.
Score 0.0 if the response is inadequate.
"""
)
]
experiment = Experiment[str, str](cases=test_cases, evaluators=evaluators)
def get_response(case: Case) -> str:
agent = Agent(callback_handler=None)
return str(agent(case.input))
report = experiment.run_evaluations(get_response)
report.run_display()The CLI can validate and run a serialized experiment. The --agent value identifies an agent factory as MODULE:CALLABLE:
strands-evals validate experiments/customer_service.json
strands-evals run experiments/customer_service.json \
--agent my_pkg.agents:build_agent \
--displayIt also supports an ad-hoc case without an experiment file:
strands-evals run \
--input "What is the capital of France?" \
--expected-output "Paris" \
--agent my_pkg.agents:build_agentDiagnose a serialized Session or generate an experiment from contextual material with:
strands-evals diagnose session.json --confidence mediumstrands-evals generate --context "$(cat tools.txt)" --num-cases 10 \
--evaluator TrajectoryEvaluator -o experiments/generated.jsonWhat are this agent's strengths and limitations?
- It covers unusually broad evaluation layers in one framework: outputs, trajectories, individual tool calls, sessions, multi-agent interactions, skills, and image-to-text responses.
- The diagnosis pipeline goes beyond scoring by combining failure detection, root-cause analysis, and actionable fix recommendations.
- ActorSimulator and ToolSimulator support adaptive conversations, schema-validated mock tool responses, and shared simulated state without requiring live business infrastructure.
- ChaosPlugin provides deterministic fault injection alongside dedicated evaluators for failure communication, partial completion, and recovery strategy.
- A Python API, five CLI subcommands, JSON serialization, and custom Evaluator classes provide practical integration points for development and CI.
- It requires Python 3.10+, and the complete agent, tracing, and plugin examples are built around Strands components; other runtimes need adapter code.
- Many advanced evaluators and simulators use LLM-as-a-Judge or model-generated responses, adding provider access, latency, cost, and nondeterminism; no cost-control mechanism is documented.
- The supplied material does not identify the required provider credentials, environment variables, or default model configuration, leaving initial provider setup to adopters.
- Trace-level evaluation requires OpenTelemetry span capture and Session mapping, which is more involved than direct input/output comparison.
- Skill evaluation recognizes a defined set of harness signals; an unrecognized harness returns empty results rather than an error, so teams must verify
parse_available_skills()before trusting scores.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Strands Evals SDK This agent | 64 · Some gaps | Library / SDKFree + model costs | ★ 251 | today | Python | — |
| Tracely | 66 · Some gaps | — | ★ 1k | 12d ago | Python | Claude Code · OpenAI API · Claude API |
| LangWatch | 78 · Good | CLIFreemium | ★ 5k | today | TypeScript | OpenAI API · Claude API |
| TanStack AI | 76 · Some gaps | Library / SDKFree + model costs | ★ 3.3k | 1d ago | TypeScript | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
The workflows show meaningful least-privilege discipline through scoped GitHub permissions, collaborator checks, approval environments, OIDC, account-ID masking, and disabled credential persistence; however, this evidence concerns CI rather than SDK runtime behavior. No user-confirmation mechanism is shown for consequential operations. The README identifies model judging, telemetry traces, HTTP images, and file output, but does not fully explain which models or services receive data, retention, or deletion. Sensitive-data evidence is limited to CI secrets/OIDC, with no privacy guidance for evaluation inputs, traces, or images. Dependencies have ranges, but there is no lockfile, vulnerability scan, hash pinning, or supply-chain policy, and some Actions/custom actions use only major tags or main. Some external effects and file writes are illustrated, and simulated tools explicitly avoid real execution, but model-network calls and real agent/tool side effects are not systematically enumerated. Experiments can be saved and versioned, yet no operational rollback or recovery procedure is documented. Repository/ecosystem links, AWS author metadata, and the Apache license provide attribution, but publisher identity is unverified and specific maintainer ownership is not established.
The README, CLI table, and project configuration generally describe one coherent evaluation framework, but telemetry.memory_exporter conflicts with telemetry.in_memory_exporter, and the final multimodal example is truncated at case.input.medi, so self-consistency is only partial. Supported Python versions, core dependencies, optional integrations, and a test matrix are clearly declared and adequate for ordinary dependency discovery; the deduction is for the absence of locked resolutions and installation troubleshooting. validate, --help, --fail-on, diagnosis flows, and explicit workflow error messages show attention to failure feedback, but the supplied sources omit the CLI and implementation code needed to establish comprehensive error coverage.
The material thoroughly addresses output, trajectory, tool, skill, multimodal, multi-turn, diagnostic, chaos, and red-team scenarios. Some boundaries are explicit: simulated tool bodies are not called, unmatched skill signals produce empty results, and red-team support is labeled experimental; model requirements, scale limits, and integration boundaries are not comprehensively summarized. Diagnosis triggers, confidence thresholds, chaos hooks, and CLI command activation are fairly precise, while generation and model-judge defaults still require unseen source code. Python requirements, virtual-environment setup, development/test modes, and several optional backends provide adequate environment fit, but cloud credentials, regional/model availability, platform differences, and resource needs are incomplete.
The README has strong navigation through overview, quick start, CLI, installation, and feature-specific examples, and its installation notes are detailed. Naming is mostly coherent, but inconsistent telemetry attributes and the truncated multimodal example justify a deduction. Examples are extensive, though there is no real FAQ and several snippets depend on omitted objects or implementations. The skill-parser empty-result limitation, experimental red-team status, and license disclaimers are disclosed, but there is no centralized limitations section. Apache-2.0 metadata matches a complete LICENSE, justifying full license credit. VCS-tag versioning, a PyPI version badge, and versioned experiments are present, but no changelog, compatibility policy, or migration guidance is supplied. AWS contact metadata, Issues, PRs, and Discord offer maintenance channels, while named maintainers, support commitments, and a security-reporting path are absent.
Structured EvaluationOutput records, displayable/JSON reports, scores, labels, reasons, trajectories, and actionable diagnosis recommendations make the outputs useful for CI and human review. Combining numerous evaluation modes, simulation, diagnosis, chaos testing, and experiment generation offers substantial marginal value over assembling these components manually. Tool simulation without live infrastructure, in-memory tracing, and CLI workflows may reduce adoption cost, but LLM charges, latency, token use, and large-scale execution costs are not quantified or bounded, preventing full cost-benefit credit.
Most feature claims are tied to named APIs, code examples, CLI commands, or project configuration, giving reasonable traceability; however, the supplied evidence lacks implementation sources and substantive tests, so claims cannot be traced through algorithms or assertions. Installation, Python support, the CLI entry point, licensing, and test tooling receive partial corroboration across the README, pyproject, and workflows, whereas advanced evaluator capabilities rely mainly on the README alone. The documentation distinguishes simulation from real execution and discloses empty skill-parser results and experimental functionality, but promotional terms such as “powerful,” “comprehensive,” and “realistic” lack measurements, so fact and inference are not fully separated.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Before evaluating personal data, confidential prompts, production traces, or remote images, verify the transmission, retention, and deletion behavior of the selected model provider, telemetry exporter, and optional integrations.
- Do not treat LLM-as-a-judge scores, generated cases, or root-cause recommendations as ground truth; cross-check them with human review, fixed benchmarks, and independent evaluators.
- Review external side effects and IAM permissions separately before running real agents, tools, or integration tests in production or credentialed environments; the no-execution assurance in the README applies only to the ToolSimulator example.
- Produce and review locked dependency resolutions, enable vulnerability and supply-chain scanning, and pin CI Actions and third-party actions to immutable commits.
- Verify or correct the inconsistent telemetry attribute names and truncated multimodal example before using those snippets for integration.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
v1.9.0 _Auto-drafted from commits in v1.8.0..v1.9.0, grouped by conventional-commit type. Edit on the release page after publish if you want a polished writeup; the canonical release notes live on the website._ 🚀 Features - feat(skills): suggest evaluators and write experiment file in session-converter (#496) (8fcc8e0) - feat(skills): add extraction steps to session-converter skill (#491) (b44ce38) - feat(cli): add --case selector to run command (#492) (32e72bb) 🔧 Chores - chore(ci): relax CI gate on workstream/optimizer-workflow branch (#499) (69d85c5)View on GitHub ↗ -
Release notes excerpt
v1.8.0 _Auto-drafted from commits in v1.7.0..v1.8.0, grouped by conventional-commit type. Edit on the release page after publish if you want a polished writeup; the canonical release notes live on the website._ 🚀 Features - feat(redteam): add CrescendoStrategy(stop_on_tool_call=) (#472) (8275e0f) ⚡ Performance - perf(experiment): run a case's evaluators concurrently (#467) (f6b894f) 📚 Documentation - docs: update README nav and examples (#470) (6511807) 📦 Build - build: drop unused strands-agents-tools dependency (#471) (2ce9b78) 👷 CI …View on GitHub ↗ -
Release notes excerpt
v1.7.0 _Auto-drafted from commits in v1.6.0..v1.7.0, grouped by conventional-commit type. Edit on the release page after publish if you want a polished writeup; the canonical release notes live on the website._ 🚀 Features - feat(redteam)!: graduate red teaming out of experimental (#465) (064a5e5) 🐛 Fixes - fix: judge EventLoopException by its original cause (#464) (b59a389) - fix: replace retired Sonnet 4 model ID in integ tests (#441) (7e7a23e)View on GitHub ↗
FAQ
Must the system under test use Strands Agent?
--agent factory or --task callable, and Python experiments accept a user-supplied task function, so basic output evaluation is not limited to one Agent class. However, the documented agent, tracing, and chaos examples use Strands components, so another framework needs adapters for its inputs, outputs, or Session representation.Does every evaluation require an LLM call?
How can tests avoid calling production services?
share_state_id. It generates simulated responses with an LLM and never executes the decorated function body.Can it act as a CI quality gate?
strands-evals validate schema-checks experiment files before execution, and strands-evals run executes them. The source also documents --fail-on exit-code rules; the exact available flags are exposed through each subcommand's --help.