Strands Evals SDK
A Python framework for testing, simulating, diagnosing, and improving AI agents and LLM applications.
Per-dimension scores and reasoning
The workflows show meaningful least-privilege discipline through scoped GitHub permissions, collaborator checks, approval environments, OIDC, account-ID masking, and disabled credential persistence; however, this evidence concerns CI rather than SDK runtime behavior. No user-confirmation mechanism is shown for consequential operations. The README identifies model judging, telemetry traces, HTTP images, and file output, but does not fully explain which models or services receive data, retention, or deletion. Sensitive-data evidence is limited to CI secrets/OIDC, with no privacy guidance for evaluation inputs, traces, or images. Dependencies have ranges, but there is no lockfile, vulnerability scan, hash pinning, or supply-chain policy, and some Actions/custom actions use only major tags or main. Some external effects and file writes are illustrated, and simulated tools explicitly avoid real execution, but model-network calls and real agent/tool side effects are not systematically enumerated. Experiments can be saved and versioned, yet no operational rollback or recovery procedure is documented. Repository/ecosystem links, AWS author metadata, and the Apache license provide attribution, but publisher identity is unverified and specific maintainer ownership is not established.
The README, CLI table, and project configuration generally describe one coherent evaluation framework, but telemetry.memory_exporter conflicts with telemetry.in_memory_exporter, and the final multimodal example is truncated at case.input.medi, so self-consistency is only partial. Supported Python versions, core dependencies, optional integrations, and a test matrix are clearly declared and adequate for ordinary dependency discovery; the deduction is for the absence of locked resolutions and installation troubleshooting. validate, --help, --fail-on, diagnosis flows, and explicit workflow error messages show attention to failure feedback, but the supplied sources omit the CLI and implementation code needed to establish comprehensive error coverage.
The material thoroughly addresses output, trajectory, tool, skill, multimodal, multi-turn, diagnostic, chaos, and red-team scenarios. Some boundaries are explicit: simulated tool bodies are not called, unmatched skill signals produce empty results, and red-team support is labeled experimental; model requirements, scale limits, and integration boundaries are not comprehensively summarized. Diagnosis triggers, confidence thresholds, chaos hooks, and CLI command activation are fairly precise, while generation and model-judge defaults still require unseen source code. Python requirements, virtual-environment setup, development/test modes, and several optional backends provide adequate environment fit, but cloud credentials, regional/model availability, platform differences, and resource needs are incomplete.
The README has strong navigation through overview, quick start, CLI, installation, and feature-specific examples, and its installation notes are detailed. Naming is mostly coherent, but inconsistent telemetry attributes and the truncated multimodal example justify a deduction. Examples are extensive, though there is no real FAQ and several snippets depend on omitted objects or implementations. The skill-parser empty-result limitation, experimental red-team status, and license disclaimers are disclosed, but there is no centralized limitations section. Apache-2.0 metadata matches a complete LICENSE, justifying full license credit. VCS-tag versioning, a PyPI version badge, and versioned experiments are present, but no changelog, compatibility policy, or migration guidance is supplied. AWS contact metadata, Issues, PRs, and Discord offer maintenance channels, while named maintainers, support commitments, and a security-reporting path are absent.
Structured EvaluationOutput records, displayable/JSON reports, scores, labels, reasons, trajectories, and actionable diagnosis recommendations make the outputs useful for CI and human review. Combining numerous evaluation modes, simulation, diagnosis, chaos testing, and experiment generation offers substantial marginal value over assembling these components manually. Tool simulation without live infrastructure, in-memory tracing, and CLI workflows may reduce adoption cost, but LLM charges, latency, token use, and large-scale execution costs are not quantified or bounded, preventing full cost-benefit credit.
Most feature claims are tied to named APIs, code examples, CLI commands, or project configuration, giving reasonable traceability; however, the supplied evidence lacks implementation sources and substantive tests, so claims cannot be traced through algorithms or assertions. Installation, Python support, the CLI entry point, licensing, and test tooling receive partial corroboration across the README, pyproject, and workflows, whereas advanced evaluator capabilities rely mainly on the README alone. The documentation distinguishes simulation from real execution and discloses empty skill-parser results and experimental functionality, but promotional terms such as “powerful,” “comprehensive,” and “realistic” lack measurements, so fact and inference are not fully separated.
- Before evaluating personal data, confidential prompts, production traces, or remote images, verify the transmission, retention, and deletion behavior of the selected model provider, telemetry exporter, and optional integrations.
- Do not treat LLM-as-a-judge scores, generated cases, or root-cause recommendations as ground truth; cross-check them with human review, fixed benchmarks, and independent evaluators.
- Review external side effects and IAM permissions separately before running real agents, tools, or integration tests in production or credentialed environments; the no-execution assurance in the README applies only to the ToolSimulator example.
- Produce and review locked dependency resolutions, enable vulnerability and supply-chain scanning, and pin CI Actions and third-party actions to immutable commits.
- Verify or correct the inconsistent telemetry attribute names and truncated multimodal example before using those snippets for integration.
What does this agent do, and when should you use it?
Strands Evals SDK is a Python evaluation framework for agent outputs, tool trajectories, multi-agent interactions, and complete conversational sessions. Its core abstractions are Case, Experiment, and Evaluator, exposed through both a Python API and the strands-evals command-line interface. Experiments execute user-supplied agent factories or task functions and produce EvaluationReport results that can be displayed or serialized as JSON. The package includes LLM-as-a-Judge evaluators, OpenTelemetry trace mapping, user and tool simulators, experiment generation, failure diagnosis, chaos injection, and adversarial red-team evaluation. It is a library and CLI rather than a hosted evaluation service, so adopters must supply the system under test and configure any model access required by judges or simulators.
A user defines inputs, expected outputs, metadata, or expected trajectories as Case objects, then combines those cases with evaluators such as OutputEvaluator, TrajectoryEvaluator, or HelpfulnessEvaluator in an Experiment. Experiment.run_evaluations() invokes a supplied task function or agent, gathers its output and optional trajectory, and applies deterministic scoring helpers or model-based judges to produce scores, pass states, reasons, and labels in an EvaluationReport. For behavioral analysis, tools_use_extractor can recover tool calls, while StrandsEvalsTelemetry captures OpenTelemetry spans and StrandsInMemorySessionMapper converts them into Session objects. ActorSimulator drives adaptive multi-turn conversations, and ToolSimulator generates schema-validated tool responses with optional shared state; ExperimentGenerator synthesizes cases from a context description. detect_failures, analyze_root_cause, and diagnose_session inspect recorded sessions and return failure findings and repair recommendations, while ChaosPlugin injects errors or response corruption. The CLI provides run, validate, report, diagnose, and generate commands for Experiment, EvaluationReport, and Session JSON workflows.
- An agent engineering team wants a repeatable release gate for response accuracy, completeness, and instruction following using explicit scoring rubrics.
- A developer building a tool-using assistant needs to verify tool selection, parameters, and call ordering against an expected trajectory.
- A customer-service or workflow team wants ActorSimulator to exercise unscripted, goal-directed conversations and measure whether the user objective is completed.
- A team without test APIs, databases, or devices wants ToolSimulator to supply Pydantic-validated responses and coherent shared state.
- A reliability engineer needs to inject timeouts, network failures, and corrupted responses to assess failure communication, partial completion, and recovery behavior.
- A safety or quality team wants adversarial red-team evaluation, automated session failure detection, root-cause analysis, and actionable repair recommendations.
What are this agent's strengths and limitations?
- It covers outputs, tool trajectories, interactions, full sessions, and image-to-text responses within one composable experiment model.
- The Python API and five CLI subcommands support schema validation, execution, report rendering, diagnosis, and suite generation for both CI and one-off analysis.
- ActorSimulator and ToolSimulator provide adaptive user conversations plus schema-validated, stateful tool substitutes when real backends are unavailable.
- Built-in failure detection, root-cause analysis, deterministic chaos effects, and recovery-focused evaluators extend evaluation beyond aggregate scores.
- The Evaluator base class, custom rubrics, trajectory extractors, and serializable Experiment objects provide concrete extension and reproducibility mechanisms.
- It requires Python 3.10+, and the shown agent examples use Strands Agents; other harnesses may need custom task adapters, trajectory extraction, or session mapping.
- Many advanced paths rely on LLM-as-a-Judge, including simulation and test generation, which introduces model access requirements, latency, cost, and nondeterminism.
- Trace-based evaluation requires telemetry setup, span collection, and Session mapping, making it more involved than plain output comparison.
- Codex and Claude Code are documented as recognizable skill-signal sources, not as native hosts for the complete evaluation runtime.
- The supplied material does not define a unified provider credential setup, provider support matrix, or production deployment architecture.
How do you install or deploy this agent?
Python 3.10 or later is required. Install the published package with:
pip install strands-agents-evalsFor local development:
python -m venv .venv
source .venv/bin/activatepip install -e .
Use pip install -e ".[test]" for test dependencies or pip install -e ".[test,dev]" for both test and development dependencies. Installation also provides the strands-evals console script. The examples import strands.Agent, and LLM-based judging, simulation, or generation requires access to the selected model service; the supplied material does not specify a universal credential variable.
How do you use this agent?
A minimal Python flow creates a Case, an evaluator, and an Experiment, then passes run_evaluations a function that returns the agent output:
from strands import Agent
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluatorcases = [Case(name="knowledge-1", input="What is the capital of France?", expected_output="Paris")]
evaluators = [OutputEvaluator(rubric="Score 1.0 for an accurate answer and 0.0 otherwise.")]
experiment = Experiment(cases=cases, evaluators=evaluators)def task(case):
return str(Agent(callback_handler=None)(case.input))report = experiment.run_evaluations(task)
report.run_display()For a serialized experiment, validate it and invoke an agent factory:
strands-evals validate experiments/customer_service.json
strands-evals run experiments/customer_service.json --agent my_pkg.agents:build_agent --displayAn ad-hoc case can run without an experiment file:
strands-evals run --input "What is the capital of France?" --expected-output "Paris" --agent my_pkg.agents:build_agentUse strands-evals diagnose session.json --confidence medium for a recorded session. Generate a starter suite with strands-evals generate --context "tool and task description" --num-cases 10 --evaluator TrajectoryEvaluator -o experiments/generated.json.