AgentEvals Trace Evaluation
Evaluate AI agent behavior from existing OpenTelemetry traces without rerunning the agent.
- Source repo
- agentevals-dev/agentevals
- Stars
- ★ 166
- Last updated
- 5d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 68/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformCodex · Claude Code (Partial support)
- Cost
- Free, no paid service needed
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- An agent team using OpenTelemetry wants deterministic tool-trajectory checks in CI, with the command exit code gating deployment.
- Not a fit if
- Teams needing end-to-end scoring of long coding sessions
- Teams requiring a stable release without breaking changes
- Source review
- 68/100 · Some gaps
What does this agent do, and when should you use it?
AgentEvals accepts OpenTelemetry data already produced by an agent and turns GenAI spans and log events into turns, tool calls, tokens, and answers. Evaluations can run through its CLI, web UI, REST API, gRPC API, or MCP server, while telemetry arrives over OTLP HTTP or gRPC. Built-in metrics cover tool trajectories, response matching, and LLM-based scoring; teams can add custom evaluators and rubrics. Results can be exported as gen_ai.evaluation.result log events parented to the evaluated span. It runs locally, in Docker, or on Kubernetes via Helm; Postgres is optional for persisted run history.
AgentEvals receives live telemetry over OTLP HTTP or gRPC and can read OTLP JSON, Jaeger JSON, or Collector file exporter output. It extracts turns, tool calls, tokens, models, and messages from invoke_agent, execute_tool, and model-call spans, as well as GenAI message attributes, span events, and log events, then compares them with a golden eval set. Users run agentevals run with built-in metrics, LLM judges, or custom evaluators; inspect span trees and live sessions in the web UI; or evaluate sessions through MCP tools and APIs. Results appear as CLI text or JSON and can be sent back to a telemetry pipeline as gen_ai.evaluation.result events with --emit-otel.
- An agent team using OpenTelemetry wants deterministic tool-trajectory checks in CI, with the command exit code gating deployment.
- Developers using Google ADK, Strands, LangChain, the OpenAI Agents SDK, or Pydantic AI want to score existing OTLP data without rerunning agent calls.
- Engineers need to inspect live agent sessions, tool inputs and outputs, and span trees through the included web UI.
- A team operating an OpenTelemetry Collector pipeline wants evaluation results returned as
gen_ai.evaluation.resultevents for backend queries or alerts. - A team with domain-specific scoring rules wants custom evaluators that exchange JSON scores through stdin and stdout, in any language.
- Operators deploying agent evaluation on Kubernetes want to use the Helm chart and optionally configure Postgres for run history.
How do you install or deploy this agent?
Install the CLI; add the live extra if you need the MCP server. AgentEvals itself does not require a paid service or API key. For a source checkout, the repository documents uv sync, followed by running commands with uv run agentevals.
How do you use this agent?
Run a tool-trajectory metric against a recorded trace; JSON output and sending results back to OTLP are optional:
agentevals run samples/helm.json \
--eval-set samples/eval_set_helm.json \
-m tool_trajectory_avg_scoreStart the server to receive live agent telemetry, then point the agent's OTLP exporter at the default HTTP endpoint. service.instance.id distinguishes reruns, and agentevals.session_name names the session:
agentevals serve --devexport OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_RESOURCE_ATTRIBUTES="agentevals.session_name=my-agent,service.instance.id=$(uuidgen)"
python your_agent.pyThe optional Python SDK can set an eval set in an AgentEvals session:
from agentevals import AgentEvals
app = AgentEvals()
with app.session(eval_set_id="my-eval"):
agent.invoke("Roll a 20-sided die for me")What are this agent's strengths and limitations?
- Scores recorded telemetry without rerunning the agent, so the same run can be rescored without additional agent tokens.
- Accepts standard OTLP over HTTP and gRPC and documents integrations with multiple frameworks and Collector pipelines.
- Offers deterministic matching, LLM judges, rubrics, and custom evaluators written in any language.
- Emits standard
gen_ai.evaluation.resultevents linked to evaluated spans for existing telemetry backends.
- The project is under active development and warns that breaking changes may occur.
- Evaluation depends on telemetry containing recognizable GenAI spans or messages; coding sessions without suitable wrapper spans may not yield turns.
- End-to-end scoring of long coding sessions is outside its stated scope.
- Persisted run history requires Postgres; that storage layer is a preview and its schema may change between minor versions.
How does this agent compare with similar options?
The README compares AgentEvals with LangSmith, Langfuse, Opik, and Bedrock AgentCore evaluation, describing those alternatives as rerunning agents or requiring their own SDK and storage before scoring; AgentEvals scores existing OpenTelemetry data. It also contrasts ADK eval, which runs and scores an agent in one step, with AgentEvals, which scores telemetry only and uses ADK's metrics and eval-set format.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| AgentEvals Trace Evaluation This agent | 68 · Some gaps | CLIFree | ★ 166 | 5d ago | Python | — |
| AI Eval Skills | 52 · Major gaps | Agent plugin / skill | ★ 1.5k | 24d ago | — | — |
| Voice Lab | 27 · Major gaps | CLIFree + model costs | ★ 175 | 1y ago | Python | OpenAI API |
| LangWatch | 78 · Good | CLIFreemium | ★ 5k | today | TypeScript | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
The evidence shows CI permissions limited to contents: read, an SDK that does not read other backends' OTEL_EXPORTER_OTLP_* credentials, and descriptions of telemetry fields and OTLP output; dependency configuration also lists several security version floors. Deductions: the product handles message content, but the material does not describe sensitive-field filtering, access controls, or retention policy; evaluation, telemetry emission, and persistence do not require confirmation, remote evaluators and MCP can have external effects, and rollback guidance is absent. ADK metrics and format are acknowledged, but publisher identity and maintenance ownership are unverified.
The README's descriptions of offline file evaluation, online service, optional Postgres, and optional SDK are broadly consistent; CI checks the lockfile and covers multiple Python versions, and an example shows a trajectory mismatch failure explanation. Deductions: the material does not establish ongoing dependency availability or clear errors for every failure path; CI configuration is not execution verification in this review.
The material lists frameworks and CLI, UI, MCP, Docker, Kubernetes, and CI use cases, and explicitly says Claude Code and similar scenarios are only partly supported and that end-to-end scoring of long coding sessions is outside scope. Deductions: although entry points are described, permission boundaries, deployment boundaries, and security configuration across environments are not fully specified.
The README has substantial navigation, installation commands, quick starts, framework examples, FAQs, and limitations; the repository includes an Apache-2.0 license. Deductions: the project says it is under active development and expects breaking changes, but the material gives no concrete migration policy or changelog; a contributing guide and CI exist, but no maintainer is identified.
Results can be emitted as JSON or OpenTelemetry evaluation events, examples show readable pass/fail explanations, and evaluating existing traces avoids rerunning the agent, demonstrating clear added value. Deduction: LLM judges, service deployment, and optional persistence add costs, and the material does not quantify their tradeoff against benefits.
The README gives concrete descriptions of inputs, outputs, supported formats, limits, and installation; pyproject, CI, and some tests corroborate dependency floors, version ranges, and persistence behavior. Deductions: many capability statements remain README claims without corresponding implementation evidence in the supplied material; this review is static and did not execute or independently verify the product.
- Incoming telemetry may contain message content; check collection scope, access controls, and retention settings before deployment.
- The project says it is under active development and may introduce breaking changes; check compatibility before upgrading.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
What's Changed * feat: v1alpha1 protobuf contract + native gRPC by @krisztianfekete in https://github.com/agentevals-dev/agentevals/pull/251 * Give rubric_based_* metrics a public rubric path (#132) by @erauner12 in https://github.com/agentevals-dev/agentevals/pull/229 * fix: resolve the package version from v* tags only by @krisztianfekete in https://github.com/agentevals-dev/agentevals/pull/254 * chore(ui): bump dev dependencies by @krisztianfekete in https://github.com/agentevals-dev/agentevals/pull/255 …View on GitHub ↗ -
Release notes excerpt
agentevals v0.10.0 agentevals now reads, stores and evaluates agent telemetry as plain OpenTelemetry. Send OTLP from any GenAI instrumentation, evaluate turns against a golden run, and get the scores back into your observability pipeline as OTel events. Highlights - OpenTelemetry native ingestion. OTLP over HTTP (:4318, now spec compliant with gzip, proper 400 errors and protobuf responses) and gRPC (:4317). Files can be OTLP JSON or protobuf, Jaeger, JSONL, or Collector file exporter output. See [Sending telemetry to agentevals](https://github.com/agentevals-dev/agentevals/blob/v0.10. …View on GitHub ↗ -
Release notes excerpt
What's Changed * chore(deps): bump the npm group in /ui with 10 updates by @dependabot[bot] in https://github.com/agentevals-dev/agentevals/pull/217 * ci: hold mcp at 1.x in the dependabot python group by @krisztianfekete in https://github.com/agentevals-dev/agentevals/pull/219 * chore(deps): bump astral-sh/setup-uv from 10.0.1 to 10.1.0 in the github-actions group by @dependabot[bot] in https://github.com/agentevals-dev/agentevals/pull/221 …View on GitHub ↗
FAQ
Does evaluation use additional agent model tokens?
Do I need to run a server or database?
Can it evaluate a complete Claude Code or Codex coding session?
Which agent frameworks and telemetry transports work?
How do I gate a CI deployment on evaluation?
agentevals run trace files, an eval set, metrics, and a threshold. The CLI can use its exit code as a deployment gate and supports machine-readable output with --output json.