TruLens Agent Evaluation & Tracing
Trace agent behavior and evaluate quality, latency, and cost with explainable LLM judges.
- Source repo
- truera/trulens
- Stars
- ★ 3.6k
- Last updated
- 4d ago
- License
- MIT
- Primary language
- Python
- FA score
- 61/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI APIClaude API (Partial support)
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A RAG engineering team that needs to inspect retrieved context and groundedness after each answer, then identify the trace step behind a poor response.
- Not a fit if
- Teams wanting evaluation with no code instrumentation
- Users who want one provider without extra trulens-providers-* packages
- Source review
- 61/100 · Some gaps
What does this agent do, and when should you use it?
TruLens is a Python toolkit for evaluating and tracking LLM experiments and AI agents, rather than an autonomous task-executing agent. It uses OpenTelemetry to capture function calls, LLM generations, retrievals, and tool invocations as structured spans that can be exported to OTLP-compatible backends. Evaluations can run inline with an application or in batch over an existing dataset through the Run API. The repository documents seven agent-focused evaluators: LogicalConsistency, ExecutionEfficiency, PlanAdherence, PlanQuality, ToolSelection, ToolCalling, and ToolQuality. Provider packages cover OpenAI, LiteLLM, Google Gemini, AWS Bedrock, Snowflake Cortex, HuggingFace, and LangChain models, with separate integration packages for LangChain/LangGraph and LlamaIndex.
A developer decorates application functions with trulens.core.otel.instrument.instrument and assigns types such as SpanAttributes.SpanType.RETRIEVAL or SpanAttributes.SpanType.MCP. TruLens records trace data including inputs, outputs, latency, tokens, cost, retrieved context, and MCP tool names, arguments, outputs, and latency. Metrics can target recorded fields with Metric and Selector methods such as Selector.select_record_input() and Selector.select_context(). For inline evaluation, the app call runs inside with tru_recorder as recording; for offline work, RunConfig is passed to tru_app.add_run(), followed by run.start() and run.compute_metrics(). The resulting traces and evaluation data support version-level comparison of scores, latency, and cost, and traces can be exported to Jaeger, Grafana Tempo, Datadog, or another OTLP-compatible backend.
- A RAG engineering team that needs to inspect retrieved context and groundedness after each answer, then identify the trace step behind a poor response.
- An engineer maintaining a LangChain, LangGraph, or LlamaIndex workflow who wants tracing and evaluation without replacing the application framework.
- An agent team whose workflows create plans and invoke tools, and needs checks for plan adherence, tool choice, argument handling, and wasted execution.
- An ML team with a pre-collected evaluation dataset that needs a RunConfig-based batch run and metric recomputation.
- A platform team already operating an OTLP observability backend and needing LLM, retrieval, and MCP tool spans alongside existing telemetry.
How do you install or deploy this agent?
Install the base package with: pip install trulens. For OpenAI or Azure OpenAI feedback evaluation, run: pip install trulens trulens-providers-openai. For LiteLLM-supported providers including Anthropic, run: pip install trulens trulens-providers-litellm. Additional documented provider packages are trulens-providers-google, trulens-providers-bedrock, trulens-providers-cortex, trulens-providers-huggingface, and trulens-providers-langchain; framework packages are trulens-apps-langchain and trulens-apps-llamaindex. The supplied material does not specify a Python version, credential setup, or environment-variable names.
How do you use this agent?
Decorate the function to observe with @instrument and configure its span_type and SpanAttributes mapping; a retrieval function can use SpanAttributes.SpanType.RETRIEVAL with its query mapped to "query" and returned contexts mapped to "return". Define a Metric with selectors that locate the recorded input and context. For inline evaluation, invoke the application inside with tru_recorder as recording; for batch evaluation, create RunConfig, call tru_app.add_run(run_config=...), run.start(), and then run.compute_metrics([relevance, groundedness]). The supplied material does not include standalone initialization for tru_app, tru_recorder, provider, relevance, or groundedness, so it does not provide a fully runnable first invocation.
What are this agent's strengths and limitations?
- OpenTelemetry-native spans cover functions, generations, retrieval, and tool calls, with export paths to Jaeger, Grafana Tempo, Datadog, and other OTLP backends.
- It supports both inline evaluation and Run API batch evaluation, covering live application runs and historical datasets.
- Seven named evaluators address distinct agent behaviors, including planning, tool selection, tool calls, tool reliability, and execution efficiency.
- Documented packages provide multiple model-provider paths plus LangChain/LangGraph and LlamaIndex integrations.
- The supplied material omits Python-version requirements, credential injection, environment variables, and persistence-backend setup.
- A complete runnable example still depends on initialization code for tru_app, tru_recorder, provider, and metric objects that is not shown.
- Each model-provider path requires its matching trulens-providers-* package, so setup and behavior can vary by provider.
- Fine-grained traces require application code to be annotated with @instrument and explicit span-attribute mappings.
How does this agent compare with similar options?
In the cited AIMultiple RAG-evaluation comparison, TruLens is reported as leading WandB Weave, RAGAS, DeepEval, and UpTrain on three of four ranking metrics for context relevance NDCG@5. That is a benchmark-specific result, not evidence of universal superiority across workloads.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| TruLens Agent Evaluation & Tracing This agent | 61 · Some gaps | Library / SDKFree + model costs | ★ 3.6k | 4d ago | Python | OpenAI API |
| OpenInference | 66 · Some gaps | Library / SDKFree | ★ 1.3k | 1d ago | Python | OpenAI API · Claude API |
| traceAI | 59 · Major gaps | Library / SDKFree | ★ 266 | today | Python | — |
| Langtrace | 48 · Major gaps | Library / SDKFreemium | ★ 1.2k | 11mo ago | TypeScript | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Workflow and CI permissions are scoped: the certification workflow has read-only contents access and does not execute PR code, while CLA writes go to a separate signatures branch. However, the product describes recording inputs, outputs, tokens, and cost by default, without showing sensitive-field filtering or per-run confirmation, so those criteria lose points. OpenTelemetry export paths and external LLM evaluation are disclosed, but runtime side-effect confirmation and a clear recovery guide are limited. MIT copyright and package author metadata aid attribution; publisher identity remains unknown.
The README, installation guide, and project configuration broadly agree on Python support, package structure, and core features, and the configuration lists providers and dependency constraints. The source sample is limited and does not establish comprehensive failure messaging. Installation guidance covers environments and package choices, but gives little detail on handling unavailable dependencies.
The README targets LLM application and agent developers and covers tracing, online and offline evaluation, RAG, MCP, multiple model providers, and frameworks. Selectors and custom instrumentation show configurability. Boundaries such as reliance on LLM judges and configuration of a provider are partly described, while trigger conditions and deployment constraints are not comprehensively specified.
The README is clearly organized around features, installation, examples, and providers; AGENTS.md separately gives contributor guidance, project structure, and coding patterns. Quick usage, code examples, and installation documentation are substantial. The MIT license text is complete, and Python/Poetry requirements, deprecated compatibility package, and some version limits are documented. The supplied material has no changelog or clear maintenance responsibility/update path, and a deprecated interface remains as a shim.
Tracing, step-level metrics, cost/latency comparisons, and batch evaluation can help diagnose failures and compare versions, with usable Python examples. The product primarily observes and evaluates rather than automatically fixing issues. LLM judges and external providers add setup and call costs; the materials do not quantify those costs or explain cost controls, so cost-benefit receives a low score.
Feature claims have some support across README examples, package configuration, and the deprecated compatibility shim. Benchmark figures include metrics, sample sizes, or references, making them traceable, though some comparisons depend on external material and the sample does not cover the full implementation. Evaluation steps and some boundaries are identifiable, but promotional performance claims are not consistently separated from facts directly demonstrated by the supplied source.
- Telemetry records step inputs, outputs, tokens, and cost; confirm retention, export destinations, and sensitive-data handling before using production data.
- Feedback evaluation requires a configured model provider and may make external API calls with associated costs; the supplied material does not show comprehensive cost controls or runtime confirmation.
- AGENTS.md is contributor guidance, not an end-user product security policy; the packet also omits the full implementation and a dependency audit.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/trulens)<a href="https://followagents.com/en/agents/trulens"><img src="https://followagents.com/badge/trulens.svg" alt="FARS review"></a>Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
trulens-2.16.0 TruLens 2.16.0
Release notes excerpt
TruLens 2.16 This release adds new evaluation metrics and Jury reliability metadata, improves handling of unparsable judge scores, and fixes cost accounting, batch evaluation, and OpenTelemetry tracing. Breaking changes and compatibility - Streamlit-in-Snowflake dashboard support has been removed. ([#2784](https://github.com/truera/trulens/pull/2784)) - All packages now require Python 3.10 or later. The test matrix now includes Python 3.14. ([#2896](https://github.com/truera/trulens/pull/2896)) New features …View on GitHub ↗ -
trulens-2.15.0 TruLens 2.15.0
Release notes excerpt
TruLens 2.15.0 OTel-native traces that travel end to end and stay evaluable Three changes in this release work on the same OTel path: the trace now stays whole across the HTTP call to the provider, GenAI message content survives both persistence and export, and the finished trace can reach OTLP backends that only speak HTTP/protobuf. …View on GitHub ↗ -
trulens-2.14.0 TruLens 2.14.0
Release notes excerpt
TruLens 2.14.0 Release Notes Headline Features Coding Agent Instrumentation Instrument IDE hooks and coding agent workflows — including Claude Code and Cursor — with TruLens tracing. Capture the full execution trace of agentic coding sessions for evaluation and observability. (#2732) by @joshreini1 Production Trace Curation into Evaluation Datasets …View on GitHub ↗
FAQ
Does TruLens run or orchestrate my agent for me?
Can it be used with Anthropic models?
Must I use a TruLens-only observability backend?
What can it capture for MCP tools?
How are API credentials configured?
Thanks — we use this to improve the page.