TruLens Agent Evaluation & Tracing
Trace agent behavior and evaluate quality, latency, and cost with explainable LLM judges.
- Source repo
- truera/trulens
- Stars
- ★ 3.6k
- Last updated
- 7d ago
- License
- MIT
- Primary language
- Python
- FA score
- 51/100 · Major gaps
At a glance
- Works with
- Universal · cross-platformOpenAI APIClaude API (Partial support)
- You'll need
- Typical use
- A RAG engineering team that needs to inspect retrieved context and groundedness after each answer, then identify the trace step behind a poor response.
- Main limitation
- The supplied material omits Python-version requirements, credential injection, environment variables, and persistence-backend setup.
- Source review
- 51/100 · Major gaps 1 safety controls not found
What does this agent do, and when should you use it?
TruLens is a Python toolkit for evaluating and tracking LLM experiments and AI agents, rather than an autonomous task-executing agent. It uses OpenTelemetry to capture function calls, LLM generations, retrievals, and tool invocations as structured spans that can be exported to OTLP-compatible backends. Evaluations can run inline with an application or in batch over an existing dataset through the Run API. The repository documents seven agent-focused evaluators: LogicalConsistency, ExecutionEfficiency, PlanAdherence, PlanQuality, ToolSelection, ToolCalling, and ToolQuality. Provider packages cover OpenAI, LiteLLM, Google Gemini, AWS Bedrock, Snowflake Cortex, HuggingFace, and LangChain models, with separate integration packages for LangChain/LangGraph and LlamaIndex.
A developer decorates application functions with trulens.core.otel.instrument.instrument and assigns types such as SpanAttributes.SpanType.RETRIEVAL or SpanAttributes.SpanType.MCP. TruLens records trace data including inputs, outputs, latency, tokens, cost, retrieved context, and MCP tool names, arguments, outputs, and latency. Metrics can target recorded fields with Metric and Selector methods such as Selector.select_record_input() and Selector.select_context(). For inline evaluation, the app call runs inside with tru_recorder as recording; for offline work, RunConfig is passed to tru_app.add_run(), followed by run.start() and run.compute_metrics(). The resulting traces and evaluation data support version-level comparison of scores, latency, and cost, and traces can be exported to Jaeger, Grafana Tempo, Datadog, or another OTLP-compatible backend.
- A RAG engineering team that needs to inspect retrieved context and groundedness after each answer, then identify the trace step behind a poor response.
- An engineer maintaining a LangChain, LangGraph, or LlamaIndex workflow who wants tracing and evaluation without replacing the application framework.
- An agent team whose workflows create plans and invoke tools, and needs checks for plan adherence, tool choice, argument handling, and wasted execution.
- An ML team with a pre-collected evaluation dataset that needs a RunConfig-based batch run and metric recomputation.
- A platform team already operating an OTLP observability backend and needing LLM, retrieval, and MCP tool spans alongside existing telemetry.
How do you install or deploy this agent?
Install the base package with: pip install trulens. For OpenAI or Azure OpenAI feedback evaluation, run: pip install trulens trulens-providers-openai. For LiteLLM-supported providers including Anthropic, run: pip install trulens trulens-providers-litellm. Additional documented provider packages are trulens-providers-google, trulens-providers-bedrock, trulens-providers-cortex, trulens-providers-huggingface, and trulens-providers-langchain; framework packages are trulens-apps-langchain and trulens-apps-llamaindex. The supplied material does not specify a Python version, credential setup, or environment-variable names.
How do you use this agent?
Decorate the function to observe with @instrument and configure its span_type and SpanAttributes mapping; a retrieval function can use SpanAttributes.SpanType.RETRIEVAL with its query mapped to "query" and returned contexts mapped to "return". Define a Metric with selectors that locate the recorded input and context. For inline evaluation, invoke the application inside with tru_recorder as recording; for batch evaluation, create RunConfig, call tru_app.add_run(run_config=...), run.start(), and then run.compute_metrics([relevance, groundedness]). The supplied material does not include standalone initialization for tru_app, tru_recorder, provider, relevance, or groundedness, so it does not provide a fully runnable first invocation.
What are this agent's strengths and limitations?
- OpenTelemetry-native spans cover functions, generations, retrieval, and tool calls, with export paths to Jaeger, Grafana Tempo, Datadog, and other OTLP backends.
- It supports both inline evaluation and Run API batch evaluation, covering live application runs and historical datasets.
- Seven named evaluators address distinct agent behaviors, including planning, tool selection, tool calls, tool reliability, and execution efficiency.
- Documented packages provide multiple model-provider paths plus LangChain/LangGraph and LlamaIndex integrations.
- The supplied material omits Python-version requirements, credential injection, environment variables, and persistence-backend setup.
- A complete runnable example still depends on initialization code for tru_app, tru_recorder, provider, and metric objects that is not shown.
- Each model-provider path requires its matching trulens-providers-* package, so setup and behavior can vary by provider.
- Fine-grained traces require application code to be annotated with @instrument and explicit span-attribute mappings.
How does this agent compare with similar options?
In the cited AIMultiple RAG-evaluation comparison, TruLens is reported as leading WandB Weave, RAGAS, DeepEval, and UpTrain on three of four ranking metrics for context relevance NDCG@5. That is a benchmark-specific result, not evidence of universal superiority across workloads.
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| TruLens Agent Evaluation & Tracing This agent | 51 · Major gaps | ★ 3.6k | 7d ago | Python | OpenAI API |
| OpenInference | 71 · Some gaps | ★ 1.2k | 2d ago | Python | OpenAI API · Claude API |
| Langtrace | 42 · Major gaps | ★ 1.2k | 10mo ago | TypeScript | OpenAI API · Claude API |
| Future AGI | 51 · Major gaps | ★ 2.1k | 1d ago | Python | Claude Code · OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: OpenTelemetry-based tracing provides good data flow transparency (2); but least privilege design not evident (1), user confirmation missing (0), sensitive data handling insufficient (1), dependency security unclear (1), external effects limited (1), rollback not specified (1), source attribution has copyright but publisher unverified (2). Deductions: lack of user confirmation and least privilege evidence.
Evidence shows: consistent structure, docs match code examples (2); dependencies declared but availability unverified (2); failure messages not detailed (1). Deductions: insufficient failure handling documentation.
Evidence shows: covers multiple scenarios (RAG, Agent), clear audience (2); capability boundaries described (2); trigger precision moderate (1); good environment fit (2). Deductions: imprecise trigger mechanism description.
Evidence shows: clear information architecture (2); detailed install notes (2); stable naming (2); rich examples (2); known limitations sparse (1); clear license (3); no changelog (1); maintenance responsibility stated (2). Deductions: missing changelog and known limitations.
Evidence shows: high output usability (2); clear marginal value (2); cost-benefit analysis lacking (1). Deductions: missing cost-benefit data.
Evidence shows: claims have citations (2); limited cross-source verification (1); fact-inference separation unclear (1). Deductions: lack of independent verification and clear separation.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Publisher identity unverified, proceed with caution.
- No user confirmation mechanism found, may execute actions automatically.
- Sensitive data handling details insufficient, review needed.
- Dependency security not explicit, check for vulnerabilities.
- No changelog, be cautious with version updates.