TruLens Agent Evaluation & Tracing
Trace agent behavior and evaluate quality, latency, and cost with explainable LLM judges.
What does this agent do, and when should you use it?
TruLens is a Python toolkit for evaluating and tracking LLM experiments and AI agents, rather than an autonomous task-executing agent. It uses OpenTelemetry to capture function calls, LLM generations, retrievals, and tool invocations as structured spans that can be exported to OTLP-compatible backends. Evaluations can run inline with an application or in batch over an existing dataset through the Run API. The repository documents seven agent-focused evaluators: LogicalConsistency, ExecutionEfficiency, PlanAdherence, PlanQuality, ToolSelection, ToolCalling, and ToolQuality. Provider packages cover OpenAI, LiteLLM, Google Gemini, AWS Bedrock, Snowflake Cortex, HuggingFace, and LangChain models, with separate integration packages for LangChain/LangGraph and LlamaIndex.
A developer decorates application functions with trulens.core.otel.instrument.instrument and assigns types such as SpanAttributes.SpanType.RETRIEVAL or SpanAttributes.SpanType.MCP. TruLens records trace data including inputs, outputs, latency, tokens, cost, retrieved context, and MCP tool names, arguments, outputs, and latency. Metrics can target recorded fields with Metric and Selector methods such as Selector.select_record_input() and Selector.select_context(). For inline evaluation, the app call runs inside with tru_recorder as recording; for offline work, RunConfig is passed to tru_app.add_run(), followed by run.start() and run.compute_metrics(). The resulting traces and evaluation data support version-level comparison of scores, latency, and cost, and traces can be exported to Jaeger, Grafana Tempo, Datadog, or another OTLP-compatible backend.
- A RAG engineering team that needs to inspect retrieved context and groundedness after each answer, then identify the trace step behind a poor response.
- An engineer maintaining a LangChain, LangGraph, or LlamaIndex workflow who wants tracing and evaluation without replacing the application framework.
- An agent team whose workflows create plans and invoke tools, and needs checks for plan adherence, tool choice, argument handling, and wasted execution.
- An ML team with a pre-collected evaluation dataset that needs a RunConfig-based batch run and metric recomputation.
- A platform team already operating an OTLP observability backend and needing LLM, retrieval, and MCP tool spans alongside existing telemetry.
What are this agent's strengths and limitations?
- OpenTelemetry-native spans cover functions, generations, retrieval, and tool calls, with export paths to Jaeger, Grafana Tempo, Datadog, and other OTLP backends.
- It supports both inline evaluation and Run API batch evaluation, covering live application runs and historical datasets.
- Seven named evaluators address distinct agent behaviors, including planning, tool selection, tool calls, tool reliability, and execution efficiency.
- Documented packages provide multiple model-provider paths plus LangChain/LangGraph and LlamaIndex integrations.
- The supplied material omits Python-version requirements, credential injection, environment variables, and persistence-backend setup.
- A complete runnable example still depends on initialization code for tru_app, tru_recorder, provider, and metric objects that is not shown.
- Each model-provider path requires its matching trulens-providers-* package, so setup and behavior can vary by provider.
- Fine-grained traces require application code to be annotated with @instrument and explicit span-attribute mappings.
How do you install or deploy this agent?
Install the base package with: pip install trulens. For OpenAI or Azure OpenAI feedback evaluation, run: pip install trulens trulens-providers-openai. For LiteLLM-supported providers including Anthropic, run: pip install trulens trulens-providers-litellm. Additional documented provider packages are trulens-providers-google, trulens-providers-bedrock, trulens-providers-cortex, trulens-providers-huggingface, and trulens-providers-langchain; framework packages are trulens-apps-langchain and trulens-apps-llamaindex. The supplied material does not specify a Python version, credential setup, or environment-variable names.
How do you use this agent?
Decorate the function to observe with @instrument and configure its span_type and SpanAttributes mapping; a retrieval function can use SpanAttributes.SpanType.RETRIEVAL with its query mapped to "query" and returned contexts mapped to "return". Define a Metric with selectors that locate the recorded input and context. For inline evaluation, invoke the application inside with tru_recorder as recording; for batch evaluation, create RunConfig, call tru_app.add_run(run_config=...), run.start(), and then run.compute_metrics([relevance, groundedness]). The supplied material does not include standalone initialization for tru_app, tru_recorder, provider, relevance, or groundedness, so it does not provide a fully runnable first invocation.
How does this agent compare with similar options?
In the cited AIMultiple RAG-evaluation comparison, TruLens is reported as leading WandB Weave, RAGAS, DeepEval, and UpTrain on three of four ranking metrics for context relevance NDCG@5. That is a benchmark-specific result, not evidence of universal superiority across workloads.