Judgeval

What does this agent do, and when should you use it?

Judgeval is an open-source Python SDK from Judgment Labs for agent improvement. It provides tracing and agent-judge evaluation for LLM-powered applications, helping you detect failures, understand root causes, and validate fixes against real production cases. Tracing is built on OpenTelemetry, using `@Tracer.observe()` to automatically capture function inputs, outputs, and token usage. Agent judges are prompt-based scorers that produce structured, scored labels for agent behaviors, accumulating into a searchable history. Online monitoring scores live traffic server-side with no latency impact and triggers Slack alerts on regressions. The SDK supports major model providers (OpenAI, Anthropic, Google GenAI, Together AI) and frameworks (LangGraph, OpenLit, Claude Agent SDK). It includes a CLI and MCP server for managing traces, judges, behaviors, and evals.

Judgeval provides a Python package installable via pip install judgeval, requiring environment variables JUDGMENT_API_KEY and JUDGMENT_ORG_ID. Core components: Tracer class initializes a project (Tracer.init(project_name=...)), @Tracer.observe() decorator captures function execution, and wrap() auto-instruments OpenAI clients. Users define prompt-based scorers ('agent judges') to evaluate agent behaviors, producing structured scored labels. JQL (Judgeval Query Language) allows querying traces via the Judgeval client, e.g., client.query(traces().where(eq("session", "session-123")).ids()). A CLI and MCP server enable terminal and AI-tool integration. The end-to-end flow: instrument code -> run agents -> traces captured -> define judges -> run evaluations online or on historical traces -> collect behavior signals -> trigger alerts or validate fixes.

1. AI engineers building agents can detect production failures by using Judgeval's tracing to capture detailed logs and pinpoint root causes.
2. Platform teams validating a new version can replay historical traces to confirm fixes work before deploying, avoiding regressions.
3. Ops teams monitoring live traffic can set up Slack alerts to be notified when agent behavior deviates from expected norms.
4. Data scientists analyzing agent behavior patterns can use JQL to query historical traces, filtering by session or other criteria.
5. Developers using LangGraph or LangChain can enable auto-instrumentation without modifying business logic, gaining observability quickly.
6. Organizations with existing OpenTelemetry-based observability stacks can integrate Judgeval due to its compatibility.

What are this agent's strengths and limitations?

How do you install or deploy this agent?

Install via pip:

pip install judgeval

Set environment variables:

export JUDGMENT_API_KEY=...
export JUDGMENT_ORG_ID=...

How do you use this agent?

Initialize the tracer and wrap OpenAI client:

from judgeval import Tracer, wrap
from openai import OpenAI

Tracer.init(project_name="my-project")
client = wrap(OpenAI())

Decorate functions to trace:

@Tracer.observe(span_type="tool")
def search(query: str) -> str:
    # ...
    return results

@Tracer.observe(span_type="agent")
def run_agent(question: str) -> str:
    # ...
    response = client.chat.completions.create(...)
    return response.choices[0].message.content

Run your agent; traces are automatically captured. Query with JQL:

from judgeval import Judgeval
from judgeval.jql import eq, traces

client = Judgeval(project_name="my-project")
result = client.query(traces().where(eq("session", "session-123")).ids())

Refer to the official docs for further use.

FAQ