DeepEval

What does this agent do, and when should you use it?

DeepEval is an open-source evaluation framework for large language model systems, designed to feel like Pytest but aimed at unit testing LLM applications. It ships a large library of metric classes — G-Eval, answer relevancy, hallucination, RAG metrics, agent trajectory metrics, multi-turn, voice, MCP and multimodal metrics — that score outputs from 0 to 1 using LLM-as-a-judge or local NLP models and always return an explanation. You can write a test_ai_agent.py and run it with deepeval test run, call evaluate() in a notebook, or use a metric standalone via measure(). It evaluates black-box end-to-end apps, full agent trajectories, and individual agent steps such as tool calls and retrieval, and can generate single- and multi-turn synthetic datasets. The framework plugs into any LLM backend and any CI/CD pipeline; an optional Confident AI platform hosts shareable reports and production monitoring.

A test target is an LLMTestCase holding fields like input, actual_output, expected_output and retrieval_context. The user picks a metric class (GEval, AnswerRelevancyMetric, TaskCompletionMetric, FaithfulnessMetric and many others), sets a threshold, and then calls assert_test(), evaluate([...], [...]) or metric.measure(test_case); the metric invokes the configured LLM or a local model and returns a 0-1 score plus a reason. For agents, tracing via the @observe() decorator or an integration (OpenAI, LangChain, LangGraph, CrewAI, LlamaIndex, Google ADK, Anthropic and more) captures the ordered sequence of decisions and tool calls, and dataset.evals_iterator(metrics=[...]) runs trajectory-level metrics over it. deepeval login uploads test runs to Confident AI for shareable reports, and the framework can also benchmark any LLM on MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval and GSM8K in under ten lines of code.

1. Application teams adding regression tests for a support chatbot before each release, catching hallucinated answers
2. RAG engineers using Contextual Recall and Contextual Precision to isolate retrieval failures from generation failures
3. Agent developers scoring multi-step tool-calling trajectories with TaskCompletionMetric and ToolCorrectness
4. Multi-turn product teams validating long conversations with Knowledge Retention and Role Adherence
5. Platform teams wiring deepeval test run into CI/CD as a gate so prompt edits cannot silently regress quality

How do you install or deploy this agent?

DeepEval requires Python>=3.9. Install with pip:

pip install -U deepeval

Model-based metrics need a model API key, for example:

export OPENAI_API_KEY="..."

At import time DeepEval auto-loads .env.local then .env from the working directory, with precedence process env > .env.local > .env; set DEEPEVAL_DISABLE_DOTENV=1 to opt out.

How do you use this agent?

Create a test file and write your first end-to-end test case:

import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

def test_case():
    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
        threshold=0.5
    )
    test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output="You have 30 days to get a full refund at no extra cost.",
        expected_output="We offer a 30-day full refund at no extra costs.",
    )
    assert_test(test_case, [correctness_metric])

Run it from the command line:

deepeval test run test_ai_agent.py

Or evaluate without Pytest, which suits notebooks:

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.7)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund at no extra costs.",
    retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
)
evaluate([test_case], [answer_relevancy_metric])

For cloud reports, run deepeval login and paste the API key.

What are this agent's strengths and limitations?

How does this agent compare with similar options?

Key facts side by side with the most closely related agents.

How does FollowAgents rate this agent?

FAQ