Back to all tools
DeepEval
Pytest for LLMs — unit test your AI outputs with 20+ built-in evaluation metrics.
0Open Source
LLM Evaluation
Overview
DeepEval is an open-source LLM evaluation framework that works like pytest — write unit tests for your LLM outputs with built-in metrics for correctness, hallucination, RAG quality, and safety.
Key Features
- Pytest-compatible — run with deepeval test run or pytest
- 20+ metrics: correctness, hallucination, faithfulness, bias, toxicity
- RAG-specific metrics: context relevancy, contextual recall, RAGAS
- LLM-as-judge using GPT-4o or a custom evaluator model
- Confident AI platform for evaluation result dashboards
- Red teaming module for safety and jailbreak testing
Real-World Workflows
Write unit tests for your LLM application
- 1Define test cases with input, actual_output, and expected_output
- 2Choose metrics: AnswerRelevancyMetric, HallucinationMetric, etc.
- 3Run: deepeval test run test_llm.py
- 4View pass/fail results per test case with score breakdowns
Red team your LLM for safety issues
- 1Use DeepEval's red teaming module to generate adversarial prompts
- 2Run prompts against your LLM and score with toxicity/bias metrics
- 3Get a vulnerability report with specific attack vectors
- 4Add failing cases as regression tests before release
Getting Started
pip install deepeval
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
def test_answer_relevancy():
test_case = LLMTestCase(
input='What is the capital of France?',
actual_output='Paris is the capital of France.',
)
metric = AnswerRelevancyMetric(threshold=0.7)
assert_test(test_case, [metric])Compare Alternatives
See how DeepEval stacks up against similar tools.