TroveGrid
Back to all tools
Ragas logo

Ragas

Evaluate your RAG pipeline with LLM-as-judge metrics — faithfulness, relevance, recall.

0Open Source
RAG Frameworks AI Observability
Official Website View on GitHub
Share:XLinkedInWhatsApp

Overview

Ragas is an open-source framework for evaluating RAG pipelines using LLM-based metrics including faithfulness, answer relevance, context precision, and context recall.

Key Features

  • Faithfulness — measures if the answer is grounded in retrieved context
  • Answer Relevance — checks if the answer addresses the question
  • Context Precision and Recall — evaluates retriever quality
  • LLM-as-judge evaluation — no labeled ground truth needed
  • Integrates with LangChain, LlamaIndex, and any RAG pipeline
  • Testset generation — automatically create evaluation datasets

Real-World Workflows

Benchmark RAG pipeline before going to production

  1. 1Generate a test set from your documents with Ragas TestsetGenerator
  2. 2Run your RAG pipeline on all test questions
  3. 3Evaluate with ragas.evaluate() — get faithfulness and relevance scores
  4. 4Iterate on chunk size, retrieval k, and prompt until scores improve

CI quality gate for RAG changes

  1. 1Add a Ragas evaluation step to your GitHub Actions pipeline
  2. 2Fail the build if faithfulness drops below a threshold
  3. 3Track scores over time with W&B or LangSmith integration
  4. 4Catch regressions before they reach production users

Getting Started

pip install ragas

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

data = {
    'question': ['What is RAG?'],
    'answer': ['RAG stands for Retrieval-Augmented Generation...'],
    'contexts': [['RAG is a technique that combines retrieval...']],
    'ground_truth': ['RAG is Retrieval-Augmented Generation']
}

result = evaluate(Dataset.from_dict(data), metrics=[faithfulness, answer_relevancy])
print(result)

Compare Alternatives

See how Ragas stacks up against similar tools.

VSevidentlyVSlangsmithVSarizeVStrulens