Back to all tools
OpenAI Evals
OpenAI's open-source framework for building and running custom LLM evaluations.
0Open Source
LLM Evaluation
Overview
OpenAI Evals is an open-source framework for evaluating LLMs and AI systems, providing a standard way to define custom evaluation tasks and measure model quality on domain-specific benchmarks.
Key Features
- Built-in eval types: match, includes, fuzzy match, model-graded
- Model-graded evals for open-ended responses
- Custom eval definition via YAML
- Eval registry with hundreds of community-contributed benchmarks
- Compare performance across model versions
- Integration with OpenAI API for automated scoring
Real-World Workflows
Measure quality before upgrading model versions
- 1Define eval tasks from your real production use cases
- 2Create a dataset of inputs and expected outputs
- 3Run evals against both old and new model versions
- 4Only upgrade if the new model scores equal or better on your tasks
Build a domain-specific benchmark
- 1Collect 50–100 representative queries from your application logs
- 2Label ideal responses with human annotators
- 3Define a model-graded eval with GPT-4o as the judge
- 4Track benchmark score in CI on every prompt or model change
Getting Started
pip install evals # Define your eval in a YAML file # evals/my_eval.yaml: # id: my_eval # description: Custom evaluation # metrics: [accuracy] # Run the eval oaieval gpt-4o my_eval \ --record_path results/my_eval.jsonl
Compare Alternatives
See how OpenAI Evals stacks up against similar tools.