Back to all tools
LM Evaluation Harness
The industry-standard LLM benchmark suite — evaluate any model on 60+ tasks from one CLI.
0Open Source
LLM Evaluation
Overview
EleutherAI's LM Evaluation Harness is the industry-standard framework for benchmarking language models across 60+ tasks including MMLU, HellaSwag, TruthfulQA, GSM8K, and HumanEval.
Key Features
- 60+ built-in benchmarks: MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag
- Evaluate any HuggingFace model, OpenAI API, or local model
- Few-shot prompting with configurable shot count
- Parallelized evaluation across GPUs
- Used by Hugging Face Open LLM Leaderboard
- Custom task support via YAML config
Real-World Workflows
Benchmark a fine-tuned model before deployment
- 1Install the harness: pip install lm-eval
- 2Run against your model: lm_eval --model hf --model_args pretrained=your/model --tasks mmlu,gsm8k
- 3Compare accuracy scores against the base model and leading models
- 4Use results to decide if the fine-tune improved or degraded capabilities
Add LLM capability regression tests to CI
- 1Select a fast subset of tasks (e.g. hellaswag with 100 samples)
- 2Run lm-eval in GitHub Actions after each model checkpoint
- 3Fail the build if accuracy drops more than 2% from baseline
- 4Track scores over training runs to detect capability regressions
Getting Started
pip install lm-eval # Evaluate GPT-4o-mini on MMLU and GSM8K lm_eval \ --model openai-chat-completions \ --model_args model=gpt-4o-mini \ --tasks mmlu,gsm8k \ --num_fewshot 5 \ --output_path results/
Compare Alternatives
See how LM Evaluation Harness stacks up against similar tools.