Back to all tools
HELM
Stanford's holistic LLM benchmark — evaluate accuracy, fairness, bias, and efficiency together.
0Free
LLM Evaluation
Overview
HELM (Holistic Evaluation of Language Models) is Stanford CRFM's comprehensive benchmark framework evaluating LLMs across accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.
Key Features
- Holistic metrics: accuracy, calibration, robustness, fairness, bias, efficiency
- 42 scenarios covering NLP, coding, reasoning, and knowledge tasks
- Standardized prompting methodology for fair model comparison
- Public leaderboard comparing GPT-4, Claude, Llama, Gemini, and more
- Modular scenario and metric system for custom evaluations
- Supports local models via HuggingFace and API models
Real-World Workflows
Compare models for a regulated industry use case
- 1Select HELM scenarios relevant to your domain (e.g. medical QA, legal reasoning)
- 2Run evaluation across candidate models
- 3Compare not just accuracy but also toxicity and calibration scores
- 4Choose the model with the best balance of capability and safety metrics
Getting Started
pip install crfm-helm # Run a quick HELM evaluation helm-run \ --conf src/helm/benchmark/presentation/run_specs.conf \ --suite my_eval \ --max-eval-instances 100 \ --models-to-run openai/gpt-4o-mini helm-summarize --suite my_eval helm-server # view results at http://localhost:8000
Compare Alternatives
See how HELM stacks up against similar tools.