lm-eval
A framework for evaluating language models (lm-evaluation-harness)
homepage ↗homepage ↗ pypi: lm-eval github: EleutherAI/lm-evaluation-harness
Available in
| Overlay | Newest | Ebuilds | Last activity | |
|---|---|---|---|---|
| stuff GitHub ↗ | 0.4.12 | 2 | 2 d | details › |
Versions & arches
Use flags of 0.4.12
- +api Wire dependencies for OpenAI/HF Inference / generic HTTP API model backends
- ifeval Enable instruction-following evaluation tasks (leaderboard_ifeval and friends) — checks generated text against structural constraints like length limits, language hints, and required keywords. Auto-downloads the NLTK punkt_tab tokenizer at task-load time (not deferred until eval); seed ~/nltk_data ahead of time on offline hosts.
- math Enable math-grading tasks (minerva_math, leaderboard math, hendrycks_math, etc.) — parses LaTeX answers and verifies symbolic equality between predicted and ground-truth solutions
- sentencepiece Pull sci-ml/sentencepiece for tasks that tokenise via SentencePiece
- statsmodels Pull dev-python/statsmodels for the discrim_eval task family
- vllm Wire dev-python/vllm for the vLLM model backend