evaLLM

Evaluate LLM outputs with composable, schema-validated metrics.

Get Started API Reference

Why evaLLM?

• Composable metrics: Combine multiple metrics and run them in one pass.
• Validated configuration: Metric/runtime config is schema-checked.
• JSONL-first workflow: Stream record-level results for easy downstream analysis.
• Two interfaces: Use either the Python API or the CLI.

Quick Start

Install

pip install evallm-qa

Optional extras for HuggingFace tokenizer-backed encoding metrics:

pip install "evallm-qa[hf-tokenizers]"

Run your first evaluation

evallm run \
    --input-file ./data/dataset.jsonl \
    --text-key text \
    --record-id-key id \
    --metric CountsMetric

Learn by Interface

Python API

• Start at evallm.application
• Use evallm.application.api for builders and metric helper factories

CLI

• See evallm.cli
• Explore metrics with:

evallm metrics list
evallm metrics info CountsMetric
evallm metrics presets

Core Building Blocks

• Application layer: request modeling, validation, orchestration
• Metrics layer: base interfaces, built-ins, composite container, registry
• Readers layer: dataset readers and adapters

Need Architecture Context?

The diagram above gives a system-level view. For internals, continue with:

• Architecture overview
• Application API reference
• Metrics reference
• Readers reference