CLI

This page documents the command-line interface for evaLLM.

cli

CLI for evaLLM

Use mouse to pan and zoom

Use mouse to pan and zoom

CLI Usage

Evaluate a dataset

Recommended: use a config fileAlternative: use flags only

evallm run --config ./run.yaml

evallm run \
  --input-file ./data/dataset.jsonl \
  --text-key text \
  --record-id-key id \
  --metric CountsMetric

This works best for metrics that require no runtime configuration, or for preset forms.

Configuration model

evallm run needs two top-level sections:

1. dataset
   • input_file (required): path to JSONL file
   • text_key (required): key containing text to evaluate
   • record_id_key (optional): key for record IDs (#IDX# fallback)
2. metrics
   • required list of one or more metric specs
   • each spec supports name, optional config, optional result_key

You can provide configuration in two ways:

• through CLI flags, or
• as a config file (--config, JSON or YAML)

Warning

If both are used, config file values are loaded first and CLI flags override them. If any --metric flags are passed, they replace config.metrics!

Supported --metric forms

Registered metric nameConvenience presetPreset with argumentInline JSON metric specMetric spec from JSON/YAML file

evallm run -f ./data/dataset.jsonl -t text -i id -m CountsMetric

evallm run -f ./data/dataset.jsonl -t text -i id -m count_characters

evallm run -f ./data/dataset.jsonl -t text -i id -m count_tokens:wordpunct
evallm run -f ./data/dataset.jsonl -t text -i id -m count_tokens:whitespace
evallm run -f ./data/dataset.jsonl -t text -i id -m count_encodings:bert-base-uncased

evallm run \
  -f ./data/dataset.jsonl -t text -i id \
  -m '{"name":"CountsMetric","config":{"segments":"tokens"},"result_key":"token_count"}'

evallm run -f ./data/dataset.jsonl -t text -i id -m ./metrics/counts.json
evallm run -f ./data/dataset.jsonl -t text -i id -m ./metrics/counts.yaml

Configuration file

YAMLJSON

# eval_config.yml
dataset:
    input_file: ./data/dataset.jsonl
    text_key: text
    record_id_key: id
metrics:
    - name: CountsMetric
        result_key: chars
    - name: CountsMetric
        config:
        segments: tokens
        segmentation: whitespace
        result_key: tokens
    - name: CountsMetric
        config:
        segments: encodings
        segmentation: meta-llama/Llama-3.1-8B
        result_key: llama_3_1
    - name: JsonFormatMetric
        config:
        expected_schema:
            type: object
            properties:
            question:
                type: string
                pattern: '\?$'
            setup:
                type: string
            joke:
                type: string
            funniness_score:
                type: number
                minimum: 0.0
                maximum: 1.0
            required:
            - question
            - joke
        check_formats: true
        result_key: structured_output

// eval_config.json
{
    "dataset": {
        "input_file": "./data/dataset.jsonl",
        "text_key": "text",
        "record_id_key": "id"
    },
    "metrics": [
        {
            "name": "CountsMetric",
            "result_key": "chars"
        },
        {
            "name": "CountsMetric",
            "config": {
                "segments": "tokens",
                "segmentation": "whitespace"
            },
            "result_key": "tokens"
        },
        {
            "name": "CountsMetric",
            "config": {
                "segments": "encodings",
                "segmentation": "meta-llama/Llama-3.1-8B"
            },
            "result_key": "llama_3_1"
        },
        {
            "name": "JsonFormatMetric",
            "config": {
                "expected_schema": {
                    "type": "object",
                    "properties": {
                        "question": {
                            "type": "string",
                            "pattern": "\\?$"
                        },
                        "setup": {
                            "type": "string"
                        },
                        "joke": {
                            "type": "string"
                        },
                        "funniness_score": {
                            "type": "number",
                            "minimum": 0.0,
                            "maximum": 1.0
                        }
                    },
                    "required": [
                        "question",
                        "joke"
                    ]
                },
                "check_formats": true
            },
            "result_key": "structured_output"
        }
    ]
}

Process output

Each record's evaluation result is serialized as a JSON object. By default, results are streamed to the console, which lets you pipe them however you want. You can of course also have them written to a results file in JSONL format.

Stream to stdout (default)Write to a file

evallm run -f ./data/dataset.jsonl -t text -i id -m CountsMetric

evallm run \
  -f ./data/dataset.jsonl -t text -i id \
  -m CountsMetric \
  --output ./results.jsonl

Help and discovery

General help

evallm --help
evallm run --help
evallm metrics --help

Metric discovery and configuration help

evallm metrics list
evallm metrics info CountsMetric
evallm metrics info CountsMetric --format json
evallm metrics info CountsMetric --format yaml
evallm metrics presets

Referenced by:

• evaLLM

MODULE	DESCRIPTION
main	Use mouse to pan and zoom Use mouse to pan and zoom Typer-based CLI for evaLLM evaluation.