metrics

Core metric architecture for evaLLM.

Use mouse to pan and zoom

Use mouse to pan and zoom

This package provides the metric contract, composition model, registration/discovery, and configuration-schema machinery used by evaLLM's evaluation pipeline.

Abstract

evallm.metrics defines the metric subsystem end-to-end: base specifies the shared metric contract and traversal helpers, composite provides recursive multi-metric aggregation, config handles runtime schema definition and validation, registry provides discovery and registration, and builtin contains reference implementations shipped with evaLLM.

Design Decisions

Decision: Use a Composite tree for multi-metric execution.

Motivation: evaLLM must run all configured metrics and merge their outputs for the same input record, not short-circuit through handlers.

Decision: Enforce tree invariants with identity-based membership.

Motivation: CompositeMetric.add/remove operates on node identity (is), not class equivalence, so duplicate classes are allowed while preserving strict parent ownership and cycle-free structure.

Decision: Validate metric value type at class-definition time.

Motivation: BaseMetric[T] captures T on subclass creation, verifies it is serializable, and validates evaluate signature, failing fast for invalid metric implementations.

Decision: Keep metric configuration declarative and strict.

Motivation: config_schema defines runtime parameters; validation happens in BaseMetric.__init__/from_config, and DefaultSchema forbids parameters when none are declared.

Decision: Discover metrics through an import-time catalog.

Motivation: @register_metric writes classes into a Catalog implementation (MetricsCatalog by default), enabling pluggability without hard-coded metric lists.

Decision: Distinguish class identity from result-field identity.

Motivation: name identifies the metric class in the registry, while result_key names one metric instance's output field. This allows multiple differently-configured instances of the same class.

Composite Invariants

The composite implementation enforces the following invariants:

• Tree shape: add/remove operations do not create cycles.
• Single parent: each metric instance has at most one parent at a time.
• Identity-based membership: node identity, not class equality, determines whether two metrics are "the same node" in the tree.

Extension Path for Custom Metrics

1. Subclass BaseMetric[T] with a serializable value type.
2. Implement evaluate(text) -> dict[str, T].
3. Optionally define config_schema (or use @config_schema) for runtime parameters.
4. Optionally register with @register_metric to make the metric discoverable.

Example

A custom metric that computes token lengths.

from evallm.metrics.base import BaseMetric
from evallm.metrics.registry import register_metric


@register_metric(name="token_lengths")  # (1)!
class CustomMetric(
    BaseMetric[list[int]]  # (2)!
):
    def evaluate(self, text: str) -> dict[str, list[int]]:
        res: list[int]  # (3)!

        if text is None or text.strip() == "":  # (4)!
            res = []
        else:  # (5)!
            res = [len(tok) for tok in text.split()]

        return self.result(res)  # (6)!

1. or simply @register_metric to use the metric class' name as the metric's name in the catalog.
2. parametrize BaseMetric with the desired value type (in this case, list[int]).
3. Ensure that the computed value's type matches the metric's declared value_type.
4. Make sure to handle edge cases!
5. Implement the evaluation logic
6. Return the computed value in a dictionary, keyed by the metric's result_key using the result(value) helper method.

1. Parametrizing BaseMetric with a value type

When creating a custom metric, the first question you need to answer is: "What type of value do I want my metric to return?"

This could be any JSON-serializable type, such as a string, number, boolean, list, or dictionary (see SerializableValueType for the full list of supported types).

The metric's value type is then specified by parametrizing the BaseMetric[T] generic class with the desired type (i.e. binding your desired type to T).

2. Implementing the evaluate method

In order for your metric to be used in an evaluation pipeline, you need to tell evaLLM how to compute its value from an input string. At the least, this involves:

• handling edge cases (how should an empty or None input be evaluated?),
• defining the core evaluation logic,
• ensuring that the computed value's type matches with the type you declared in step 1, and
• returning it in a dictionary, keyed by the metric's result_key (the result(value) helper does this for you).

3. Registering your metric

Finally, if you want your metric to be discoverable by evaLLM's evaluation pipelines, decorate your metric class with the @register_metric decorator. By default, this will register your metric in the MetricsCatalog under the name of your metric class (use the name argument of the decorator to specify a custom name ).

Referenced by:

• evaLLM
  • Core Building Blocks
  • Need Architecture Context?

MODULE	DESCRIPTION
base	Abstract generic base class BaseMetric[T] and its DefaultSchema configuration
builtin	Concrete implementations of built-in metrics
composite	CompositeMetric — tree container for other metrics
config	Configuration schema & runtime parameters for metrics
registry	Global registry for automatic metrics discovery
types	Types and serialization helper functions