application

Application orchestration layer for evaLLM.

Use mouse to pan and zoom

Use mouse to pan and zoom

This package coordinates dataset access and metric execution while exposing a stable public API for library and CLI consumers.

Abstract

evallm.application is the orchestration layer that turns validated requests into executable evaluation flows. api exposes ergonomic construction primitives, models defines canonical contracts, services resolves and executes requests, adapters normalize raw reader output into DTOs, validation enforces cross-object constraints, and errors defines user-facing failure semantics.

Design Decisions

Decision: Use immutable DTOs as canonical request/result contracts.

Motivation: models uses frozen pydantic DTOs to keep boundary data validated, serializable, and mutation-safe across layers.

Decision: Separate structural validation from runtime/contextual resolution.

Motivation: shape validation happens at DTO creation, while checks requiring runtime context (catalog membership, metric config validation, reader binding) happen in resolver/services.

Decision: Resolve requests into two runtime artifacts.

Motivation: RequestResolver returns (records_context_manager, composite_metric), decoupling data streaming from metric composition/execution.

Decision: Enforce result-key uniqueness before execution.

Motivation: per-record results are merged into a single dictionary, so duplicate result keys would overwrite values unless rejected early.

Decision: Keep request construction ergonomic via builder + helpers.

Motivation: BaseEvaluationBuilder/ JsonlEvaluationBuilder and helper factories (metric, count_*, json_format) provide concise API ergonomics while still yielding canonical DTOs.

Decision: Depend on abstract contracts where possible.

Motivation: using abstractions like Catalog improves testability and supports custom registries without changing orchestration logic.

Referenced by:

• evaLLM
  • Core Building Blocks
  • Learn by Interface Python API

MODULE	DESCRIPTION
adapters	Use mouse to pan and zoom Use mouse to pan and zoom Internal adapters for application-layer DTOs
api	Use mouse to pan and zoom Use mouse to pan and zoom Public API, providing an interface for evaluation
errors	Use mouse to pan and zoom Use mouse to pan and zoom Application-layer exceptions
models	Use mouse to pan and zoom Use mouse to pan and zoom Canonical internal data contracts as DTOs
services	Use mouse to pan and zoom Use mouse to pan and zoom Application services for evaLLM
validation	Use mouse to pan and zoom Use mouse to pan and zoom Shared validation helpers

CLASS	DESCRIPTION
BaseEvaluationBuilder	Abstract mutable builder for creating evaluation requests.
JsonlEvaluationBuilder	Concrete builder for JSONL-based evaluations.
BaseDatasetSpec	Abstract base class for dataset specifications.
DatasetRecord	DTO representing a single record to evaluate in a dataset
EvaluationRequest	DTO specifying which configuration of metrics to evaluate a given dataset on
EvaluationResult	DTO representing the evaluated metrics' results for a single record
JsonlDatasetSpec	DTO specifying a structured JSONL dataset to evaluate
MetricSpec	DTO specifying a metric and its configuration
Evaluator	Component responsible for orchestrating the evaluation process.

FUNCTION	DESCRIPTION
count_characters	Create a metric spec for counting the number of characters in the evaluated text.
count_encodings	Create a metric spec for counting the number of model-specific encodings in a text.
count_tokens	Create a metric spec for counting the number of tokens in the evaluated text.
json_format	Create a metric spec for validating JSON output format.
metric	Create a generic metric spec using a metric's name.

Classes

BaseEvaluationBuilder

BaseEvaluationBuilder(*, catalog: type[Catalog] = MetricsCatalog)

              flowchart TD
              evallm.application.BaseEvaluationBuilder[BaseEvaluationBuilder]

              

              click evallm.application.BaseEvaluationBuilder href "" "evallm.application.BaseEvaluationBuilder"
            

Use mouse to pan and zoom

Abstract mutable builder for creating evaluation requests.

METHOD	DESCRIPTION
use	Attach a single metric spec and return the same builder instance.
build_dataset	Build the dataset spec for the concrete builder.
build_request	Build the canonical evaluation request.
build_evaluator	Build an evaluator from the current builder state.
run	Lazily execute evaluation and yield individual results.
from_request	Rebuild a concrete builder from a canonical request.

ATTRIBUTE	DESCRIPTION
catalog	Immutable catalog used for metric resolution. TYPE: type[Catalog]
metrics	Immutable view of attached metric specs. TYPE: tuple[MetricSpec, ...]

Source code in src/evallm/application/api.py

def __init__(self, *, catalog: type[Catalog] = MetricsCatalog) -> None:
    if not issubclass(catalog, Catalog):
        raise TypeError(
            f"Provided catalog {catalog.__name__} doesn't implement the Catalog interface"
        )
    self._catalog = catalog
    self._metrics = []

Attributes

catalog property

catalog: type[Catalog]

Immutable catalog used for metric resolution.

RETURNS	DESCRIPTION
type[Catalog]	type[Catalog]: Catalog class used for metric resolution in this builder.

metrics property

metrics: tuple[MetricSpec, ...]

Immutable view of attached metric specs.

RETURNS	DESCRIPTION
tuple[MetricSpec, ...]	tuple[MetricSpec, ...]: Tuple of currently attached metric specs.

Functions

use

use(metric_spec: MetricSpec) -> Self

Attach a single metric spec and return the same builder instance.

RETURNS	DESCRIPTION
Self	The mutated builder instance. TYPE: Self

RAISES	DESCRIPTION
TypeError	If metric_spec is not a MetricSpec instance.
DuplicateMetricResultKeyError	If attaching the metric would cause a final result-key collision.

Source code in src/evallm/application/api.py

def use(self, metric_spec: MetricSpec) -> Self:
    """Attach a single metric spec and return the same builder instance.

    Returns:
        Self: The mutated builder instance.

    Raises:
        TypeError: If `metric_spec` is not a
            [`MetricSpec`][evallm.application.models.MetricSpec] instance.
        DuplicateMetricResultKeyError: If attaching the metric would cause a final
            result-key collision.
    """
    if not isinstance(metric_spec, MetricSpec):
        raise TypeError("metric_spec must be an instance of MetricSpec")

    try:
        ensure_unique_metric_result_keys([*self._metrics, metric_spec])
    except DuplicateMetricResultKeyError as e:
        raise e

    self._metrics.append(metric_spec)
    return self

build_dataset abstractmethod

build_dataset() -> DatasetSpecT

Build the dataset spec for the concrete builder.

Source code in src/evallm/application/api.py

@abstractmethod
def build_dataset(self) -> DatasetSpecT:
    """Build the dataset spec for the concrete builder."""

build_request

build_request() -> EvaluationRequest

Build the canonical evaluation request.

RETURNS	DESCRIPTION
EvaluationRequest	Canonical request assembled from current builder state. TYPE: EvaluationRequest

RAISES	DESCRIPTION
ValueError	If no metrics are attached.

Source code in src/evallm/application/api.py

def build_request(self) -> EvaluationRequest:
    """Build the canonical evaluation request.

    Returns:
        EvaluationRequest: Canonical request assembled from current builder state.

    Raises:
        ValueError: If no metrics are attached.
    """
    if not self._metrics:
        raise ValueError("At least one metric must be attached before building")
    return EvaluationRequest(
        dataset=self.build_dataset(), metrics=list(self._metrics)
    )

build_evaluator

build_evaluator() -> Evaluator

Build an evaluator from the current builder state.

RETURNS	DESCRIPTION
Evaluator	Evaluator configured with the built request and catalog. TYPE: Evaluator

Source code in src/evallm/application/api.py

def build_evaluator(self) -> Evaluator:
    """Build an evaluator from the current builder state.

    Returns:
        Evaluator: Evaluator configured with the built request and catalog.
    """
    return Evaluator(self.build_request(), catalog=self._catalog)

run

run() -> Iterator[EvaluationResult]

Lazily execute evaluation and yield individual results.

YIELDS	DESCRIPTION
EvaluationResult	Per-record evaluation result. TYPE:: EvaluationResult

Source code in src/evallm/application/api.py

def run(self) -> Iterator[EvaluationResult]:
    """Lazily execute evaluation and yield individual results.

    Yields:
        EvaluationResult: Per-record evaluation result.
    """
    yield from self.build_evaluator().run()

from_request abstractmethod classmethod

from_request(
    request: EvaluationRequest, *, catalog: type[Catalog] = MetricsCatalog
) -> Self

Rebuild a concrete builder from a canonical request.

Source code in src/evallm/application/api.py

@classmethod
@abstractmethod
def from_request(
    cls,
    request: EvaluationRequest,
    *,
    catalog: type[Catalog] = MetricsCatalog,
) -> Self:
    """Rebuild a concrete builder from a canonical request."""
    raise NotImplementedError(
        "from_request must be implemented by concrete builder classes"
    )  # pragma: no cover

JsonlEvaluationBuilder

JsonlEvaluationBuilder(
    input_file: str | PathLike[str],
    *,
    text_key: str,
    record_id_key: str = "#IDX#",
    catalog: type[Catalog] = MetricsCatalog,
)

              flowchart TD
              evallm.application.JsonlEvaluationBuilder[JsonlEvaluationBuilder]
              evallm.application.api.BaseEvaluationBuilder[BaseEvaluationBuilder]

                              evallm.application.api.BaseEvaluationBuilder --> evallm.application.JsonlEvaluationBuilder
                


              click evallm.application.JsonlEvaluationBuilder href "" "evallm.application.JsonlEvaluationBuilder"
              click evallm.application.api.BaseEvaluationBuilder href "" "evallm.application.api.BaseEvaluationBuilder"
            

Use mouse to pan and zoom

Concrete builder for JSONL-based evaluations.

Returned by:

• API Reference evallm application
  •  api Classes  JsonlEvaluationBuilder Functions  from_request
  •  application Classes  JsonlEvaluationBuilder Functions  from_request

METHOD	DESCRIPTION
use	Attach a single metric spec and return the same builder instance.
build_request	Build the canonical evaluation request.
build_evaluator	Build an evaluator from the current builder state.
run	Lazily execute evaluation and yield individual results.
build_dataset	Build a JSONL dataset spec from the builder state.
from_request	Rebuild a JSONL builder from an existing request.

ATTRIBUTE	DESCRIPTION
catalog	Immutable catalog used for metric resolution. TYPE: type[Catalog]
metrics	Immutable view of attached metric specs. TYPE: tuple[MetricSpec, ...]
input_file	Input JSONL dataset path. TYPE: Path
text_key	Key of the field containing text to evaluate in each JSONL record. TYPE: str
record_id_key	Key of the field containing record ids. TYPE: str

Source code in src/evallm/application/api.py

def __init__(
    self,
    input_file: str | os.PathLike[str],
    *,
    text_key: str,
    record_id_key: str = "#IDX#",
    catalog: type[Catalog] = MetricsCatalog,
) -> None:
    super().__init__(catalog=catalog)
    self._input_file = Path(input_file)
    self._text_key = text_key
    self._record_id_key = record_id_key

Attributes

catalog property

catalog: type[Catalog]

Immutable catalog used for metric resolution.

RETURNS	DESCRIPTION
type[Catalog]	type[Catalog]: Catalog class used for metric resolution in this builder.

metrics property

metrics: tuple[MetricSpec, ...]

Immutable view of attached metric specs.

RETURNS	DESCRIPTION
tuple[MetricSpec, ...]	tuple[MetricSpec, ...]: Tuple of currently attached metric specs.

input_file property

input_file: Path

Input JSONL dataset path.

RETURNS	DESCRIPTION
Path	The input JSONL dataset as a normalized path. TYPE: Path

text_key property

text_key: str

Key of the field containing text to evaluate in each JSONL record.

RETURNS	DESCRIPTION
str	The field's name. TYPE: str

record_id_key property

record_id_key: str

Key of the field containing record ids.

RETURNS	DESCRIPTION
str	str | Literal["#IDX#"]: The field's name or the reserved literal "#IDX#" if record ids aren't present and should instead be auto-generated as 0-based indices.

Functions

use

use(metric_spec: MetricSpec) -> Self

Attach a single metric spec and return the same builder instance.

RETURNS	DESCRIPTION
Self	The mutated builder instance. TYPE: Self

RAISES	DESCRIPTION
TypeError	If metric_spec is not a MetricSpec instance.
DuplicateMetricResultKeyError	If attaching the metric would cause a final result-key collision.

Source code in src/evallm/application/api.py

def use(self, metric_spec: MetricSpec) -> Self:
    """Attach a single metric spec and return the same builder instance.

    Returns:
        Self: The mutated builder instance.

    Raises:
        TypeError: If `metric_spec` is not a
            [`MetricSpec`][evallm.application.models.MetricSpec] instance.
        DuplicateMetricResultKeyError: If attaching the metric would cause a final
            result-key collision.
    """
    if not isinstance(metric_spec, MetricSpec):
        raise TypeError("metric_spec must be an instance of MetricSpec")

    try:
        ensure_unique_metric_result_keys([*self._metrics, metric_spec])
    except DuplicateMetricResultKeyError as e:
        raise e

    self._metrics.append(metric_spec)
    return self

build_request

build_request() -> EvaluationRequest

Build the canonical evaluation request.

RETURNS	DESCRIPTION
EvaluationRequest	Canonical request assembled from current builder state. TYPE: EvaluationRequest

RAISES	DESCRIPTION
ValueError	If no metrics are attached.

Source code in src/evallm/application/api.py

def build_request(self) -> EvaluationRequest:
    """Build the canonical evaluation request.

    Returns:
        EvaluationRequest: Canonical request assembled from current builder state.

    Raises:
        ValueError: If no metrics are attached.
    """
    if not self._metrics:
        raise ValueError("At least one metric must be attached before building")
    return EvaluationRequest(
        dataset=self.build_dataset(), metrics=list(self._metrics)
    )

build_evaluator

build_evaluator() -> Evaluator

Build an evaluator from the current builder state.

RETURNS	DESCRIPTION
Evaluator	Evaluator configured with the built request and catalog. TYPE: Evaluator

Source code in src/evallm/application/api.py

def build_evaluator(self) -> Evaluator:
    """Build an evaluator from the current builder state.

    Returns:
        Evaluator: Evaluator configured with the built request and catalog.
    """
    return Evaluator(self.build_request(), catalog=self._catalog)

run

run() -> Iterator[EvaluationResult]

Lazily execute evaluation and yield individual results.

YIELDS	DESCRIPTION
EvaluationResult	Per-record evaluation result. TYPE:: EvaluationResult

Source code in src/evallm/application/api.py

def run(self) -> Iterator[EvaluationResult]:
    """Lazily execute evaluation and yield individual results.

    Yields:
        EvaluationResult: Per-record evaluation result.
    """
    yield from self.build_evaluator().run()

build_dataset

build_dataset() -> JsonlDatasetSpec

Build a JSONL dataset spec from the builder state.

RETURNS	DESCRIPTION
JsonlDatasetSpec	JSONL dataset spec built from current builder fields. TYPE: JsonlDatasetSpec

Source code in src/evallm/application/api.py

def build_dataset(self) -> JsonlDatasetSpec:
    """Build a JSONL dataset spec from the builder state.

    Returns:
        JsonlDatasetSpec: JSONL dataset spec built from current builder fields.
    """
    return JsonlDatasetSpec(
        input_file=self._input_file,
        text_key=self._text_key,
        record_id_key=self._record_id_key,
    )

from_request classmethod

from_request(
    request: EvaluationRequest, *, catalog: type[Catalog] = MetricsCatalog
) -> JsonlEvaluationBuilder

Rebuild a JSONL builder from an existing request.

RETURNS	DESCRIPTION
JsonlEvaluationBuilder	Builder reconstructed from the provided request. TYPE: JsonlEvaluationBuilder

RAISES	DESCRIPTION
TypeError	If request.dataset is not a JsonlDatasetSpec.

Source code in src/evallm/application/api.py

@classmethod
def from_request(
    cls,
    request: EvaluationRequest,
    *,
    catalog: type[Catalog] = MetricsCatalog,
) -> JsonlEvaluationBuilder:
    """Rebuild a JSONL builder from an existing request.

    Returns:
        JsonlEvaluationBuilder: Builder reconstructed from the provided request.

    Raises:
        TypeError: If `request.dataset` is not a
            [`JsonlDatasetSpec`][evallm.application.models.JsonlDatasetSpec].
    """
    if not isinstance(request.dataset, JsonlDatasetSpec):
        raise TypeError(
            "JsonlEvaluationBuilder requires an EvaluationRequest with JsonlDatasetSpec"
        )

    builder = cls(
        request.dataset.input_file,
        text_key=request.dataset.text_key,
        record_id_key=request.dataset.record_id_key,
        catalog=catalog,
    )
    for spec in request.metrics:
        builder.use(spec)
    return builder

BaseDatasetSpec pydantic-model

Abstract base class for dataset specifications.

Fields:

• data_format (str)

Attributes

data_format pydantic-field

data_format: str

Dataset format discriminator used for dataset spec resolution.

Functions

create classmethod

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
obj	The DTO's fields as a native Python object (e.g. a dict). TYPE: Any | None DEFAULT: None
json_payload	The DTO's fields as a JSON string- or bytes-payload. TYPE: str | bytes | bytearray | None DEFAULT: None
**kwargs	Keyword arguments corresponding to the DTO's fields. TYPE: Any DEFAULT: {}

RETURNS	DESCRIPTION
Self	An instance of the DTO created from the provided input. TYPE: Self

RAISES	DESCRIPTION
ValueError	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

infer_data_format classmethod

infer_data_format(data: Mapping[str, Any]) -> str | None

Infer dataset format from the input-file suffix, if possible.

PARAMETER	DESCRIPTION
data	Raw dataset payload. TYPE: Mapping[str, Any]

RETURNS	DESCRIPTION
str | None	str | None: Inferred dataset format, or None when no inference is possible.

Source code in src/evallm/application/models.py

@classmethod
def infer_data_format(cls, data: Mapping[str, Any]) -> str | None:
    """Infer dataset format from the input-file suffix, if possible.

    Args:
        data (Mapping[str, Any]): Raw dataset payload.

    Returns:
        str | None: Inferred dataset format, or `None` when no inference is
            possible.
    """
    path_like = data.get("input_file")
    if path_like is None:  # pragma: no cover
        return None
    suffix = Path(str(path_like)).suffix.strip().lower().lstrip(".")
    if not suffix:
        return None
    return suffix

resolve classmethod

resolve(data: BaseDatasetSpec | Mapping[str, Any]) -> BaseDatasetSpec

Resolve a raw or already-validated dataset payload to a concrete dataset DTO.

PARAMETER	DESCRIPTION
data	Raw or typed dataset payload. TYPE: BaseDatasetSpec | Mapping[str, Any]

RETURNS	DESCRIPTION
BaseDatasetSpec	Resolved dataset spec instance. TYPE: BaseDatasetSpec

RAISES	DESCRIPTION
ValueError	If the dataset format is missing, unsupported, or conflicts with the inferred format.

Source code in src/evallm/application/models.py

@classmethod
def resolve(
    cls,
    data: BaseDatasetSpec | Mapping[str, Any],
) -> BaseDatasetSpec:
    """Resolve a raw or already-validated dataset payload to a concrete dataset DTO.

    Args:
        data (BaseDatasetSpec | Mapping[str, Any]): Raw or typed dataset payload.

    Returns:
        BaseDatasetSpec: Resolved dataset spec instance.

    Raises:
        ValueError: If the dataset format is missing, unsupported, or conflicts
            with the inferred format.
    """
    if isinstance(data, BaseDatasetSpec):  # pragma: no cover
        return data

    payload: dict[str, Any] = dict(data)
    explicit_value = payload.get("data_format")
    explicit: str | None = None
    if explicit_value is not None:
        if (
            not isinstance(explicit_value, str) or not explicit_value.strip()
        ):  # pragma: no cover
            raise ValueError("dataset.data_format must be a non-empty string")
        explicit = explicit_value.strip().lower()

    inferred = cls.infer_data_format(payload)
    if explicit is not None and inferred is not None and explicit != inferred:
        raise ValueError(
            "dataset.data_format does not match inferred format from input_file "
            + f"(explicit={explicit!r}, inferred={inferred!r})"
        )

    resolved_format = explicit or inferred
    if resolved_format is None:
        raise ValueError(
            "Could not infer dataset.data_format from input_file. "
            + "Provide dataset.data_format explicitly."
        )

    if resolved_format not in cls._specs_by_format:
        available = ", ".join(sorted(cls._specs_by_format))
        raise ValueError(
            f"Unsupported dataset format {resolved_format!r}. "
            + f"Available formats: {available}."
        )

    payload["data_format"] = resolved_format
    dataset_cls = cls._specs_by_format[resolved_format]
    return dataset_cls.model_validate(payload)

DatasetRecord pydantic-model

DTO representing a single record to evaluate in a dataset

Fields:

• record_id (str)
• text (str)
• metadata (dict[str, JsonValue] | None)

Validators:

• _stringify_text → text

Attributes

record_id pydantic-field

record_id: str

The record's unique identifier, specified as a string. The value of this field corresponds to the dataset field specified by JsonlDatasetSpec.record_id_key.

Referenced by:

• API Reference evallm application
  •  application Classes
    •  DatasetRecord Attributes  text
    •  EvaluationResult Attributes  record_id
  •  models Classes
    •  DatasetRecord Attributes  text
    •  EvaluationResult Attributes  record_id

text pydantic-field

text: str

The record's value to evaluate. The value of this field corresponds to the dataset field specified by JsonlDatasetSpec.text_key, and indexed by record_id.

Type coercion for text field

During validation, None values will be converted to the empty string (i.e. ""), values of JSON-serializable types that aren't strings will be converted to their string representation (e.g. {'key': 123} will become "{'key': 123}"), and string values are kept as-is (e.g. " " will remain " ").

metadata pydantic-field

metadata: dict[str, JsonValue] | None = None

Optional dictionary containing any additional fields from the dataset record.

Functions

create classmethod

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
obj	The DTO's fields as a native Python object (e.g. a dict). TYPE: Any | None DEFAULT: None
json_payload	The DTO's fields as a JSON string- or bytes-payload. TYPE: str | bytes | bytearray | None DEFAULT: None
**kwargs	Keyword arguments corresponding to the DTO's fields. TYPE: Any DEFAULT: {}

RETURNS	DESCRIPTION
Self	An instance of the DTO created from the provided input. TYPE: Self

RAISES	DESCRIPTION
ValueError	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

EvaluationRequest pydantic-model

DTO specifying which configuration of metrics to evaluate a given dataset on

Fields:

• dataset (BaseDatasetSpec)
• metrics (list[MetricSpec])

Validators:

• _validate_dataset_spec → dataset

Attributes

dataset pydantic-field

dataset: BaseDatasetSpec

The dataset to evaluate, specified as a concrete BaseDatasetSpec subtype.

Referenced by:

• API Reference evallm application  application Classes  Evaluator

metrics pydantic-field

metrics: list[MetricSpec]

List of one or more metrics to use for evaluation, each specified as a MetricSpec.

Functions

create classmethod

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
obj	The DTO's fields as a native Python object (e.g. a dict). TYPE: Any | None DEFAULT: None
json_payload	The DTO's fields as a JSON string- or bytes-payload. TYPE: str | bytes | bytearray | None DEFAULT: None
**kwargs	Keyword arguments corresponding to the DTO's fields. TYPE: Any DEFAULT: {}

RETURNS	DESCRIPTION
Self	An instance of the DTO created from the provided input. TYPE: Self

RAISES	DESCRIPTION
ValueError	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

EvaluationResult pydantic-model

DTO representing the evaluated metrics' results for a single record

Fields:

• record_id (str)
• results (dict[str, JsonValue])

Attributes

record_id pydantic-field

record_id: str

The unique identifier for the evaluated record, specified as a string. The value of this field corresponds to the dataset field specified by JsonlDatasetSpec.record_id_key and mirrors the value of the corresponding DatasetRecord.record_id.

results pydantic-field

results: dict[str, JsonValue]

A dictionary mapping metric result keys to their computed results.

Each key corresponds to the resolved metric instance's result_key, which defaults to the metric's name unless explicitly overridden.

See also

• MetricSpec.result_key
• BaseMetric.result_key
• CompositeMetric.evaluate

Functions

create classmethod

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
obj	The DTO's fields as a native Python object (e.g. a dict). TYPE: Any | None DEFAULT: None
json_payload	The DTO's fields as a JSON string- or bytes-payload. TYPE: str | bytes | bytearray | None DEFAULT: None
**kwargs	Keyword arguments corresponding to the DTO's fields. TYPE: Any DEFAULT: {}

RETURNS	DESCRIPTION
Self	An instance of the DTO created from the provided input. TYPE: Self

RAISES	DESCRIPTION
ValueError	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

JsonlDatasetSpec pydantic-model

DTO specifying a structured JSONL dataset to evaluate

Fields:

• data_format (Literal['jsonl'])
• input_file (FilePath)
• record_id_key (str)
• text_key (str)

Validators:

• _validate_jsonl_path → input_file
• _validate_keys

Attributes

data_format pydantic-field

data_format: Literal['jsonl'] = 'jsonl'

Dataset format discriminator.

input_file pydantic-field

input_file: FilePath

Path to a local JSONL-file containing the dataset to evaluate.

record_id_key pydantic-field

record_id_key: str = '#IDX#'

The dataset field/ column name that contains the unique identifier for each record, specified as a non-empty string. Defaults to "#IDX#", which indicates that record IDs should be generated based on the record's (0-indexed) position in the dataset.

Referenced by:

• API Reference evallm application
  •  application Classes
    •  DatasetRecord Attributes  record_id
    •  EvaluationResult Attributes  record_id
    •  Evaluator
  •  models Classes
    •  DatasetRecord Attributes  record_id
    •  EvaluationResult Attributes  record_id

text_key pydantic-field

text_key: str

The dataset field/ column name that contains the text to evaluate, specified as a non-empty string.

Referenced by:

• API Reference evallm application
  •  application Classes
    •  DatasetRecord Attributes  text
    •  Evaluator
  •  models Classes  DatasetRecord Attributes  text

Functions

create classmethod

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
obj	The DTO's fields as a native Python object (e.g. a dict). TYPE: Any | None DEFAULT: None
json_payload	The DTO's fields as a JSON string- or bytes-payload. TYPE: str | bytes | bytearray | None DEFAULT: None
**kwargs	Keyword arguments corresponding to the DTO's fields. TYPE: Any DEFAULT: {}

RETURNS	DESCRIPTION
Self	An instance of the DTO created from the provided input. TYPE: Self

RAISES	DESCRIPTION
ValueError	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

infer_data_format classmethod

infer_data_format(data: Mapping[str, Any]) -> str | None

Infer dataset format from the input-file suffix, if possible.

PARAMETER	DESCRIPTION
data	Raw dataset payload. TYPE: Mapping[str, Any]

RETURNS	DESCRIPTION
str | None	str | None: Inferred dataset format, or None when no inference is possible.

Source code in src/evallm/application/models.py

@classmethod
def infer_data_format(cls, data: Mapping[str, Any]) -> str | None:
    """Infer dataset format from the input-file suffix, if possible.

    Args:
        data (Mapping[str, Any]): Raw dataset payload.

    Returns:
        str | None: Inferred dataset format, or `None` when no inference is
            possible.
    """
    path_like = data.get("input_file")
    if path_like is None:  # pragma: no cover
        return None
    suffix = Path(str(path_like)).suffix.strip().lower().lstrip(".")
    if not suffix:
        return None
    return suffix

resolve classmethod

resolve(data: BaseDatasetSpec | Mapping[str, Any]) -> BaseDatasetSpec

Resolve a raw or already-validated dataset payload to a concrete dataset DTO.

PARAMETER	DESCRIPTION
data	Raw or typed dataset payload. TYPE: BaseDatasetSpec | Mapping[str, Any]

RETURNS	DESCRIPTION
BaseDatasetSpec	Resolved dataset spec instance. TYPE: BaseDatasetSpec

RAISES	DESCRIPTION
ValueError	If the dataset format is missing, unsupported, or conflicts with the inferred format.

Source code in src/evallm/application/models.py

@classmethod
def resolve(
    cls,
    data: BaseDatasetSpec | Mapping[str, Any],
) -> BaseDatasetSpec:
    """Resolve a raw or already-validated dataset payload to a concrete dataset DTO.

    Args:
        data (BaseDatasetSpec | Mapping[str, Any]): Raw or typed dataset payload.

    Returns:
        BaseDatasetSpec: Resolved dataset spec instance.

    Raises:
        ValueError: If the dataset format is missing, unsupported, or conflicts
            with the inferred format.
    """
    if isinstance(data, BaseDatasetSpec):  # pragma: no cover
        return data

    payload: dict[str, Any] = dict(data)
    explicit_value = payload.get("data_format")
    explicit: str | None = None
    if explicit_value is not None:
        if (
            not isinstance(explicit_value, str) or not explicit_value.strip()
        ):  # pragma: no cover
            raise ValueError("dataset.data_format must be a non-empty string")
        explicit = explicit_value.strip().lower()

    inferred = cls.infer_data_format(payload)
    if explicit is not None and inferred is not None and explicit != inferred:
        raise ValueError(
            "dataset.data_format does not match inferred format from input_file "
            + f"(explicit={explicit!r}, inferred={inferred!r})"
        )

    resolved_format = explicit or inferred
    if resolved_format is None:
        raise ValueError(
            "Could not infer dataset.data_format from input_file. "
            + "Provide dataset.data_format explicitly."
        )

    if resolved_format not in cls._specs_by_format:
        available = ", ".join(sorted(cls._specs_by_format))
        raise ValueError(
            f"Unsupported dataset format {resolved_format!r}. "
            + f"Available formats: {available}."
        )

    payload["data_format"] = resolved_format
    dataset_cls = cls._specs_by_format[resolved_format]
    return dataset_cls.model_validate(payload)

MetricSpec pydantic-model

DTO specifying a metric and its configuration

Fields:

• name (str)
• config (dict[str, JsonValue] | None)
• result_key (str | None)

Attributes

name pydantic-field

name: str

Non-empty name under which a metric is registered.

See also

• BaseMetric.name
• @register_metric
• MetricsCatalog.available

Referenced by:

• API Reference evallm application
  •  application Classes
    •  Evaluator
    •  MetricSpec Functions  resolve_metric
  •  models Classes  MetricSpec Functions  resolve_metric

config pydantic-field

config: dict[str, JsonValue] | None = None

Runtime configuration for the metric or None (default), if not needed.

See also

• BaseMetric.config_schema
• BaseMetric.from_config

Referenced by:

• API Reference evallm application
  •  application Classes
    •  Evaluator
    •  MetricSpec Functions  resolve_metric
  •  models Classes  MetricSpec Functions  resolve_metric

result_key pydantic-field

result_key: str | None = None

Optional key to use in the evaluation result dictionary for this metric instance's value.

Referenced by:

• API Reference evallm application
  •  application Classes  EvaluationResult Attributes  results
  •  models Classes  EvaluationResult Attributes  results

Functions

create classmethod

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
obj	The DTO's fields as a native Python object (e.g. a dict). TYPE: Any | None DEFAULT: None
json_payload	The DTO's fields as a JSON string- or bytes-payload. TYPE: str | bytes | bytearray | None DEFAULT: None
**kwargs	Keyword arguments corresponding to the DTO's fields. TYPE: Any DEFAULT: {}

RETURNS	DESCRIPTION
Self	An instance of the DTO created from the provided input. TYPE: Self

RAISES	DESCRIPTION
ValueError	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

resolve_metric

resolve_metric(catalog: type[Catalog]) -> BaseMetric

Resolve the metric specification to a BaseMetric instance.

Performs contextual validation of the DTO by:

1. looking up the specified name in the provided catalog and
2. trying to instantiate the metric with the provided config.

If either of these steps fail, an appropriate error is raised.

Metric instance's result_key

The resolved metric instance's result_key attribute is not yet set to the value of this DTO's result_key field. This is because at this point there's no guarantee that the provided result_key is unique among all resolved metrics for a given evaluation request.

PARAMETER	DESCRIPTION
catalog	A catalog class to use for looking up the metric. TYPE: type[Catalog]

RETURNS	DESCRIPTION
BaseMetric	An instance of the specified metric, if resolution and validation succeed. TYPE: BaseMetric

RAISES	DESCRIPTION
TypeError	If the provided catalog does not implement the Catalog interface.
UnknownMetricError	If the specified metric name is not registered in the catalog.
MetricConfigurationError	If the metric cannot be instantiated from the provided config.

Referenced by:

• API Reference evallm application
  •  api Functions  metric
  •  application Functions  metric

Source code in src/evallm/application/models.py

def resolve_metric(self, catalog: type[Catalog]) -> BaseMetric:
    """Resolve the metric specification to a
    [`BaseMetric`][evallm.metrics.base.BaseMetric] instance.

    Performs contextual validation of the DTO by:

    1. looking up the specified [`name`][..name] in the provided
    [`catalog`][evallm.metrics.registry.Catalog] and
    2. trying to instantiate the metric with the provided [`config`][..config].

    If either of these steps fail, an appropriate error is raised.

    !!! note "Metric instance's `result_key`"
        The resolved metric instance's `result_key` attribute is **not** yet set to
        the value of this DTO's `result_key` field. This is because at this point
        there's no guarantee that the provided `result_key` is unique among all
        resolved metrics for a given evaluation request.

    Args:
        catalog (type[Catalog]): A catalog class to use for looking up the metric.

    Returns:
        BaseMetric: An instance of the specified metric, if resolution and
            validation succeed.

    Raises:
        TypeError: If the provided catalog does not implement the
            [`Catalog`][evallm.metrics.registry.Catalog] interface.
        UnknownMetricError: If the specified metric name is not registered in the
            catalog.
        MetricConfigurationError: If the metric cannot be instantiated from the
            provided config.
    """
    if not issubclass(catalog, Catalog):
        raise TypeError(
            f"Provided catalog {catalog.__name__} doesn't implement the Catalog "
            + "interface"
        )
    if self.name not in (available := catalog.available()):
        raise UnknownMetricError(metric_name=self.name, available=available)
    metric_cls: type[BaseMetric] = catalog.get_metric(self.name)
    try:
        return metric_cls.from_config(**cast(dict[str, Any], self.config or {}))
    except ValueError as exc:
        raise MetricConfigurationError(
            metric_name=self.name,
            config=self.config,
            reason=str(exc),
        ) from exc

Evaluator

Evaluator(request: EvaluationRequest, *, catalog: type[Catalog] = MetricsCatalog)

Component responsible for orchestrating the evaluation process.

Takes an EvaluationRequest as initialization parameter, resolves it, and provides a run method that executes the evaluation, lazily yielding individual EvaluationResults for each record in the specified dataset.

PARAMETER	DESCRIPTION
request	The evaluation request specifying the dataset and metrics to use for evaluation. TYPE: EvaluationRequest

PARAMETER	DESCRIPTION
catalog	An optional catalog class to use for resolving metric specifications. Defaults to MetricsCatalog. (Keyword-only) TYPE: type[Catalog] | None

Usage

from pathlib import Path
from pprint import pprint

from evallm.application import Evaluator
from evallm.application.models import (
    EvaluationRequest,
    JsonlDatasetSpec,
    MetricSpec,
)
from evallm.metrics import builtin  # (1)!

request: EvaluationRequest = EvaluationRequest.create(  # (2)!
    obj={  # (7)!
        "dataset": {  # (3)!
            "input_file": "C:/Users/.../Opus-4.6-Reasoning-3000x-filtered.jsonl",
            "record_id_key": "id",
            "text_key": "thinking",
        },
        "metrics": [  # (4)!
            {
                "name": "CountsMetric",
                "config": {
                    "segments": "tokens",
                },
            },
            {
                "name": "JsonFormatMetric",
                "config": {
                    "expected_schema": {
                        "type": "object",
                        "properties": {
                            "question": {"type": "string"},
                            "setup": {"type": "string"},
                            "joke": {"type": "string"},
                            "funniness_score": {
                                "type": "number",
                                "minimum": 0.0,
                                "maximum": 1.0,
                            },
                        },
                        "required": ["question", "joke"],
                    }
                },
            },
        ],
    }
)

evaluator: Evaluator = Evaluator(request)  # (5)!

for i, res in enumerate(evaluator.run()):  # (6)!
    if i > 3:
        break
    pprint(res.model_dump(), sort_dicts=False)

Output

{
    "record_id": "orca_3452",
    "results": {
        "CountsMetric": 257,
        "JsonFormatMetric": {
            "question": False,
            "setup": False,
            "joke": False,
            "funniness_score": False,
        },
    },
}
{
    "record_id": "drive_minimax_m2.1_questions_220036",
    "results": {
        "CountsMetric": 121,
        "JsonFormatMetric": {
            "question": False,
            "setup": False,
            "joke": False,
            "funniness_score": False,
        },
    },
}
{
    "record_id": "orc_73480",
    "results": {
        "CountsMetric": 243,
        "JsonFormatMetric": {
            "question": False,
            "setup": False,
            "joke": False,
            "funniness_score": False,
        },
    },
}
{
    "record_id": "drive_minimax_m2.1_questions_184095",
    "results": {
        "CountsMetric": 181,
        "JsonFormatMetric": {
            "question": False,
            "setup": False,
            "joke": False,
            "funniness_score": False,
        },
    },
}

1. Important: Remember to import the module where the metrics that you're going to use are defined. Otherwise evaLLM won't be able to discover them!
2. Construct an EvaluationRequest programatically.
3. Define the dataset to evaluate. Here we specify a JSONL file, along with its record ID and text keys.
4. Define a list of metrics to compute. Each MetricSpec needs a name under which the metric is registered and - depending on the metric - can optionally include runtime configuration.
5. Create an Evaluator with the evaluation request.
6. Lazily iterate over the EvaluationResults.
7. You could also post the same request as a JSON payload:

   {
   "dataset": {
       "input_file": "C:/Users/.../Opus-4.6-Reasoning-3000x-filtered.jsonl",
       "record_id_key": "id",
       "text_key": "thinking"
   },
   "metrics": [
       {
       "name": "CountsMetric",
       "config": {
           "segments": "tokens"
       }
       },
       {
       "name": "JsonFormatMetric",
       "config": {
           "expected_schema": {
           "type": "object",
           "properties": {
               "question": { "type": "string" },
               "setup": { "type": "string" },
               "joke": { "type": "string" },
               "funniness_score": {
               "type": "number",
               "minimum": 0.0,
               "maximum": 1.0
               }
           },
           "required": ["question", "joke"]
           }
       }
       }
   ]
   }
   

EvaluationRequest JSON schema

{'$defs': { 'JsonValue': {},
            'JsonlDatasetSpec': {
                'additionalProperties': false,
                'description': 'DTO specifying a structured JSONL dataset to evaluate',
                'properties': {
                    'input_file': {
                        'description': 'Path to a local JSONL-file containing the dataset to evaluate.',
                        'format': 'file-path',
                        'title': 'Input File',
                        'type': 'string'
                    },
                    'record_id_key': {
                        'default': '#IDX#',
                        'description': 'The dataset field/ column name that contains the unique identifier for each record, specified as a non-empty string. Defaults to `"#IDX#"`, which indicates that record IDs should be generated based on the record's (0-indexed) position in the dataset.',
                        'minLength': 1,
                        'title': 'Record Id Key',
                        'type': 'string'
                    },
                    'text_key': {
                        'description': 'The dataset field/ column name that contains the text to evaluate, specified as a non-empty string.',
                        'minLength': 1,
                        'title': 'Text Key',
                        'type': 'string'
                    }
                },
                'required': ['input_file', 'text_key'],
                'title': 'JsonlDatasetSpec',
                'type': 'object'
            },
            'MetricSpec': {
                'additionalProperties': false,
                'description': 'DTO specifying a metric and its configuration',
                'properties': {
                    'name': {
                        'description': 'Non-empty name under which a metric is registered.  !!! note "See also" - [`BaseMetric.name`](https://psandhaas.github.io/evaLLM/reference/evallm/metrics/base/#evallm.metrics.base.BaseMetric.name) - [`@register_metric`](https://psandhaas.github.io/evaLLM/reference/evallm/metrics/registry/#evallm.metrics.registry.register_metric) - [`MetricsCatalog.available`](https://psandhaas.github.io/evaLLM/reference/evallm/metrics/registry/#evallm.metrics.registry.MetricsCatalog.available)',
                        'minLength': 1,
                        'title': 'Name',
                        'type': 'string'
                    },
                    'config': {
                        'anyOf': [
                            {'additionalProperties': {'$ref': '#/$defs/JsonValue'}, 'type': 'object'},
                            {'type': 'null'}
                        ],
                        'default': null,
                        'description': 'Runtime configuration for the metric or `null` (default), if not needed.  !!! note "See also" - [`BaseMetric.config_schema`](https://psandhaas.github.io/evaLLM/reference/evallm/metrics/base/#evallm.metrics.base.BaseMetric.config_schema) - [`BaseMetric.from_config`](https://psandhaas.github.io/evaLLM/reference/evallm/metrics/base/#evallm.metrics.base.BaseMetric.from_config)',
                        'title': 'Config'
                    }
                },
                'required': ['name'],
                'title': 'MetricSpec',
                'type': 'object'}
            },
 'additionalProperties': false,
 'description': 'DTO specifying which configuration of metrics to evaluate a given dataset on',
 'properties': {
    'dataset': {
        '$ref': '#/$defs/JsonlDatasetSpec',
        'description': 'The dataset to evaluate, specified as a [`JsonlDatasetSpec`](https://psandhaas.github.io/evaLLM/reference/evallm/application/models/#evallm.application.models.JsonlDatasetSpec).'
    },
    'metrics': {
        'description': 'List of one or more metrics to use for evaluation, each specified as a [`MetricSpec`](https://psandhaas.github.io/evaLLM/reference/evallm/application/models/#evallm.application.models.MetricSpec).',
        'items': {'$ref': '#/$defs/MetricSpec'},
        'minItems': 1,
        'title': 'Metrics',
        'type': 'array'
    }
 },
 'required': ['dataset', 'metrics'],
 'title': 'EvaluationRequest',
 'type': 'object'}

Referenced by:

• API Reference evallm application  application Classes  Evaluator

Returned by:

• API Reference evallm application
  •  api Classes
    •  BaseEvaluationBuilder Functions  build_evaluator
    •  JsonlEvaluationBuilder Functions  build_evaluator
  •  application Classes
    •  BaseEvaluationBuilder Functions  build_evaluator
    •  JsonlEvaluationBuilder Functions  build_evaluator

METHOD	DESCRIPTION
run	Evaluate the records in the request using the specified metrics.

Source code in src/evallm/application/services.py

def __init__(
    self, request: EvaluationRequest, *, catalog: type[Catalog] = MetricsCatalog
) -> None:
    self._resolved_records, self._composite_metric = RequestResolver(
        catalog=catalog
    )(request)

Functions

run

run() -> Iterator[EvaluationResult]

Evaluate the records in the request using the specified metrics.

This method iterates over the resolved dataset records and applies the composite metric to each record, yielding an EvaluationResult for each one.

YIELDS	DESCRIPTION
EvaluationResult	An evaluation result for each record, containing the record's unique identifier and a dictionary mapping metric result keys to their computed values. TYPE:: EvaluationResult

Referenced by:

• API Reference evallm application  application Classes  Evaluator

Source code in src/evallm/application/services.py

def run(self) -> Iterator[EvaluationResult]:
    """Evaluate the records in the request using the specified metrics.

    This method iterates over the resolved dataset records and applies the
    composite metric to each record, yielding an
    [`EvaluationResult`][evallm.application.models.EvaluationResult] for each one.

    Yields:
        EvaluationResult: An evaluation result for each record, containing the
            record's unique identifier and a dictionary mapping metric result keys
            to their computed values.
    """
    with self._resolved_records as records:
        for record in records:
            yield EvaluationResult(
                record_id=record.record_id,
                results=self._evaluate_record(record),
            )

Functions

count_characters

count_characters(*, result_key: str | None = None) -> MetricSpec

Create a metric spec for counting the number of characters in the evaluated text.

PARAMETER	DESCRIPTION
result_key	Optional explicit result key for the metric. TYPE: str | None DEFAULT: None

RETURNS	DESCRIPTION
MetricSpec	Character-count metric specification. TYPE: MetricSpec

Source code in src/evallm/application/api.py

def count_characters(*, result_key: str | None = None) -> MetricSpec:
    """Create a metric spec for counting the number of characters in the evaluated text.

    Args:
        result_key (str | None, optional): Optional explicit
            [result key][evallm.metrics.base.BaseMetric.result_key] for the metric.

    Returns:
        MetricSpec: Character-count metric specification.
    """
    return metric("CountsMetric", result_key=result_key)

count_encodings

count_encodings(
    model: str, *, hf_token: str | None = None, result_key: str | None = None
) -> MetricSpec

Create a metric spec for counting the number of model-specific encodings in a text.

This metric allows for specifying any

Use mouse to pan and zoom

Use mouse to pan and zoom

HuggingFace model to use for segmenting the text into encodings.

Counting encodings

The 'hf-tokenizers' extra of evaLLM must be installed to use this metric, as it relies on the HuggingFace tokenizers library for encoding text.

Install it with:

pip install "evallm-qa[hf-tokenizers]"

PARAMETER	DESCRIPTION
model	The name of a HuggingFace model that provides a pretrained tokenizer, which will be used to segment the text into model-specific encodings. TYPE: str

PARAMETER	DESCRIPTION
hf_token	User access token for HuggingFace. This is only required when specifying a gated model to use for encoding. Note that you can also set the HF_TOKEN environment variable to avoid passing the token directly in the configuration. TYPE: str | None
result_key	Optional explicit result key for the metric. TYPE: str | None

RETURNS	DESCRIPTION
MetricSpec	Encoding-count metric specification. TYPE: MetricSpec

Source code in src/evallm/application/api.py

def count_encodings(
    model: str,
    *,
    hf_token: str | None = None,
    result_key: str | None = None,
) -> MetricSpec:
    """Create a metric spec for counting the number of model-specific encodings in a text.

    This metric allows for specifying any
    [:hugging: HuggingFace model](https://huggingface.co/models?library=transformers)
    to use for segmenting the text into encodings.

    !!! note "Counting encodings"
        The `'hf-tokenizers'` extra of evaLLM must be installed to use this metric, as
        it relies on the HuggingFace tokenizers library for encoding text.

        Install it with:
        ```bash
        pip install "evallm-qa[hf-tokenizers]"
        ```

    Args:
        model (str): The name of a
            [HuggingFace model](https://huggingface.co/models?library=transformers)
            that provides a pretrained tokenizer, which will be used to segment the
            text into model-specific encodings.

    Keyword Args:
        hf_token (str | None): [User access token for HuggingFace](https://huggingface.co/docs/hub/security-tokens).
            This is only required when specifying a
            [gated model](https://huggingface.co/docs/hub/models-gated) to use for
            encoding. Note that you can also set the `HF_TOKEN` environment variable to
            avoid passing the token directly in the configuration.
        result_key (str | None, optional): Optional explicit
            [result key][evallm.metrics.base.BaseMetric.result_key] for the metric.

    Returns:
        MetricSpec: Encoding-count metric specification.
    """
    config: dict[str, JsonValue] = {
        "segments": "encodings",
        "segmentation": model,
    }
    if hf_token is not None:
        config["hf_token"] = hf_token
    return metric("CountsMetric", config=config, result_key=result_key)

count_tokens

count_tokens(
    *,
    segmentation: Literal["whitespace", "wordpunct"] = "whitespace",
    result_key: str | None = None,
) -> MetricSpec

Create a metric spec for counting the number of tokens in the evaluated text.

PARAMETER	DESCRIPTION
segmentation	Tokenization method to use. Defaults to "whitespace". (see also: CountsConfig) TYPE: Literal['whitespace', 'wordpunct'] DEFAULT: 'whitespace'
result_key	Optional explicit result key for the metric. TYPE: str | None DEFAULT: None

RETURNS	DESCRIPTION
MetricSpec	Token-count metric specification. TYPE: MetricSpec

Source code in src/evallm/application/api.py

def count_tokens(
    *,
    segmentation: Literal["whitespace", "wordpunct"] = "whitespace",
    result_key: str | None = None,
) -> MetricSpec:
    """Create a metric spec for counting the number of tokens in the evaluated text.

    Args:
        segmentation (Literal["whitespace", "wordpunct"], optional): Tokenization
            method to use. Defaults to "whitespace". (see also:
            [`CountsConfig`][evallm.metrics.builtin.CountsConfig])
        result_key (str | None, optional): Optional explicit
            [result key][evallm.metrics.base.BaseMetric.result_key] for the metric.

    Returns:
        MetricSpec: Token-count metric specification.
    """
    return metric(
        "CountsMetric",
        config={
            "segments": "tokens",
            "segmentation": segmentation,
        },
        result_key=result_key,
    )

json_format

json_format(
    expected_schema: Mapping[str, Any] | str,
    *,
    check_formats: bool = False,
    result_key: str | None = None,
) -> MetricSpec

Create a metric spec for validating JSON output format.

RETURNS	DESCRIPTION
MetricSpec	JSON-format validation metric specification. TYPE: MetricSpec

Source code in src/evallm/application/api.py

def json_format(
    expected_schema: Mapping[str, Any] | str,
    *,
    check_formats: bool = False,
    result_key: str | None = None,
) -> MetricSpec:
    """Create a metric spec for validating JSON output format.

    Returns:
        MetricSpec: JSON-format validation metric specification.
    """
    schema_value: JsonValue = (
        cast(JsonValue, dict(expected_schema))
        if isinstance(expected_schema, Mapping)
        else cast(JsonValue, expected_schema)
    )
    return metric(
        "JsonFormatMetric",
        config={
            "expected_schema": schema_value,
            "check_formats": check_formats,
        },
        result_key=result_key,
    )

metric

metric(
    name: str,
    *,
    config: Mapping[str, JsonValue] | None = None,
    result_key: str | None = None,
) -> MetricSpec

Create a generic metric spec using a metric's name.

Warning

This function only performs syntactic validation so as to create canonical MetricSpec objects. It does not guarantee the existence of the metric's name in the catalog, nor does it validate the provided config against the metric's config schema.

See also

• MetricSpec.resolve_metric(catalog) for runtime resolution and validation of metric specs against a catalog.

PARAMETER	DESCRIPTION
name	Name of the metric as registered in the catalog. TYPE: str
config	Optional config dict for the metric. (see also: BaseMetric.config_schema) TYPE: Mapping[str, JsonValue] | None DEFAULT: None
result_key	Optional explicit result key for the metric. TYPE: str | None DEFAULT: None

RETURNS	DESCRIPTION
MetricSpec	Canonical metric specification. TYPE: MetricSpec

Source code in src/evallm/application/api.py

def metric(
    name: str,
    *,
    config: Mapping[str, JsonValue] | None = None,
    result_key: str | None = None,
) -> MetricSpec:
    """Create a generic metric spec using a metric's name.

    !!! warning
        This function **only performs syntactic validation** so as to create canonical
        `MetricSpec` objects. It **does not guarantee the existence** of the metric's
        name in the catalog, **nor does it validate the provided config** against the
        metric's [config schema][evallm.metrics.base.BaseMetric.config_schema].

    !!! info "See also"
        - [`MetricSpec.resolve_metric(catalog)`][evallm.application.models.MetricSpec.resolve_metric]
            for runtime resolution and validation of metric specs against a catalog.

    Args:
        name (str): Name of the metric as registered in the catalog.
        config (Mapping[str, JsonValue] | None, optional): Optional config dict for the
            metric. (see also:
            [`BaseMetric.config_schema`][evallm.metrics.base.BaseMetric.config_schema])
        result_key (str | None, optional): Optional explicit
            [result key][evallm.metrics.base.BaseMetric.result_key] for the metric.

    Returns:
        MetricSpec: Canonical metric specification.
    """
    copied_config: dict[str, JsonValue] | None = (
        dict(config) if config is not None else None
    )
    return MetricSpec(name=name, config=copied_config, result_key=result_key)