Skip to content

base

Abstract generic base class BaseMetric[T] and its DefaultSchema configuration

status
Use mouse to pan and zoom
Use mouse to pan and zoom

CLASS DESCRIPTION
DefaultSchema

Default empty schema as a pydantic.BaseModel.

BaseMetric

Abstract base class for all metrics, parameterized by a serializable value type.

Type Aliases

Classes

DefaultSchema pydantic-model

Default empty schema as a pydantic.BaseModel.

Used as the default value for the config_schema class variable in BaseMetric to match the shared interface and indicate that a metric doesn't take any configuration parameters. Schema enforces that no configuration parameters can be passed at instantiation by forbidding extra fields and having no defined fields. ( see also: BaseMetric.from_config(**kwargs))

Config:

  • extra: forbid
  • frozen: True

BaseMetric

BaseMetric(**kwargs: dict[str, Any])

              flowchart TD
              evallm.metrics.base.BaseMetric[BaseMetric]

              

              click evallm.metrics.base.BaseMetric href "" "evallm.metrics.base.BaseMetric"
            
Use mouse to pan and zoom

Abstract base class for all metrics, parameterized by a serializable value type.

This generic class defines the shared interface implemented by all metrics. It includes convenience methods and properties for tree traversal, as well as an abstract evaluate method that must be implemented by all concrete metric classes.

Metric interface

  • The evaluate method is expected to return a dictionary of the form dict[str, T], where the key is this metric instance's result_key and the value is of SerializableValueType.
  • A metric's name identifies its class in a Catalog and serves as the default result_key unless a custom result key is set on the instance. The result_key on the other hand, is an instance-level property that determines the key under which this metric's value is stored in the evaluation results.
  • The (invariant) generic type parameter T declares the expected serializable value shape for a metric. At class-definition time, BaseMetric validates that T is compatible with SerializableValueType, and evaluate is validated against dict[str, T].
  • At runtime, calling result(value) enforces that value matches T and returns {self.result_key: value}. Implementations that bypass result(value) are expected, by convention, to preserve the same contract.
PARAMETER DESCRIPTION
**kwargs

The runtime configuration parameters for this metric instance, as a dictionary. The expected parameters depend on the metric class' defined config_schema and the provided parameters must strictly conform to it.

TYPE: dict[str, Any]

RAISES DESCRIPTION
ValidationError

If trying to instantiate a metric class with invalid runtime parameters (i.e. missing required parameters, extra unknown parameters, or parameters of the wrong type).

ValueError

If trying to pass runtime parameters to a metric class that doesn't define a config_schema.

METHOD DESCRIPTION
__init_subclass__

Called when a class inherits from BaseMetric.

__eq__

Check if this metric is equal to another metric.

__iter__

Iterate over the subtree rooted at this metric in breadth-first order.

__contains__

Check if a metric is contained in the subtree rooted at this metric.

__repr__

String representation of the subtree, rooted in this metric instance.

result

Helper method for wrapping the value of a metric's evaluation result in a

evaluate

Evaluate the metric on the given text.

descendants

Yield nodes in the subtree rooted at self (cycle-safe).

from_config

Factory method to create a metric instance from kwargs or JSON payload.

ATTRIBUTE DESCRIPTION
value_type

The metric class' parametrized type, binding the generic type parameter T.

TYPE: object

config_schema

A schema defining the expected runtime configuration parameters for this metric.

TYPE: type[BaseModel]

parent

Get the parent of this metric.

TYPE: CompositeMetric | None

result_key

The key to use for this metric's value in the evaluation result dictionary.

TYPE: str

name

The name of this metric's class.

TYPE: str

is_composite

Check if this metric is a composite metric (i.e. can have child metrics).

TYPE: bool

is_root

Check if this metric is a root metric (i.e. has no parent).

TYPE: bool

root

Get the root metric of the tree this metric belongs to.

TYPE: BaseMetric[SerializableValueType]

depth

Get the depth of this metric in the tree (i.e. the number of edges from this

TYPE: int

children

Get an iterator over the children of this metric.

TYPE: Iterator[BaseMetric[SerializableValueType]]

config

Get the runtime configuration of this metric instance as a dictionary.

TYPE: dict[str, Any]

Source code in src/evallm/metrics/base.py
def __init__(self, **kwargs: dict[str, Any]):
    """ """
    self.parent: BaseMetric[SerializableValueType] | None = None
    self._config: dict[str, Any] = {}  # default

    # raise error when trying to pass runtime params without a config schema
    if (schema := type(self).config_schema).model_json_schema().get(
        "properties", {}
    ) == {}:
        if kwargs:
            raise ValueError(f"{type(self)._name} takes no runtime parameters.")
    else:  # validate runtime params against config_schema
        try:
            model: BaseModel = schema.model_validate(kwargs)
            self._config = model.model_dump()
        except ValidationError:
            raise

Attributes

config_schema class-attribute
config_schema: type[BaseModel]

A schema defining the expected runtime configuration parameters for this metric.

If the metric class does not explicitly define a config schema, this defaults to an empty pydantic model (see DefaultSchema) that takes no parameters.

Info

A class' config schema is always configured such that:

  • all declared fields must be provided unless they are declared as optional
  • no additional fields beyond those declared in the schema are allowed (see also pydantic.ConfigDict.extra)
  • the schema is frozen/immutable once defined (see also pydantic.ConfigDict.frozen)
  • configuration parameters provided at runtime are strictly validated against the schema, raising a ValidationError if the provided parameters violate it (see also pydantic Conversion Table)
parent property writable
parent: CompositeMetric | None

Get the parent of this metric.

RETURNS DESCRIPTION
CompositeMetric | None

CompositeMetric | None: The parent metric, or None if this metric has no parent, i.e. is a leaf node.

result_key property writable
result_key: str

The key to use for this metric's value in the evaluation result dictionary.

By default, this is the same as the metric's name, but it can be overridden on a per-instance basis. This is useful when multiple instances of the same metric class are used in a single evaluation request with different runtime parameters and therefore need distinct keys in the output.

RETURNS DESCRIPTION
result_key

The key of this metric instance's value in the evaluation result dictionary. Defaults to the metric's name.

TYPE: str

name property
name: str

The name of this metric's class.

This is a class-, not an instance-level property that's used for identifying this metric in the catalog. It also serves as the default result_key for metric instances unless explicitly overridden.

Metric class registration and name updates

When a metric class is registered in the catalog under a custom name using @register_metric(name=...), the name-property of any pre-existing instances of that class will be updated to reflect the new name in the catalog.

RETURNS DESCRIPTION
name

The name of this metric. Defaults to the class name if not overwritten by the @register_metric decorator at definition time.

TYPE: str

is_composite property
is_composite: bool

Check if this metric is a composite metric (i.e. can have child metrics).

RETURNS DESCRIPTION
bool

True if this metric is a composite metric, False otherwise.

TYPE: bool

is_root property
is_root: bool

Check if this metric is a root metric (i.e. has no parent).

RETURNS DESCRIPTION
bool

True if this metric has no parent, False otherwise.

TYPE: bool

root property

Get the root metric of the tree this metric belongs to.

RETURNS DESCRIPTION
BaseMetric

The root metric of the tree this metric belongs to.

TYPE: BaseMetric[SerializableValueType]

depth property
depth: int

Get the depth of this metric in the tree (i.e. the number of edges from this metric to the root).

RETURNS DESCRIPTION
depth

The depth of this metric in the tree. Root metrics have depth 0, their children have depth 1, and so on.

TYPE: int

children property

Get an iterator over the children of this metric.

YIELDS DESCRIPTION
children

An iterator over the child metrics (i.e. direct descendants) of this metric. If this metric is a leaf node, this will be an empty iterator.

TYPE:: BaseMetric

config property
config: dict[str, Any]

Get the runtime configuration of this metric instance as a dictionary.

RETURNS DESCRIPTION
config

A dictionary containing the runtime configuration of this metric instance, validated against the metric's schema.

TYPE: dict[str, Any]

Functions

__init_subclass__
__init_subclass__(**kwargs)

Called when a class inherits from BaseMetric.

Sets the default name and config schema of the metric class and validates & stores the parametrized value type of the metric class (i.e. the T in BaseMetric[T]) as the value_type class variable and ensures that evaluate is implemented with the correct signature.

RAISES DESCRIPTION
TypeError

If the metric class does not implement the evaluate method with the correct signature.

UnserializableValueTypeError

If the metric class's parametrized value type is not an instance of SerializableValueType.

Source code in src/evallm/metrics/base.py
def __init_subclass__(cls, **kwargs):
    """Called when a class inherits from `BaseMetric`.

    Sets the default name and config schema of the metric class and validates
    & stores the parametrized value type of the metric class (i.e.
    the `T` in `BaseMetric[T]`) as the
    [`value_type`][evallm.metrics.base.BaseMetric.value_type] class variable
    and ensures that `evaluate` is implemented with the correct signature.

    Raises:
        TypeError: If the metric class does not implement the `evaluate` method
            with the correct signature.
        UnserializableValueTypeError: If the metric class's parametrized value type
            is not an instance of
            [`SerializableValueType`][evallm.metrics.types.SerializableValueType].
    """
    super().__init_subclass__(**kwargs)

    # Set default class name if not already set
    if getattr(cls, "_name", None) is None:
        cls._name = cls.__name__

    # Set default config schema if not already set
    if getattr(cls, "config_schema", None) is None:
        cls.config_schema = DefaultSchema

    # Set the metric's parametrized value type as a class variable
    for base in get_original_bases(cls):
        if get_origin(base) is BaseMetric:
            value_type = get_args(base)[0]
            if not is_serializable_type(value_type):
                raise UnserializableValueTypeError(
                    f"Metric class '{cls.__name__}' has invalid value type annotation "
                    + f"{value_type!r}. Expected a SerializableValueType."
                )
            cls.value_type = value_type
            break

    try:  # Ensure evaluate method is implemented & has correct signature
        _ = check_evaluate_signature(cls)
    except TypeError:
        raise
__eq__
__eq__(other: object) -> bool

Check if this metric is equal to another metric.

Two metrics are considered equal iff they are the same instance.

PARAMETER DESCRIPTION
other

The other metric to compare against.

TYPE: object

RETURNS DESCRIPTION
bool

True if this metric is the same instance as the other metric, False otherwise.

TYPE: bool

Source code in src/evallm/metrics/base.py
def __eq__(self, other: object) -> bool:
    """Check if this metric is equal to another metric.

    Two metrics are considered equal iff they are the same instance.

    Args:
        other (object): The other metric to compare against.

    Returns:
        bool: `True` if this metric is the same instance as the other metric,
            `False` otherwise.
    """
    return self is other
__iter__

Iterate over the subtree rooted at this metric in breadth-first order.

YIELDS DESCRIPTION
metric

An iterator over the subtree rooted at this metric in breadth-first order, excluding this metric itself.

TYPE:: BaseMetric

Source code in src/evallm/metrics/base.py
def __iter__(self) -> Iterator[BaseMetric[SerializableValueType]]:
    """Iterate over the subtree rooted at this metric in breadth-first order.

    Yields:
        metric (BaseMetric): An iterator over the subtree rooted at this metric in
            breadth-first order, excluding this metric itself.
    """
    yield from self.descendants(include_self=False, order="bfs")
__contains__
__contains__(metric: str | BaseMetric[SerializableValueType]) -> bool

Check if a metric is contained in the subtree rooted at this metric.

Note

Passing a metric instance checks for identity (i.e. whether exactly that instance is contained in this subtree), whereas passing a metric's name checks for membership (i.e. whether any metric with that name is contained in this subtree).

Example

leaf1: MetricA = MetricA()  # name='MetricA'
leaf2: MetricA = MetricA()  # name='MetricA'
composite: CompositeMetric = CompositeMetric()
composite.add(leaf1)
Identity-based membership check:
>>> leaf1 in composite
True
>>> leaf2 in composite
False
Name-based membership check:
>>> "MetricA" in composite
True
>>> "NonExistentMetric" in composite
False

PARAMETER DESCRIPTION
metric

The name of the metric to check for, or the metric instance itself.

TYPE: str | BaseMetric

RETURNS DESCRIPTION
bool

True if the given metric is a descendant of this metric (excluding this metric itself), False otherwise.

TYPE: bool

Source code in src/evallm/metrics/base.py
def __contains__(self, metric: str | BaseMetric[SerializableValueType]) -> bool:
    """Check if a metric is contained in the subtree rooted at this metric.

    Note:
        Passing a metric **instance** checks for **identity** (i.e. whether exactly
        that instance is contained in this subtree), whereas passing a metric's
        **name** checks for **membership** (i.e. whether any metric with that name
        is contained in this subtree).

    Example:
        ```python
        leaf1: MetricA = MetricA()  # name='MetricA'
        leaf2: MetricA = MetricA()  # name='MetricA'
        composite: CompositeMetric = CompositeMetric()
        composite.add(leaf1)
        ```
        Identity-based membership check:
        ```pycon
        >>> leaf1 in composite
        True
        >>> leaf2 in composite
        False
        ```
        Name-based membership check:
        ```pycon
        >>> "MetricA" in composite
        True
        >>> "NonExistentMetric" in composite
        False
        ```

    Args:
        metric (str | BaseMetric): The name of the metric to check for, or the
            metric instance itself.

    Returns:
        bool: `True` if the given metric is a descendant of this metric
            (excluding this metric itself), `False` otherwise.
    """
    if isinstance(metric, str):
        return any(node.name == metric for node in self.descendants())
    return any(node is metric for node in self.descendants())
__repr__
__repr__() -> str

String representation of the subtree, rooted in this metric instance.

RETURNS DESCRIPTION
representation

A string representation of the metric instance, including its name and class.

TYPE: str

Source code in src/evallm/metrics/base.py
def __repr__(self) -> str:
    """String representation of the subtree, rooted in this metric instance.

    Returns:
        representation (str): A string representation of the metric instance,
            including its name and class.
    """
    return self._repr_tree()
result
result(value: T) -> dict[str, T]

Helper method for wrapping the value of a metric's evaluation result in a dictionary to conform to the expected return type of evaluate.

Example
class CustomMetric(BaseMetric[list[int]]):
    def evaluate(self, text: str) -> dict[str, list[int]]:
        # Perform some evaluation logic and compute the metric's value
        res: list[int] = [len(text.split())]

        # Wrap the result in a dictionary keyed by the metric's result_key
        return self.result(res)
PARAMETER DESCRIPTION
value

The result of computing the metric, where T is the metric's parametrized value type.

TYPE: T

RETURNS DESCRIPTION
result

A dictionary containing the given value, keyed by the metric's result_key.

TYPE: dict[str, T]

RAISES DESCRIPTION
TypeError

If the given value is not of the expected type T or is not JSON-serializable.

Source code in src/evallm/metrics/base.py
def result(self, value: T) -> dict[str, T]:  # pragma: no cover
    """Helper method for wrapping the value of a metric's evaluation result in a
    dictionary to conform to the expected return type of `evaluate`.

    Example:
        ```python
        class CustomMetric(BaseMetric[list[int]]):
            def evaluate(self, text: str) -> dict[str, list[int]]:
                # Perform some evaluation logic and compute the metric's value
                res: list[int] = [len(text.split())]

                # Wrap the result in a dictionary keyed by the metric's result_key
                return self.result(res)
        ```

    Args:
        value (T): The result of computing the metric, where `T` is the metric's
            parametrized value type.

    Returns:
        result (dict[str, T]): A dictionary containing the given value, keyed by
            the metric's [`result_key`][..result_key].

    Raises:
        TypeError: If the given value is not of the expected type `T` or is not
            JSON-serializable.
    """
    if not matches_annotation(value, self.value_type):
        raise TypeError(
            f"Value {value!r} of type {type(value)!r} is not of expected type "
            + f"{self.value_type!r} for metric '{self.name}'."
        )
    return {self.result_key: value}
evaluate abstractmethod
evaluate(text: str) -> dict[str, T]

Evaluate the metric on the given text.

PARAMETER DESCRIPTION
text

The text to evaluate the metric on.

TYPE: str

RETURNS DESCRIPTION
value

The evaluation result keyed by the metric's result_key, where T is the metric's parametrized value type.

TYPE: dict[str, T]

Source code in src/evallm/metrics/base.py
@abstractmethod
def evaluate(self, text: str) -> dict[str, T]:
    """Evaluate the metric on the given text.

    Args:
        text (str): The text to evaluate the metric on.

    Returns:
        value (dict[str, T]): The evaluation result keyed by the metric's
            [`result_key`][..result_key], where `T` is the metric's parametrized
            value type.
    """
    raise NotImplementedError(  # pragma: no cover
        "Subclasses must implement evaluate() method."
    )
descendants
descendants(
    *, include_self: bool = False, order: Literal["dfs", "bfs"] = "dfs"
) -> Iterator[BaseMetric[SerializableValueType]]

Yield nodes in the subtree rooted at self (cycle-safe).

PARAMETER DESCRIPTION
include_self

Whether to include self in the yielded nodes.

TYPE: bool DEFAULT: False

order

The order to traverse the tree. "dfs" for depth-first search, "bfs" for breadth-first search.

TYPE: 'dfs' | 'bfs' DEFAULT: 'dfs'

YIELDS DESCRIPTION
descendants

An iterator over the descendant nodes of this metric, in the specified traversal order.

TYPE:: Iterator[BaseMetric]

Source code in src/evallm/metrics/base.py
def descendants(
    self,
    *,
    include_self: bool = False,
    order: Literal["dfs", "bfs"] = "dfs",
) -> Iterator[BaseMetric[SerializableValueType]]:
    """Yield nodes in the subtree rooted at self (cycle-safe).

    Args:
        include_self (bool): Whether to include `self` in the yielded nodes.
        order ("dfs" | "bfs"): The order to traverse the tree. "dfs" for depth-first
            search, "bfs" for breadth-first search.

    Yields:
        descendants (Iterator[BaseMetric]): An iterator over the descendant nodes of this metric,
            in the specified traversal order.
    """
    visited: set[int] = set()

    # Populate the frontier with the initial node(s) based on traversal order
    frontier: deque[BaseMetric[SerializableValueType]] = deque()
    if include_self:
        frontier.append(cast(BaseMetric[SerializableValueType], self))
    else:  # keep order of children consistent across DFS/BFS
        if order == "dfs":
            frontier.extend(reversed(list(self.children)))
        else:
            frontier.extend(list(self.children))

    while frontier:
        # Get the next node to visit, depending on traversal order
        node: BaseMetric[SerializableValueType] = (
            frontier.pop() if order == "dfs" else frontier.popleft()
        )

        # Skip visited nodes to prevent infinite loops
        if (node_id := id(node)) in visited:
            continue
        visited.add(node_id)

        yield node

        # Expand neighbors
        if order == "dfs":
            frontier.extend(  # reverse for correct DFS order
                reversed(list(node.children))
            )
        else:
            frontier.extend(list(node.children))
from_config classmethod
from_config(*, json_payload: str | None = None, **kwargs: Any) -> BaseMetric[T]

Factory method to create a metric instance from kwargs or JSON payload.

Info

This method can be used to create any metric instance, so long as the provided config parameters conform to the metric's schema. If the metric class doesn't define a config schema (the default), calling this method without kwargs is equivalent to calling the metric class constructor directly. The expected parameters depend on the metric's declared config_schema (see also @config_schema decorator) and the provided parameters must strictly conform to it.

PARAMETER DESCRIPTION
json_payload

A JSON string containing the runtime parameters for this metric instance.

TYPE: str | None DEFAULT: None

PARAMETER DESCRIPTION
**kwargs

The runtime parameters to use for this metric instance.

TYPE: Any

RETURNS DESCRIPTION
metric

An instance of this metric class created using the provided configuration parameters.

TYPE: BaseMetric[T]

RAISES DESCRIPTION
TypeError

If json_payload is not a string or None.

ValueError

If

  • trying to pass runtime parameters to a metric class that doesn't define a config_schema, or
  • trying to instantiate a metric class with invalid runtime parameters (i.e. missing required parameters, extra unknown parameters, or parameters of the wrong type), or
  • passing an invalid JSON payload, or
  • passing both json_payload and kwargs at the same time.
Source code in src/evallm/metrics/base.py
@classmethod
def from_config(
    cls, *, json_payload: str | None = None, **kwargs: Any
) -> BaseMetric[T]:
    """Factory method to create a metric instance from kwargs or JSON payload.

    !!! info
        This method can be used to create any metric instance, so long as the
        provided config parameters conform to the metric's schema. If the metric
        class doesn't define a config schema (the default), calling this method
        without kwargs is equivalent to calling the metric class constructor
        directly.
        The expected parameters depend on the metric's declared config_schema
        (see also [`@config_schema` decorator][evallm.metrics.config.config_schema])
        and the provided parameters must strictly conform to it.

    Args:
        json_payload (str | None): A JSON string containing the runtime parameters
            for this metric instance.

    Keyword Args:
        **kwargs (Any): The runtime parameters to use for this metric instance.

    Returns:
        metric (BaseMetric[T]): An instance of this metric class created using the
            provided configuration parameters.

    Raises:
        TypeError: If `json_payload` is not a string or `None`.
        ValueError: If

            - trying to pass runtime parameters to a metric class that doesn't
                define a `config_schema`, or
            - trying to instantiate a metric class with invalid runtime parameters
                (i.e. missing required parameters, extra unknown parameters, or
                parameters of the wrong type), or
            - passing an invalid JSON payload, or
            - passing both `json_payload` and `kwargs` at the same time.
    """
    runtime_params: dict[str, Any]
    if json_payload is not None:
        if kwargs:
            raise ValueError("Cannot mix json_payload with kwargs.")
        if not isinstance(json_payload, str):
            raise TypeError("json_payload must be a string or None.")
        try:
            runtime_params = orjson.loads(json_payload)
        except orjson.JSONDecodeError:
            raise ValueError(
                "Invalid json_payload: not a valid JSON string."
            ) from None
    else:
        runtime_params = kwargs

    return cls(**runtime_params)

Functions