composite

CompositeMetric — tree container for other metrics

Use mouse to pan and zoom

CLASS	DESCRIPTION
`CompositeMetric`	A container for other metrics following the `BaseMetric` interface.

Type Aliases

Classes

CompositeMetric

CompositeMetric()


              flowchart TD
              evallm.metrics.composite.CompositeMetric[CompositeMetric]
              evallm.metrics.base.BaseMetric[BaseMetric]

                              evallm.metrics.base.BaseMetric --> evallm.metrics.composite.CompositeMetric
                


              click evallm.metrics.composite.CompositeMetric href "" "evallm.metrics.composite.CompositeMetric"
              click evallm.metrics.base.BaseMetric href "" "evallm.metrics.base.BaseMetric"

Use mouse to pan and zoom

A container for other metrics following the BaseMetric interface.

Note

Implements the Composite design pattern by allowing a metric to contain other BaseMetric & CompositeMetric instances so that calling evaluate on a composite metric will recursively call evaluate on all of its children.

Returned by:

API Reference evallm metrics

METHOD	DESCRIPTION
`add`	Add a child metric to this composite metric instance.
`remove`	Remove a contained metric from this composite.
`evaluate`	Evaluate all child metrics on the given text.
`__init_subclass__`	Called when a class inherits from `BaseMetric`.
`__eq__`	Check if this metric is equal to another metric.
`__iter__`	Iterate over the subtree rooted at this metric in breadth-first order.
`__contains__`	Check if a metric is contained in the subtree rooted at this metric.
`__repr__`	String representation of the subtree, rooted in this metric instance.
`result`	Helper method for wrapping the value of a metric's evaluation result in a
`descendants`	Yield nodes in the subtree rooted at self (cycle-safe).
`from_config`	Factory method to create a metric instance from kwargs or JSON payload.

ATTRIBUTE	DESCRIPTION
`value_type`	The metric class' parametrized type, binding the generic type parameter `T`. TYPE: `object`
`config_schema`	A schema defining the expected runtime configuration parameters for this metric. TYPE: `type[BaseModel]`
`parent`	Get the parent of this metric. TYPE: `CompositeMetric \| None`
`result_key`	The key to use for this metric's value in the evaluation result dictionary. TYPE: `str`
`name`	The name of this metric's class. TYPE: `str`
`is_composite`	Check if this metric is a composite metric (i.e. can have child metrics). TYPE: `bool`
`is_root`	Check if this metric is a root metric (i.e. has no parent). TYPE: `bool`
`root`	Get the root metric of the tree this metric belongs to. TYPE: `BaseMetric[SerializableValueType]`
`depth`	Get the depth of this metric in the tree (i.e. the number of edges from this TYPE: `int`
`children`	Get an iterator over the children of this metric. TYPE: `Iterator[BaseMetric[SerializableValueType]]`
`config`	Get the runtime configuration of this metric instance as a dictionary. TYPE: `dict[str, Any]`

Source code in src/evallm/metrics/composite.py

def __init__(self) -> None:
    super().__init__()
    self._children: list[BaseMetric[SerializableValueType]] = []

Attributes

value_type `class-attribute`

value_type: object

The metric class' parametrized type, binding the generic type parameter T.

Referenced by:

API Reference evallm metrics

config_schema `class-attribute`

config_schema: type[BaseModel]

A schema defining the expected runtime configuration parameters for this metric.

If the metric class does not explicitly define a config schema, this defaults to an empty pydantic model (see DefaultSchema) that takes no parameters.

Info

A class' config schema is always configured such that:

all declared fields must be provided unless they are declared as optional
no additional fields beyond those declared in the schema are allowed (see also pydantic.ConfigDict.extra)
the schema is frozen/immutable once defined (see also pydantic.ConfigDict.frozen)
configuration parameters provided at runtime are strictly validated against the schema, raising a ValidationError if the provided parameters violate it (see also pydantic Conversion Table)

Referenced by:

API Reference evallm
- application
- metrics
  - base Classes BaseMetric
  - config Functions config_schema

parent `property` `writable`

parent: CompositeMetric | None

Get the parent of this metric.

RETURNS	DESCRIPTION
`CompositeMetric \| None`	CompositeMetric \| None: The parent metric, or `None` if this metric has no parent, i.e. is a leaf node.

result_key `property` `writable`

result_key: str

The key to use for this metric's value in the evaluation result dictionary.

By default, this is the same as the metric's name, but it can be overridden on a per-instance basis. This is useful when multiple instances of the same metric class are used in a single evaluation request with different runtime parameters and therefore need distinct keys in the output.

RETURNS	DESCRIPTION
`result_key`	The key of this metric instance's value in the evaluation result dictionary. Defaults to the metric's `name`. TYPE: `str`

Referenced by:

API Reference evallm
- metrics
- application

name `property`

name: str

The name of this metric's class.

This is a class-, not an instance-level property that's used for identifying this metric in the catalog. It also serves as the default result_key for metric instances unless explicitly overridden.

Metric class registration and name updates

When a metric class is registered in the catalog under a custom name using @register_metric(name=...), the name-property of any pre-existing instances of that class will be updated to reflect the new name in the catalog.

RETURNS	DESCRIPTION
`name`	The name of this metric. Defaults to the class name if not overwritten by the `@register_metric` decorator at definition time. TYPE: `str`

Referenced by:

API Reference evallm
- metrics
- application
  - application Classes
    - EvaluationResult Attributes results
    - MetricSpec Attributes name
  - models Classes
    - EvaluationResult Attributes results
    - MetricSpec Attributes name

is_composite `property`

is_composite: bool

Check if this metric is a composite metric (i.e. can have child metrics).

RETURNS	DESCRIPTION
`bool`	`True` if this metric is a composite metric, `False` otherwise. TYPE: `bool`

is_root `property`

is_root: bool

Check if this metric is a root metric (i.e. has no parent).

RETURNS	DESCRIPTION
`bool`	`True` if this metric has no parent, `False` otherwise. TYPE: `bool`

root `property`

root: BaseMetric[SerializableValueType]

Get the root metric of the tree this metric belongs to.

RETURNS	DESCRIPTION
`BaseMetric`	The root metric of the tree this metric belongs to. TYPE: `BaseMetric[SerializableValueType]`

depth `property`

depth: int

Get the depth of this metric in the tree (i.e. the number of edges from this metric to the root).

RETURNS	DESCRIPTION
`depth`	The depth of this metric in the tree. Root metrics have depth 0, their children have depth 1, and so on. TYPE: `int`

children `property`

children: Iterator[BaseMetric[SerializableValueType]]

Get an iterator over the children of this metric.

YIELDS	DESCRIPTION
`children`	An iterator over the child metrics (i.e. direct descendants) of this metric. If this metric is a leaf node, this will be an empty iterator. TYPE:: `BaseMetric`

config `property`

config: dict[str, Any]

Get the runtime configuration of this metric instance as a dictionary.

RETURNS	DESCRIPTION
`config`	A dictionary containing the runtime configuration of this metric instance, validated against the metric's schema. TYPE: `dict[str, Any]`

Referenced by:

API Reference evallm metrics config Functions config_schema

Functions

add

add(metric: BaseMetric[SerializableValueType], *, move: bool = False)

Add a child metric to this composite metric instance.

Note

A composite metric can contain any number of child metrics, which can be simple metrics or other composite. The only restriction is that the same instance of a metric can only be added to exactly one composite metric at any given time so as to ensure a well-formed tree structure.

Note however that different instances of the same class of metric can be added arbitrarily often to the same or to different composite metrics.

PARAMETER	DESCRIPTION
`metric`	The child metric to add (can be simple or composite). The parent of the child will be set to this composite metric. If this composite is already the metric's parent, this is a no-op. TYPE: `BaseMetric[SerializableValueType]`
`move`	Whether to remove the `metric` from its current parent (if any) before adding it to this composite. If `False`, trying to add a metric that already has a parent will raise a `ValueError`. TYPE: `bool` DEFAULT: `False`

RAISES	DESCRIPTION
`ValueError`	If trying to add self as a child of itself, or add a metric instance that is already contained in this composite metric, or add a metric instance that is already contained in another composite metric and `move=False`.

Source code in src/evallm/metrics/composite.py

def add(self, metric: BaseMetric[SerializableValueType], *, move: bool = False):
    """Add a child metric to this composite metric instance.

    Note:
        A composite metric can contain any number of child metrics, which can be
        simple metrics or other composite. The only restriction is that the *same*
        *instance* of a metric can only be added to exactly one composite metric at
        any given time so as to ensure a well-formed tree structure.

        Note however that *different instances* of the *same class* of metric can be
        added arbitrarily often to the same or to different composite metrics.

    Args:
        metric (BaseMetric[SerializableValueType]): The child metric to add (can
            be simple or composite). The parent of the child will be set to this
            composite metric. If this composite is already the metric's parent,
            this is a no-op.
        move (bool): Whether to remove the `metric` from its current parent (if
            any) before adding it to this composite. If `False`, trying to add a
            metric that already has a parent will raise a `ValueError`.

    Raises:
        ValueError: If trying to

            - add self as a child of itself, or
            - add a metric instance that is already contained in this
                composite metric, or
            - add a metric instance that is already contained in another composite
                metric and `move=False`.
    """
    if metric is self:
        raise ValueError("Cannot add self as a child of itself.")
    if metric in self:
        if metric.parent is self:
            return
        raise ValueError(
            f"{metric!r} is already a descendant of this composite metric."
        )
    if metric.parent is not None:
        if not move:
            raise ValueError(
                f"{metric!r} is already a descendant of {metric.parent!r}\n"
                + "Set move=True to automatically remove it from its current "
                + "parent and add it to this composite metric."
            )
        # remove from current parent before adding to this composite
        metric.parent.remove(metric)

    self._children.append(metric)
    metric.parent = self

remove

remove(metric: BaseMetric[SerializableValueType]) -> BaseMetric[SerializableValueType]

Remove a contained metric from this composite.

Important

When removing a composite metric, all of its descendants are also removed.

PARAMETER	DESCRIPTION
`metric`	The child metric to remove. The parent of the child will be unset (set to `None`). TYPE: `BaseMetric[SerializableValueType]`

RETURNS	DESCRIPTION
`removed_metric`	The metric that was removed. TYPE: `BaseMetric[SerializableValueType]`

RAISES	DESCRIPTION
`ValueError`	If trying to remove this composite from itself, or the specified metric is not a descendant of this composite.

Source code in src/evallm/metrics/composite.py

def remove(
    self, metric: BaseMetric[SerializableValueType]
) -> BaseMetric[SerializableValueType]:
    """Remove a contained metric from this composite.

    Important:
        When removing a composite metric, all of its descendants are also removed.

    Args:
        metric (BaseMetric[SerializableValueType]): The child metric to remove.
            The parent of the child will be unset (set to `None`).

    Returns:
        removed_metric (BaseMetric[SerializableValueType]): The metric that was
            removed.

    Raises:
        ValueError: If

            - trying to remove this composite from itself, or
            - the specified metric is not a descendant of this composite.
    """
    if metric is self:
        raise ValueError("Cannot remove self from itself.")
    return self._remove_child(metric)

evaluate

evaluate(text: str) -> dict[str, dict[str, SerializableValueType]]

Evaluate all child metrics on the given text.

This method calls evaluate on all child metrics and return the results as a nested dictionary.

PARAMETER	DESCRIPTION
`text`	The text to evaluate the metrics on. TYPE: `str`

RETURNS	DESCRIPTION
`result`	A dictionary containing the results of all child metric evaluations, where each child's result contributes entries keyed by that child's `result_key`. The combined child results are then wrapped in a dictionary under this composite's own name. TYPE: `dict[str, dict[str, SerializableValueType]]`

Referenced by:

API Reference evallm application
- application Classes EvaluationResult Attributes results
- models Classes EvaluationResult Attributes results

Source code in src/evallm/metrics/composite.py

def evaluate(self, text: str) -> dict[str, dict[str, SerializableValueType]]:
    """Evaluate all child metrics on the given text.

    This method calls `evaluate` on all child metrics and return the results as
    a nested dictionary.

    Args:
        text (str): The text to evaluate the metrics on.

    Returns:
        result (dict[str, dict[str, SerializableValueType]]): A dictionary
            containing the results of all child metric evaluations, where each
            child's result contributes entries keyed by that child's
            [`result_key`][evallm.metrics.base.BaseMetric.result_key]. The combined
            child results are then wrapped in a dictionary under this composite's
            own name.
    """
    res: dict[str, SerializableValueType] = {}
    for c in self._children:
        res.update(c.evaluate(text))
    return {self.name: res}

__init_subclass__

__init_subclass__(**kwargs)

Called when a class inherits from BaseMetric.

Sets the default name and config schema of the metric class and validates & stores the parametrized value type of the metric class (i.e. the T in BaseMetric[T]) as the value_type class variable and ensures that evaluate is implemented with the correct signature.

RAISES	DESCRIPTION
`TypeError`	If the metric class does not implement the `evaluate` method with the correct signature.
`UnserializableValueTypeError`	If the metric class's parametrized value type is not an instance of `SerializableValueType`.

Source code in src/evallm/metrics/base.py

def __init_subclass__(cls, **kwargs):
    """Called when a class inherits from `BaseMetric`.

    Sets the default name and config schema of the metric class and validates
    & stores the parametrized value type of the metric class (i.e.
    the `T` in `BaseMetric[T]`) as the
    [`value_type`][evallm.metrics.base.BaseMetric.value_type] class variable
    and ensures that `evaluate` is implemented with the correct signature.

    Raises:
        TypeError: If the metric class does not implement the `evaluate` method
            with the correct signature.
        UnserializableValueTypeError: If the metric class's parametrized value type
            is not an instance of
            [`SerializableValueType`][evallm.metrics.types.SerializableValueType].
    """
    super().__init_subclass__(**kwargs)

    # Set default class name if not already set
    if getattr(cls, "_name", None) is None:
        cls._name = cls.__name__

    # Set default config schema if not already set
    if getattr(cls, "config_schema", None) is None:
        cls.config_schema = DefaultSchema

    # Set the metric's parametrized value type as a class variable
    for base in get_original_bases(cls):
        if get_origin(base) is BaseMetric:
            value_type = get_args(base)[0]
            if not is_serializable_type(value_type):
                raise UnserializableValueTypeError(
                    f"Metric class '{cls.__name__}' has invalid value type annotation "
                    + f"{value_type!r}. Expected a SerializableValueType."
                )
            cls.value_type = value_type
            break

    try:  # Ensure evaluate method is implemented & has correct signature
        _ = check_evaluate_signature(cls)
    except TypeError:
        raise

eq

__eq__(other: object) -> bool

Check if this metric is equal to another metric.

Two metrics are considered equal iff they are the same instance.

PARAMETER	DESCRIPTION
`other`	The other metric to compare against. TYPE: `object`

RETURNS	DESCRIPTION
`bool`	`True` if this metric is the same instance as the other metric, `False` otherwise. TYPE: `bool`

Source code in src/evallm/metrics/base.py

def __eq__(self, other: object) -> bool:
    """Check if this metric is equal to another metric.

    Two metrics are considered equal iff they are the same instance.

    Args:
        other (object): The other metric to compare against.

    Returns:
        bool: `True` if this metric is the same instance as the other metric,
            `False` otherwise.
    """
    return self is other

iter

__iter__() -> Iterator[BaseMetric[SerializableValueType]]

Iterate over the subtree rooted at this metric in breadth-first order.

YIELDS	DESCRIPTION
`metric`	An iterator over the subtree rooted at this metric in breadth-first order, excluding this metric itself. TYPE:: `BaseMetric`

Source code in src/evallm/metrics/base.py

def __iter__(self) -> Iterator[BaseMetric[SerializableValueType]]:
    """Iterate over the subtree rooted at this metric in breadth-first order.

    Yields:
        metric (BaseMetric): An iterator over the subtree rooted at this metric in
            breadth-first order, excluding this metric itself.
    """
    yield from self.descendants(include_self=False, order="bfs")

contains

__contains__(metric: str | BaseMetric[SerializableValueType]) -> bool

Check if a metric is contained in the subtree rooted at this metric.

Note

Passing a metric instance checks for identity (i.e. whether exactly that instance is contained in this subtree), whereas passing a metric's name checks for membership (i.e. whether any metric with that name is contained in this subtree).

Example

leaf1: MetricA = MetricA()  # name='MetricA'
leaf2: MetricA = MetricA()  # name='MetricA'
composite: CompositeMetric = CompositeMetric()
composite.add(leaf1)

Identity-based membership check:

>>> leaf1 in composite
True
>>> leaf2 in composite
False

Name-based membership check:

>>> "MetricA" in composite
True
>>> "NonExistentMetric" in composite
False

PARAMETER	DESCRIPTION
`metric`	The name of the metric to check for, or the metric instance itself. TYPE: `str \| BaseMetric`

RETURNS	DESCRIPTION
`bool`	`True` if the given metric is a descendant of this metric (excluding this metric itself), `False` otherwise. TYPE: `bool`

Source code in src/evallm/metrics/base.py

def __contains__(self, metric: str | BaseMetric[SerializableValueType]) -> bool:
    """Check if a metric is contained in the subtree rooted at this metric.

    Note:
        Passing a metric **instance** checks for **identity** (i.e. whether exactly
        that instance is contained in this subtree), whereas passing a metric's
        **name** checks for **membership** (i.e. whether any metric with that name
        is contained in this subtree).

    Example:
        ```python
        leaf1: MetricA = MetricA()  # name='MetricA'
        leaf2: MetricA = MetricA()  # name='MetricA'
        composite: CompositeMetric = CompositeMetric()
        composite.add(leaf1)
        ```
        Identity-based membership check:
        ```pycon
        >>> leaf1 in composite
        True
        >>> leaf2 in composite
        False
        ```
        Name-based membership check:
        ```pycon
        >>> "MetricA" in composite
        True
        >>> "NonExistentMetric" in composite
        False
        ```

    Args:
        metric (str | BaseMetric): The name of the metric to check for, or the
            metric instance itself.

    Returns:
        bool: `True` if the given metric is a descendant of this metric
            (excluding this metric itself), `False` otherwise.
    """
    if isinstance(metric, str):
        return any(node.name == metric for node in self.descendants())
    return any(node is metric for node in self.descendants())

repr

__repr__() -> str

String representation of the subtree, rooted in this metric instance.

RETURNS	DESCRIPTION
`representation`	A string representation of the metric instance, including its name and class. TYPE: `str`

Source code in src/evallm/metrics/base.py

def __repr__(self) -> str:
    """String representation of the subtree, rooted in this metric instance.

    Returns:
        representation (str): A string representation of the metric instance,
            including its name and class.
    """
    return self._repr_tree()

result

result(value: T) -> dict[str, T]

Helper method for wrapping the value of a metric's evaluation result in a dictionary to conform to the expected return type of evaluate.

Example

class CustomMetric(BaseMetric[list[int]]):
    def evaluate(self, text: str) -> dict[str, list[int]]:
        # Perform some evaluation logic and compute the metric's value
        res: list[int] = [len(text.split())]

        # Wrap the result in a dictionary keyed by the metric's result_key
        return self.result(res)

PARAMETER	DESCRIPTION
`value`	The result of computing the metric, where `T` is the metric's parametrized value type. TYPE: `T`

RETURNS	DESCRIPTION
`result`	A dictionary containing the given value, keyed by the metric's `result_key`. TYPE: `dict[str, T]`

RAISES	DESCRIPTION
`TypeError`	If the given value is not of the expected type `T` or is not JSON-serializable.

Referenced by:

API Reference evallm metrics
- base Classes BaseMetric
- metrics Extension Path for Custom Metrics 2. Implementing the evaluate method

Source code in src/evallm/metrics/base.py

def result(self, value: T) -> dict[str, T]:  # pragma: no cover
    """Helper method for wrapping the value of a metric's evaluation result in a
    dictionary to conform to the expected return type of `evaluate`.

    Example:
        ```python
        class CustomMetric(BaseMetric[list[int]]):
            def evaluate(self, text: str) -> dict[str, list[int]]:
                # Perform some evaluation logic and compute the metric's value
                res: list[int] = [len(text.split())]

                # Wrap the result in a dictionary keyed by the metric's result_key
                return self.result(res)
        ```

    Args:
        value (T): The result of computing the metric, where `T` is the metric's
            parametrized value type.

    Returns:
        result (dict[str, T]): A dictionary containing the given value, keyed by
            the metric's [`result_key`][..result_key].

    Raises:
        TypeError: If the given value is not of the expected type `T` or is not
            JSON-serializable.
    """
    if not matches_annotation(value, self.value_type):
        raise TypeError(
            f"Value {value!r} of type {type(value)!r} is not of expected type "
            + f"{self.value_type!r} for metric '{self.name}'."
        )
    return {self.result_key: value}

descendants

descendants(
    *, include_self: bool = False, order: Literal["dfs", "bfs"] = "dfs"
) -> Iterator[BaseMetric[SerializableValueType]]

Yield nodes in the subtree rooted at self (cycle-safe).

PARAMETER	DESCRIPTION
`include_self`	Whether to include `self` in the yielded nodes. TYPE: `bool` DEFAULT: `False`
`order`	The order to traverse the tree. "dfs" for depth-first search, "bfs" for breadth-first search. TYPE: `'dfs' \| 'bfs'` DEFAULT: `'dfs'`

YIELDS	DESCRIPTION
`descendants`	An iterator over the descendant nodes of this metric, in the specified traversal order. TYPE:: `Iterator[BaseMetric]`

Source code in src/evallm/metrics/base.py

def descendants(
    self,
    *,
    include_self: bool = False,
    order: Literal["dfs", "bfs"] = "dfs",
) -> Iterator[BaseMetric[SerializableValueType]]:
    """Yield nodes in the subtree rooted at self (cycle-safe).

    Args:
        include_self (bool): Whether to include `self` in the yielded nodes.
        order ("dfs" | "bfs"): The order to traverse the tree. "dfs" for depth-first
            search, "bfs" for breadth-first search.

    Yields:
        descendants (Iterator[BaseMetric]): An iterator over the descendant nodes of this metric,
            in the specified traversal order.
    """
    visited: set[int] = set()

    # Populate the frontier with the initial node(s) based on traversal order
    frontier: deque[BaseMetric[SerializableValueType]] = deque()
    if include_self:
        frontier.append(cast(BaseMetric[SerializableValueType], self))
    else:  # keep order of children consistent across DFS/BFS
        if order == "dfs":
            frontier.extend(reversed(list(self.children)))
        else:
            frontier.extend(list(self.children))

    while frontier:
        # Get the next node to visit, depending on traversal order
        node: BaseMetric[SerializableValueType] = (
            frontier.pop() if order == "dfs" else frontier.popleft()
        )

        # Skip visited nodes to prevent infinite loops
        if (node_id := id(node)) in visited:
            continue
        visited.add(node_id)

        yield node

        # Expand neighbors
        if order == "dfs":
            frontier.extend(  # reverse for correct DFS order
                reversed(list(node.children))
            )
        else:
            frontier.extend(list(node.children))

from_config `classmethod`

from_config(*, json_payload: str | None = None, **kwargs: Any) -> BaseMetric[T]

Factory method to create a metric instance from kwargs or JSON payload.

Info

This method can be used to create any metric instance, so long as the provided config parameters conform to the metric's schema. If the metric class doesn't define a config schema (the default), calling this method without kwargs is equivalent to calling the metric class constructor directly. The expected parameters depend on the metric's declared config_schema (see also @config_schema decorator) and the provided parameters must strictly conform to it.

PARAMETER	DESCRIPTION
`json_payload`	A JSON string containing the runtime parameters for this metric instance. TYPE: `str \| None` DEFAULT: `None`

PARAMETER	DESCRIPTION
`**kwargs`	The runtime parameters to use for this metric instance. TYPE: `Any`

RETURNS	DESCRIPTION
`metric`	An instance of this metric class created using the provided configuration parameters. TYPE: `BaseMetric[T]`

RAISES	DESCRIPTION
`TypeError`	If `json_payload` is not a string or `None`.
`ValueError`	If trying to pass runtime parameters to a metric class that doesn't define a `config_schema`, or trying to instantiate a metric class with invalid runtime parameters (i.e. missing required parameters, extra unknown parameters, or parameters of the wrong type), or passing an invalid JSON payload, or passing both `json_payload` and `kwargs` at the same time.

Referenced by:

API Reference evallm
- application
  - application Classes MetricSpec Attributes config
  - models Classes MetricSpec Attributes config
- metrics base Classes DefaultSchema

Source code in src/evallm/metrics/base.py

@classmethod
def from_config(
    cls, *, json_payload: str | None = None, **kwargs: Any
) -> BaseMetric[T]:
    """Factory method to create a metric instance from kwargs or JSON payload.

    !!! info
        This method can be used to create any metric instance, so long as the
        provided config parameters conform to the metric's schema. If the metric
        class doesn't define a config schema (the default), calling this method
        without kwargs is equivalent to calling the metric class constructor
        directly.
        The expected parameters depend on the metric's declared config_schema
        (see also [`@config_schema` decorator][evallm.metrics.config.config_schema])
        and the provided parameters must strictly conform to it.

    Args:
        json_payload (str | None): A JSON string containing the runtime parameters
            for this metric instance.

    Keyword Args:
        **kwargs (Any): The runtime parameters to use for this metric instance.

    Returns:
        metric (BaseMetric[T]): An instance of this metric class created using the
            provided configuration parameters.

    Raises:
        TypeError: If `json_payload` is not a string or `None`.
        ValueError: If

            - trying to pass runtime parameters to a metric class that doesn't
                define a `config_schema`, or
            - trying to instantiate a metric class with invalid runtime parameters
                (i.e. missing required parameters, extra unknown parameters, or
                parameters of the wrong type), or
            - passing an invalid JSON payload, or
            - passing both `json_payload` and `kwargs` at the same time.
    """
    runtime_params: dict[str, Any]
    if json_payload is not None:
        if kwargs:
            raise ValueError("Cannot mix json_payload with kwargs.")
        if not isinstance(json_payload, str):
            raise TypeError("json_payload must be a string or None.")
        try:
            runtime_params = orjson.loads(json_payload)
        except orjson.JSONDecodeError:
            raise ValueError(
                "Invalid json_payload: not a valid JSON string."
            ) from None
    else:
        runtime_params = kwargs

    return cls(**runtime_params)

composite

Type Aliases

Classes

CompositeMetric

Attributes

value_type class-attribute

config_schema class-attribute

parent property writable

result_key property writable

name property

is_composite property

is_root property

root property

depth property

children property

config property

Functions

add

remove

evaluate

__init_subclass__

__eq__

__iter__

__contains__

__repr__

result

descendants

from_config classmethod

value_type `class-attribute`

config_schema `class-attribute`

parent `property` `writable`

result_key `property` `writable`

name `property`

is_composite `property`

is_root `property`

root `property`

depth `property`

children `property`

config `property`

eq

iter

contains

repr

from_config `classmethod`