builtin

Concrete implementations of built-in metrics

Submodules

Use mouse to pan and zoom

Use mouse to pan and zoom

Use mouse to pan and zoom

Use mouse to pan and zoom

MODULE	DESCRIPTION
counts	CountsMetric — count the number of segments in an input text
json_format	JsonFormatMetric — validate the adherence of a text to an expected output format

CLASS	DESCRIPTION
CountsConfig	The configuration schema for the CountsMetric.
CountsMetric	Metric for counting segments in an input text.
JsonFormatConfig	The configuration schema for the JsonFormatMetric.
JsonFormatMetric	Metric for evaluating whether text matches a given JSON format.

Classes

CountsConfig pydantic-model

The configuration schema for the CountsMetric.

PARAMETER	DESCRIPTION
segments	(default = "characters") Which kind of segments to count. If "characters", the text is stripped of leading & trailing whitespace and each remaining character is considered a segment. See the segmentation parameter for how the text is split into segments when segments is set to "tokens" or "encodings". TYPE: Literal['characters', 'tokens', 'encodings'] | None
segmentation	How to split the input text into segments. If segments = "characters", this parameter is ignored. If segments = "tokens", this can be "whitespace" to split on any whitespace (default), or "wordpunct" to tokenize the text into words and punctuation using "\w+|[^\w\s]+" as a regular expression. If segments = "encodings", this must be the name of a HuggingFace model that provides a pretrained tokenizer, which will be used to segment the text into model-specific encodings. TYPE: Literal['whitespace', 'wordpunct'] | str | None
hf_token	User access token for HuggingFace. This is only required when specifying a gated model to use for encoding. Note that you can also set the HF_TOKEN environment variable to avoid passing the token directly in the configuration. TYPE: str | None

Example configuration

characters: CountsMetric = CountsMetric()  # (1)!

tokens_ws: CountsMetric = CountsMetric.from_config(
    {
        "segments": "tokens"  # (2)!
    }
)

tokens_wordpunct: CountsMetric = CountsMetric.from_config(
    **{
        "segments": "tokens",
        "segmentation": "wordpunct",  # (3)!
    }
)

encs_bart: CountsMetric = CountsMetric.from_config(
    {
        "segments": "encodings",  # (4)!
        "segmentation": "facebook/bart-base",
    }
)


encs_bert_amharic: CountsMetric = CountsMetric.from_config(
    **{
        "segments": "encodings",
        "segmentation": "rasyosef/bert-amharic-tokenizer",  # (5)!
    }
)

encs_llama: CountsMetric = CountsMetric.from_config(
    **{
        "segments": "encodings",
        "segmentation": "meta-llama/Llama-3.3-70B-Instruct",
        "hf_token": "hf_...",  # (6)!
    }
)

1. By default, the CountsMetric simply counts the number of characters.
2. To count tokens instead of characters, set segments="tokens". This will split the text into tokens separated by one or more whitespace characters by default.
3. Use a slightly more sophisticated tokenization scheme.
4. To count model-specific encodings, set segments="encodings" and specify the model to use for encoding in the segmentation field.
5. Maybe you're working with non-English texts? No problem, just specify a model that provides a tokenizer pretrained on the relevant language.
6. If you want to use a gated model for encoding, make sure to provide a valid HuggingFace user access token with the necessary permissions to access the model. You can also set the HF_TOKEN environment variable instead if you prefer not to pass the token in directly.

Fields:

• segments (Literal['characters', 'tokens', 'encodings'])
• segmentation (str | None)
• hf_token (str | None)

Validators:

• _validate_config

Attributes

tokenizer cached property

tokenizer: Callable[[str], Sequence[str | int]]

The function to use for segmenting the input text, based on the configuration.

RETURNS	DESCRIPTION
Callable[[str], Sequence[str | int]]	Callable[[str], Sequence[str | int]]: A function that takes a string as input and returns a sequence of segments, where each segment is either a string (for tokenization) or an integer (for encoding).

CountsMetric

CountsMetric(**kwargs: dict[str, Any])

              flowchart TD
              evallm.metrics.builtin.CountsMetric[CountsMetric]
              evallm.metrics.base.BaseMetric[BaseMetric]

                              evallm.metrics.base.BaseMetric --> evallm.metrics.builtin.CountsMetric
                


              click evallm.metrics.builtin.CountsMetric href "" "evallm.metrics.builtin.CountsMetric"
              click evallm.metrics.base.BaseMetric href "" "evallm.metrics.base.BaseMetric"
            

Use mouse to pan and zoom

Metric for counting segments in an input text.

Evaluation results

Given an input string, this metric produces a dictionary mapping the metric's name to a single count. The count represents the number of segments that are contained in the input text, where the type of segments counted (characters, tokens, or encodings) can be configured through the segments and segmentation runtime parameters. If no configuration is specified, this metric will simply count the number of characters in the input text (after stripping leading and trailing whitespace) by default.

Note that specifiying a model to use for encoding input texts requires installing evaLLM with the hf-tokenizers extra:

pip install evallm-qa[hf-tokenizers]

PARAMETER	DESCRIPTION
segments	(default = "characters") Which kind of segments to count. If "characters", the text is stripped of leading & trailing whitespace and each remaining character is considered a segment. See the segmentation parameter for how the text is split into segments when segments is set to "tokens" or "encodings". TYPE: Literal['characters', 'tokens', 'encodings'] | None
segmentation	How to split the input text into segments. If segments = "characters", this parameter is ignored. If segments = "tokens", this can be "whitespace" to split on any whitespace (default), or "wordpunct" to tokenize the text into words and punctuation using "\w+|[^\w\s]+" as a regular expression. If segments = "encodings", this must be the name of a HuggingFace model that provides a pretrained tokenizer, which will be used to segment the text into model-specific encodings. TYPE: Literal['whitespace', 'wordpunct'] | str | None
hf_token	User access token for HuggingFace. This is only required when specifying a gated model to use for encoding. Note that you can also set the HF_TOKEN environment variable to avoid passing the token directly in the configuration. TYPE: str | None

Referenced by:

• API Reference evallm metrics  builtin Classes  CountsConfig

METHOD	DESCRIPTION
__init_subclass__	Called when a class inherits from BaseMetric.
__eq__	Check if this metric is equal to another metric.
__iter__	Iterate over the subtree rooted at this metric in breadth-first order.
__contains__	Check if a metric is contained in the subtree rooted at this metric.
__repr__	String representation of the subtree, rooted in this metric instance.
result	Helper method for wrapping the value of a metric's evaluation result in a
descendants	Yield nodes in the subtree rooted at self (cycle-safe).
from_config	Factory method to create a metric instance from kwargs or JSON payload.
evaluate	Evaluate how many segments are contained in the input text.

ATTRIBUTE	DESCRIPTION
value_type	The metric class' parametrized type, binding the generic type parameter T. TYPE: object
config_schema	A schema defining the expected runtime configuration parameters for this metric. TYPE: type[BaseModel]
parent	Get the parent of this metric. TYPE: CompositeMetric | None
result_key	The key to use for this metric's value in the evaluation result dictionary. TYPE: str
name	The name of this metric's class. TYPE: str
is_composite	Check if this metric is a composite metric (i.e. can have child metrics). TYPE: bool
is_root	Check if this metric is a root metric (i.e. has no parent). TYPE: bool
root	Get the root metric of the tree this metric belongs to. TYPE: BaseMetric[SerializableValueType]
depth	Get the depth of this metric in the tree (i.e. the number of edges from this TYPE: int
children	Get an iterator over the children of this metric. TYPE: Iterator[BaseMetric[SerializableValueType]]
config	Get the runtime configuration of this metric instance as a dictionary. TYPE: dict[str, Any]

Source code in src/evallm/metrics/base.py

def __init__(self, **kwargs: dict[str, Any]):
    """ """
    self.parent: BaseMetric[SerializableValueType] | None = None
    self._config: dict[str, Any] = {}  # default

    # raise error when trying to pass runtime params without a config schema
    if (schema := type(self).config_schema).model_json_schema().get(
        "properties", {}
    ) == {}:
        if kwargs:
            raise ValueError(f"{type(self)._name} takes no runtime parameters.")
    else:  # validate runtime params against config_schema
        try:
            model: BaseModel = schema.model_validate(kwargs)
            self._config = model.model_dump()
        except ValidationError:
            raise

Attributes

value_type class-attribute

value_type: object

The metric class' parametrized type, binding the generic type parameter T.

Referenced by:

• API Reference evallm metrics
  •  base Classes  BaseMetric Functions  __init_subclass__
  •  builtin Classes
    •  CountsMetric Functions  __init_subclass__
    •  JsonFormatMetric Functions  __init_subclass__
  •  composite Classes  CompositeMetric Functions  __init_subclass__
  •  metrics Extension Path for Custom Metrics
  •  types Functions  check_evaluate_signature

config_schema class-attribute

config_schema: type[BaseModel]

A schema defining the expected runtime configuration parameters for this metric.

If the metric class does not explicitly define a config schema, this defaults to an empty pydantic model (see DefaultSchema) that takes no parameters.

Info

A class' config schema is always configured such that:

• all declared fields must be provided unless they are declared as optional
• no additional fields beyond those declared in the schema are allowed (see also pydantic.ConfigDict.extra)
• the schema is frozen/immutable once defined (see also pydantic.ConfigDict.frozen)
• configuration parameters provided at runtime are strictly validated against the schema, raising a ValidationError if the provided parameters violate it (see also pydantic Conversion Table)

Referenced by:

• API Reference evallm
  • application
    •  api Functions  metric
    •  application
      • Classes  MetricSpec Attributes  config
      • Functions  metric
    •  models Classes  MetricSpec Attributes  config
  • metrics
    •  base Classes  BaseMetric
    •  config Functions  config_schema

parent property writable

parent: CompositeMetric | None

Get the parent of this metric.

RETURNS	DESCRIPTION
CompositeMetric | None	CompositeMetric | None: The parent metric, or None if this metric has no parent, i.e. is a leaf node.

result_key property writable

result_key: str

The key to use for this metric's value in the evaluation result dictionary.

By default, this is the same as the metric's name, but it can be overridden on a per-instance basis. This is useful when multiple instances of the same metric class are used in a single evaluation request with different runtime parameters and therefore need distinct keys in the output.

RETURNS	DESCRIPTION
result_key	The key of this metric instance's value in the evaluation result dictionary. Defaults to the metric's name. TYPE: str

Referenced by:

• API Reference evallm
  • metrics
    •  base Classes  BaseMetric
      • Attributes  name
      • Functions
        •  evaluate
        •  result
    •  builtin Classes
      •  CountsMetric
        • Attributes  name
        • Functions
          •  evaluate
          •  result
      •  JsonFormatMetric
        • Attributes  name
        • Functions
          •  evaluate
          •  result
    •  composite Classes  CompositeMetric
      • Attributes  name
      • Functions
        •  evaluate
        •  result
    •  metrics Extension Path for Custom Metrics
    •  types Functions  check_evaluate_signature
  • application
    •  api Functions
      •  count_characters
      •  count_encodings
      •  count_tokens
      •  metric
    •  application
      • Classes  EvaluationResult Attributes  results
      • Functions
        •  count_characters
        •  count_encodings
        •  count_tokens
        •  metric
    •  models Classes  EvaluationResult Attributes  results

name property

name: str

The name of this metric's class.

This is a class-, not an instance-level property that's used for identifying this metric in the catalog. It also serves as the default result_key for metric instances unless explicitly overridden.

Metric class registration and name updates

When a metric class is registered in the catalog under a custom name using @register_metric(name=...), the name-property of any pre-existing instances of that class will be updated to reflect the new name in the catalog.

RETURNS	DESCRIPTION
name	The name of this metric. Defaults to the class name if not overwritten by the @register_metric decorator at definition time. TYPE: str

Referenced by:

• API Reference evallm
  • metrics
    •  base Classes  BaseMetric Attributes  result_key
    •  builtin Classes
      •  CountsMetric Attributes  result_key
      •  JsonFormatMetric Attributes  result_key
    •  composite Classes  CompositeMetric Attributes  result_key
  • application
    •  application Classes
      •  EvaluationResult Attributes  results
      •  MetricSpec Attributes  name
    •  models Classes
      •  EvaluationResult Attributes  results
      •  MetricSpec Attributes  name

is_composite property

is_composite: bool

Check if this metric is a composite metric (i.e. can have child metrics).

RETURNS	DESCRIPTION
bool	True if this metric is a composite metric, False otherwise. TYPE: bool

is_root property

is_root: bool

Check if this metric is a root metric (i.e. has no parent).

RETURNS	DESCRIPTION
bool	True if this metric has no parent, False otherwise. TYPE: bool

root property

root: BaseMetric[SerializableValueType]

Get the root metric of the tree this metric belongs to.

RETURNS	DESCRIPTION
BaseMetric	The root metric of the tree this metric belongs to. TYPE: BaseMetric[SerializableValueType]

depth property

depth: int

Get the depth of this metric in the tree (i.e. the number of edges from this metric to the root).

RETURNS	DESCRIPTION
depth	The depth of this metric in the tree. Root metrics have depth 0, their children have depth 1, and so on. TYPE: int

children property

children: Iterator[BaseMetric[SerializableValueType]]

Get an iterator over the children of this metric.

YIELDS	DESCRIPTION
children	An iterator over the child metrics (i.e. direct descendants) of this metric. If this metric is a leaf node, this will be an empty iterator. TYPE:: BaseMetric

config property

config: dict[str, Any]

Get the runtime configuration of this metric instance as a dictionary.

RETURNS	DESCRIPTION
config	A dictionary containing the runtime configuration of this metric instance, validated against the metric's schema. TYPE: dict[str, Any]

Referenced by:

• API Reference evallm metrics  config Functions  config_schema

Functions

__init_subclass__

__init_subclass__(**kwargs)

Called when a class inherits from BaseMetric.

Sets the default name and config schema of the metric class and validates & stores the parametrized value type of the metric class (i.e. the T in BaseMetric[T]) as the value_type class variable and ensures that evaluate is implemented with the correct signature.

RAISES	DESCRIPTION
TypeError	If the metric class does not implement the evaluate method with the correct signature.
UnserializableValueTypeError	If the metric class's parametrized value type is not an instance of SerializableValueType.

Source code in src/evallm/metrics/base.py

def __init_subclass__(cls, **kwargs):
    """Called when a class inherits from `BaseMetric`.

    Sets the default name and config schema of the metric class and validates
    & stores the parametrized value type of the metric class (i.e.
    the `T` in `BaseMetric[T]`) as the
    [`value_type`][evallm.metrics.base.BaseMetric.value_type] class variable
    and ensures that `evaluate` is implemented with the correct signature.

    Raises:
        TypeError: If the metric class does not implement the `evaluate` method
            with the correct signature.
        UnserializableValueTypeError: If the metric class's parametrized value type
            is not an instance of
            [`SerializableValueType`][evallm.metrics.types.SerializableValueType].
    """
    super().__init_subclass__(**kwargs)

    # Set default class name if not already set
    if getattr(cls, "_name", None) is None:
        cls._name = cls.__name__

    # Set default config schema if not already set
    if getattr(cls, "config_schema", None) is None:
        cls.config_schema = DefaultSchema

    # Set the metric's parametrized value type as a class variable
    for base in get_original_bases(cls):
        if get_origin(base) is BaseMetric:
            value_type = get_args(base)[0]
            if not is_serializable_type(value_type):
                raise UnserializableValueTypeError(
                    f"Metric class '{cls.__name__}' has invalid value type annotation "
                    + f"{value_type!r}. Expected a SerializableValueType."
                )
            cls.value_type = value_type
            break

    try:  # Ensure evaluate method is implemented & has correct signature
        _ = check_evaluate_signature(cls)
    except TypeError:
        raise

__eq__

__eq__(other: object) -> bool

Check if this metric is equal to another metric.

Two metrics are considered equal iff they are the same instance.

PARAMETER	DESCRIPTION
other	The other metric to compare against. TYPE: object

RETURNS	DESCRIPTION
bool	True if this metric is the same instance as the other metric, False otherwise. TYPE: bool

Source code in src/evallm/metrics/base.py

def __eq__(self, other: object) -> bool:
    """Check if this metric is equal to another metric.

    Two metrics are considered equal iff they are the same instance.

    Args:
        other (object): The other metric to compare against.

    Returns:
        bool: `True` if this metric is the same instance as the other metric,
            `False` otherwise.
    """
    return self is other

__iter__

__iter__() -> Iterator[BaseMetric[SerializableValueType]]

Iterate over the subtree rooted at this metric in breadth-first order.

YIELDS	DESCRIPTION
metric	An iterator over the subtree rooted at this metric in breadth-first order, excluding this metric itself. TYPE:: BaseMetric

Source code in src/evallm/metrics/base.py

def __iter__(self) -> Iterator[BaseMetric[SerializableValueType]]:
    """Iterate over the subtree rooted at this metric in breadth-first order.

    Yields:
        metric (BaseMetric): An iterator over the subtree rooted at this metric in
            breadth-first order, excluding this metric itself.
    """
    yield from self.descendants(include_self=False, order="bfs")

__contains__

__contains__(metric: str | BaseMetric[SerializableValueType]) -> bool

Check if a metric is contained in the subtree rooted at this metric.

Note

Passing a metric instance checks for identity (i.e. whether exactly that instance is contained in this subtree), whereas passing a metric's name checks for membership (i.e. whether any metric with that name is contained in this subtree).

Example

leaf1: MetricA = MetricA()  # name='MetricA'
leaf2: MetricA = MetricA()  # name='MetricA'
composite: CompositeMetric = CompositeMetric()
composite.add(leaf1)

Identity-based membership check:

>>> leaf1 in composite
True
>>> leaf2 in composite
False

Name-based membership check:

>>> "MetricA" in composite
True
>>> "NonExistentMetric" in composite
False

PARAMETER	DESCRIPTION
metric	The name of the metric to check for, or the metric instance itself. TYPE: str | BaseMetric

RETURNS	DESCRIPTION
bool	True if the given metric is a descendant of this metric (excluding this metric itself), False otherwise. TYPE: bool

Source code in src/evallm/metrics/base.py

def __contains__(self, metric: str | BaseMetric[SerializableValueType]) -> bool:
    """Check if a metric is contained in the subtree rooted at this metric.

    Note:
        Passing a metric **instance** checks for **identity** (i.e. whether exactly
        that instance is contained in this subtree), whereas passing a metric's
        **name** checks for **membership** (i.e. whether any metric with that name
        is contained in this subtree).

    Example:
        ```python
        leaf1: MetricA = MetricA()  # name='MetricA'
        leaf2: MetricA = MetricA()  # name='MetricA'
        composite: CompositeMetric = CompositeMetric()
        composite.add(leaf1)
        ```
        Identity-based membership check:
        ```pycon
        >>> leaf1 in composite
        True
        >>> leaf2 in composite
        False
        ```
        Name-based membership check:
        ```pycon
        >>> "MetricA" in composite
        True
        >>> "NonExistentMetric" in composite
        False
        ```

    Args:
        metric (str | BaseMetric): The name of the metric to check for, or the
            metric instance itself.

    Returns:
        bool: `True` if the given metric is a descendant of this metric
            (excluding this metric itself), `False` otherwise.
    """
    if isinstance(metric, str):
        return any(node.name == metric for node in self.descendants())
    return any(node is metric for node in self.descendants())

__repr__

__repr__() -> str

String representation of the subtree, rooted in this metric instance.

RETURNS	DESCRIPTION
representation	A string representation of the metric instance, including its name and class. TYPE: str

Source code in src/evallm/metrics/base.py

def __repr__(self) -> str:
    """String representation of the subtree, rooted in this metric instance.

    Returns:
        representation (str): A string representation of the metric instance,
            including its name and class.
    """
    return self._repr_tree()

result

result(value: T) -> dict[str, T]

Helper method for wrapping the value of a metric's evaluation result in a dictionary to conform to the expected return type of evaluate.

Example

class CustomMetric(BaseMetric[list[int]]):
    def evaluate(self, text: str) -> dict[str, list[int]]:
        # Perform some evaluation logic and compute the metric's value
        res: list[int] = [len(text.split())]

        # Wrap the result in a dictionary keyed by the metric's result_key
        return self.result(res)

PARAMETER	DESCRIPTION
value	The result of computing the metric, where T is the metric's parametrized value type. TYPE: T

RETURNS	DESCRIPTION
result	A dictionary containing the given value, keyed by the metric's result_key. TYPE: dict[str, T]

RAISES	DESCRIPTION
TypeError	If the given value is not of the expected type T or is not JSON-serializable.

Referenced by:

• API Reference evallm metrics
  •  base Classes  BaseMetric
  •  metrics Extension Path for Custom Metrics 2. Implementing the evaluate method

Source code in src/evallm/metrics/base.py

def result(self, value: T) -> dict[str, T]:  # pragma: no cover
    """Helper method for wrapping the value of a metric's evaluation result in a
    dictionary to conform to the expected return type of `evaluate`.

    Example:
        ```python
        class CustomMetric(BaseMetric[list[int]]):
            def evaluate(self, text: str) -> dict[str, list[int]]:
                # Perform some evaluation logic and compute the metric's value
                res: list[int] = [len(text.split())]

                # Wrap the result in a dictionary keyed by the metric's result_key
                return self.result(res)
        ```

    Args:
        value (T): The result of computing the metric, where `T` is the metric's
            parametrized value type.

    Returns:
        result (dict[str, T]): A dictionary containing the given value, keyed by
            the metric's [`result_key`][..result_key].

    Raises:
        TypeError: If the given value is not of the expected type `T` or is not
            JSON-serializable.
    """
    if not matches_annotation(value, self.value_type):
        raise TypeError(
            f"Value {value!r} of type {type(value)!r} is not of expected type "
            + f"{self.value_type!r} for metric '{self.name}'."
        )
    return {self.result_key: value}

descendants

descendants(
    *, include_self: bool = False, order: Literal["dfs", "bfs"] = "dfs"
) -> Iterator[BaseMetric[SerializableValueType]]

Yield nodes in the subtree rooted at self (cycle-safe).

PARAMETER	DESCRIPTION
include_self	Whether to include self in the yielded nodes. TYPE: bool DEFAULT: False
order	The order to traverse the tree. "dfs" for depth-first search, "bfs" for breadth-first search. TYPE: 'dfs' | 'bfs' DEFAULT: 'dfs'

YIELDS	DESCRIPTION
descendants	An iterator over the descendant nodes of this metric, in the specified traversal order. TYPE:: Iterator[BaseMetric]

Source code in src/evallm/metrics/base.py

def descendants(
    self,
    *,
    include_self: bool = False,
    order: Literal["dfs", "bfs"] = "dfs",
) -> Iterator[BaseMetric[SerializableValueType]]:
    """Yield nodes in the subtree rooted at self (cycle-safe).

    Args:
        include_self (bool): Whether to include `self` in the yielded nodes.
        order ("dfs" | "bfs"): The order to traverse the tree. "dfs" for depth-first
            search, "bfs" for breadth-first search.

    Yields:
        descendants (Iterator[BaseMetric]): An iterator over the descendant nodes of this metric,
            in the specified traversal order.
    """
    visited: set[int] = set()

    # Populate the frontier with the initial node(s) based on traversal order
    frontier: deque[BaseMetric[SerializableValueType]] = deque()
    if include_self:
        frontier.append(cast(BaseMetric[SerializableValueType], self))
    else:  # keep order of children consistent across DFS/BFS
        if order == "dfs":
            frontier.extend(reversed(list(self.children)))
        else:
            frontier.extend(list(self.children))

    while frontier:
        # Get the next node to visit, depending on traversal order
        node: BaseMetric[SerializableValueType] = (
            frontier.pop() if order == "dfs" else frontier.popleft()
        )

        # Skip visited nodes to prevent infinite loops
        if (node_id := id(node)) in visited:
            continue
        visited.add(node_id)

        yield node

        # Expand neighbors
        if order == "dfs":
            frontier.extend(  # reverse for correct DFS order
                reversed(list(node.children))
            )
        else:
            frontier.extend(list(node.children))

from_config classmethod

from_config(*, json_payload: str | None = None, **kwargs: Any) -> BaseMetric[T]

Factory method to create a metric instance from kwargs or JSON payload.

Info

This method can be used to create any metric instance, so long as the provided config parameters conform to the metric's schema. If the metric class doesn't define a config schema (the default), calling this method without kwargs is equivalent to calling the metric class constructor directly. The expected parameters depend on the metric's declared config_schema (see also @config_schema decorator) and the provided parameters must strictly conform to it.

PARAMETER	DESCRIPTION
json_payload	A JSON string containing the runtime parameters for this metric instance. TYPE: str | None DEFAULT: None

PARAMETER	DESCRIPTION
**kwargs	The runtime parameters to use for this metric instance. TYPE: Any

RETURNS	DESCRIPTION
metric	An instance of this metric class created using the provided configuration parameters. TYPE: BaseMetric[T]

RAISES	DESCRIPTION
TypeError	If json_payload is not a string or None.
ValueError	If trying to pass runtime parameters to a metric class that doesn't define a config_schema, or trying to instantiate a metric class with invalid runtime parameters (i.e. missing required parameters, extra unknown parameters, or parameters of the wrong type), or passing an invalid JSON payload, or passing both json_payload and kwargs at the same time.

Referenced by:

• API Reference evallm
  • application
    •  application Classes  MetricSpec Attributes  config
    •  models Classes  MetricSpec Attributes  config
  • metrics  base Classes  DefaultSchema

Source code in src/evallm/metrics/base.py

@classmethod
def from_config(
    cls, *, json_payload: str | None = None, **kwargs: Any
) -> BaseMetric[T]:
    """Factory method to create a metric instance from kwargs or JSON payload.

    !!! info
        This method can be used to create any metric instance, so long as the
        provided config parameters conform to the metric's schema. If the metric
        class doesn't define a config schema (the default), calling this method
        without kwargs is equivalent to calling the metric class constructor
        directly.
        The expected parameters depend on the metric's declared config_schema
        (see also [`@config_schema` decorator][evallm.metrics.config.config_schema])
        and the provided parameters must strictly conform to it.

    Args:
        json_payload (str | None): A JSON string containing the runtime parameters
            for this metric instance.

    Keyword Args:
        **kwargs (Any): The runtime parameters to use for this metric instance.

    Returns:
        metric (BaseMetric[T]): An instance of this metric class created using the
            provided configuration parameters.

    Raises:
        TypeError: If `json_payload` is not a string or `None`.
        ValueError: If

            - trying to pass runtime parameters to a metric class that doesn't
                define a `config_schema`, or
            - trying to instantiate a metric class with invalid runtime parameters
                (i.e. missing required parameters, extra unknown parameters, or
                parameters of the wrong type), or
            - passing an invalid JSON payload, or
            - passing both `json_payload` and `kwargs` at the same time.
    """
    runtime_params: dict[str, Any]
    if json_payload is not None:
        if kwargs:
            raise ValueError("Cannot mix json_payload with kwargs.")
        if not isinstance(json_payload, str):
            raise TypeError("json_payload must be a string or None.")
        try:
            runtime_params = orjson.loads(json_payload)
        except orjson.JSONDecodeError:
            raise ValueError(
                "Invalid json_payload: not a valid JSON string."
            ) from None
    else:
        runtime_params = kwargs

    return cls(**runtime_params)

evaluate

evaluate(text: str) -> dict[str, int]

Evaluate how many segments are contained in the input text.

PARAMETER	DESCRIPTION
text	The input text to evaluate. TYPE: str

RETURNS	DESCRIPTION
dict[str, int]	dict[str, int]: A dictionary mapping this metric instance's result_key to the count of segments in the input text. The type of segments counted (characters, tokens, or encodings) is determined by the segments configuration parameter, and how the text is split into segments is determined by the segmentation configuration parameter when applicable.

Source code in src/evallm/metrics/builtin/counts.py

def evaluate(self, text: str) -> dict[str, int]:
    """Evaluate how many segments are contained in the input text.

    Args:
        text (str): The input text to evaluate.

    Returns:
        dict[str, int]: A dictionary mapping this metric instance's
            [`result_key`][evallm.metrics.base.BaseMetric.result_key] to the count
            of segments in the input text. The type of segments counted
            (characters, tokens, or encodings) is determined by the `segments`
            configuration parameter, and how the text is split into segments is
            determined by the `segmentation` configuration parameter when
            applicable.
    """
    if text is None or text.strip() == "":
        res: int = 0
    else:
        res = len(self.config.get("tokenizer")(text))  # type: ignore[misc]

    return self.result(res)

JsonFormatConfig pydantic-model

The configuration schema for the JsonFormatMetric.

PARAMETER	DESCRIPTION
expected_schema	A valid JSON schema conforming to the JSON Schema draft-2020-12 specification that describes the expected output format. Can be provided either as a JSON payload string or as a dictionary. Note that the schema must be a top-level schema, defined as 'type': 'object' with a 'properties' attribute containing the expected fields. Fields that aren't explicitly listed in the 'required' attribute of the schema, are considered optional by default. TYPE: str | dict[str, Any]
check_formats	(default=False) Whether to check constraints in 'format' annotations of primitive types. If True, the following formats will additionally be validated where present 'date' 'date-time' 'duration' 'email' 'hostname' 'idn-email' 'idn-hostname' 'ipv4' 'ipv6' 'iri' 'iri-reference' 'json-pointer' 'regex'1 'relative-json-pointer' 'time' 'uri' 'uri-reference' 'uri-template' 'uuid' While the JSON Schema specification recommends using ECMA 262 regular expressions, validation for format annotations like 'pattern' or 'patternProperties' will use Python regular expressions instead. ↩ TYPE: bool

Example configuration

json_payload: str = '''
{
    "type": "object",
    "properties": {
        "question": {
            "type": "string"
        },
        "setup": {
            "type": "string"
        },
        "joke": {
            "type": "string"
        },
        "funniness_score": {
            "type": "number",
            "minimum": 0.0,
            "maximum": 1.0
        }
    },
    "required": ["question", "joke"]
}
'''
config: dict[str, Any] = {"expected_schema": json_payload}
metric: JsonFormatMetric = JsonFormatMetric.from_config(**config)

Fields:

• expected_schema (dict[str, Any])
• check_formats (bool)

Validators:

• _validate_expected_schema → expected_schema

JsonFormatMetric

JsonFormatMetric(**kwargs: dict[str, Any])

              flowchart TD
              evallm.metrics.builtin.JsonFormatMetric[JsonFormatMetric]
              evallm.metrics.base.BaseMetric[BaseMetric]

                              evallm.metrics.base.BaseMetric --> evallm.metrics.builtin.JsonFormatMetric
                


              click evallm.metrics.builtin.JsonFormatMetric href "" "evallm.metrics.builtin.JsonFormatMetric"
              click evallm.metrics.base.BaseMetric href "" "evallm.metrics.base.BaseMetric"
            

Use mouse to pan and zoom

Metric for evaluating whether text matches a given JSON format.

This metric evaluates whether the input text adheres to a structured output format, defined by the expected_schema.

PARAMETER	DESCRIPTION
expected_schema	The expected JSON schema as a dictionary that conforms to the JSON Schema draft-2020-12 specification. TYPE: dict[str, Any]
check_formats	If True, format annotations are also validated. Defaults to False. TYPE: bool

Referenced by:

• API Reference evallm metrics  builtin Classes  JsonFormatConfig

METHOD	DESCRIPTION
__init_subclass__	Called when a class inherits from BaseMetric.
__eq__	Check if this metric is equal to another metric.
__iter__	Iterate over the subtree rooted at this metric in breadth-first order.
__contains__	Check if a metric is contained in the subtree rooted at this metric.
__repr__	String representation of the subtree, rooted in this metric instance.
result	Helper method for wrapping the value of a metric's evaluation result in a
descendants	Yield nodes in the subtree rooted at self (cycle-safe).
from_config	Factory method to create a metric instance from kwargs or JSON payload.
evaluate	Evaluate whether the input text matches the expected JSON format.

ATTRIBUTE	DESCRIPTION
value_type	The metric class' parametrized type, binding the generic type parameter T. TYPE: object
config_schema	A schema defining the expected runtime configuration parameters for this metric. TYPE: type[BaseModel]
parent	Get the parent of this metric. TYPE: CompositeMetric | None
result_key	The key to use for this metric's value in the evaluation result dictionary. TYPE: str
name	The name of this metric's class. TYPE: str
is_composite	Check if this metric is a composite metric (i.e. can have child metrics). TYPE: bool
is_root	Check if this metric is a root metric (i.e. has no parent). TYPE: bool
root	Get the root metric of the tree this metric belongs to. TYPE: BaseMetric[SerializableValueType]
depth	Get the depth of this metric in the tree (i.e. the number of edges from this TYPE: int
children	Get an iterator over the children of this metric. TYPE: Iterator[BaseMetric[SerializableValueType]]
config	Get the runtime configuration of this metric instance as a dictionary. TYPE: dict[str, Any]

Source code in src/evallm/metrics/base.py

def __init__(self, **kwargs: dict[str, Any]):
    """ """
    self.parent: BaseMetric[SerializableValueType] | None = None
    self._config: dict[str, Any] = {}  # default

    # raise error when trying to pass runtime params without a config schema
    if (schema := type(self).config_schema).model_json_schema().get(
        "properties", {}
    ) == {}:
        if kwargs:
            raise ValueError(f"{type(self)._name} takes no runtime parameters.")
    else:  # validate runtime params against config_schema
        try:
            model: BaseModel = schema.model_validate(kwargs)
            self._config = model.model_dump()
        except ValidationError:
            raise

Attributes

value_type class-attribute

value_type: object

The metric class' parametrized type, binding the generic type parameter T.

Referenced by:

• API Reference evallm metrics
  •  base Classes  BaseMetric Functions  __init_subclass__
  •  builtin Classes
    •  CountsMetric Functions  __init_subclass__
    •  JsonFormatMetric Functions  __init_subclass__
  •  composite Classes  CompositeMetric Functions  __init_subclass__
  •  metrics Extension Path for Custom Metrics
  •  types Functions  check_evaluate_signature

config_schema class-attribute

config_schema: type[BaseModel]

A schema defining the expected runtime configuration parameters for this metric.

If the metric class does not explicitly define a config schema, this defaults to an empty pydantic model (see DefaultSchema) that takes no parameters.

Info

A class' config schema is always configured such that:

• all declared fields must be provided unless they are declared as optional
• no additional fields beyond those declared in the schema are allowed (see also pydantic.ConfigDict.extra)
• the schema is frozen/immutable once defined (see also pydantic.ConfigDict.frozen)
• configuration parameters provided at runtime are strictly validated against the schema, raising a ValidationError if the provided parameters violate it (see also pydantic Conversion Table)

Referenced by:

• API Reference evallm
  • application
    •  api Functions  metric
    •  application
      • Classes  MetricSpec Attributes  config
      • Functions  metric
    •  models Classes  MetricSpec Attributes  config
  • metrics
    •  base Classes  BaseMetric
    •  config Functions  config_schema

parent property writable

parent: CompositeMetric | None

Get the parent of this metric.

RETURNS	DESCRIPTION
CompositeMetric | None	CompositeMetric | None: The parent metric, or None if this metric has no parent, i.e. is a leaf node.

result_key property writable

result_key: str

The key to use for this metric's value in the evaluation result dictionary.

By default, this is the same as the metric's name, but it can be overridden on a per-instance basis. This is useful when multiple instances of the same metric class are used in a single evaluation request with different runtime parameters and therefore need distinct keys in the output.

RETURNS	DESCRIPTION
result_key	The key of this metric instance's value in the evaluation result dictionary. Defaults to the metric's name. TYPE: str

Referenced by:

• API Reference evallm
  • metrics
    •  base Classes  BaseMetric
      • Attributes  name
      • Functions
        •  evaluate
        •  result
    •  builtin Classes
      •  CountsMetric
        • Attributes  name
        • Functions
          •  evaluate
          •  result
      •  JsonFormatMetric
        • Attributes  name
        • Functions
          •  evaluate
          •  result
    •  composite Classes  CompositeMetric
      • Attributes  name
      • Functions
        •  evaluate
        •  result
    •  metrics Extension Path for Custom Metrics
    •  types Functions  check_evaluate_signature
  • application
    •  api Functions
      •  count_characters
      •  count_encodings
      •  count_tokens
      •  metric
    •  application
      • Classes  EvaluationResult Attributes  results
      • Functions
        •  count_characters
        •  count_encodings
        •  count_tokens
        •  metric
    •  models Classes  EvaluationResult Attributes  results

name property

name: str

The name of this metric's class.

This is a class-, not an instance-level property that's used for identifying this metric in the catalog. It also serves as the default result_key for metric instances unless explicitly overridden.

Metric class registration and name updates

When a metric class is registered in the catalog under a custom name using @register_metric(name=...), the name-property of any pre-existing instances of that class will be updated to reflect the new name in the catalog.

RETURNS	DESCRIPTION
name	The name of this metric. Defaults to the class name if not overwritten by the @register_metric decorator at definition time. TYPE: str

Referenced by:

• API Reference evallm
  • metrics
    •  base Classes  BaseMetric Attributes  result_key
    •  builtin Classes
      •  CountsMetric Attributes  result_key
      •  JsonFormatMetric Attributes  result_key
    •  composite Classes  CompositeMetric Attributes  result_key
  • application
    •  application Classes
      •  EvaluationResult Attributes  results
      •  MetricSpec Attributes  name
    •  models Classes
      •  EvaluationResult Attributes  results
      •  MetricSpec Attributes  name

is_composite property

is_composite: bool

Check if this metric is a composite metric (i.e. can have child metrics).

RETURNS	DESCRIPTION
bool	True if this metric is a composite metric, False otherwise. TYPE: bool

is_root property

is_root: bool

Check if this metric is a root metric (i.e. has no parent).

RETURNS	DESCRIPTION
bool	True if this metric has no parent, False otherwise. TYPE: bool

root property

root: BaseMetric[SerializableValueType]

Get the root metric of the tree this metric belongs to.

RETURNS	DESCRIPTION
BaseMetric	The root metric of the tree this metric belongs to. TYPE: BaseMetric[SerializableValueType]

depth property

depth: int

Get the depth of this metric in the tree (i.e. the number of edges from this metric to the root).

RETURNS	DESCRIPTION
depth	The depth of this metric in the tree. Root metrics have depth 0, their children have depth 1, and so on. TYPE: int

children property

children: Iterator[BaseMetric[SerializableValueType]]

Get an iterator over the children of this metric.

YIELDS	DESCRIPTION
children	An iterator over the child metrics (i.e. direct descendants) of this metric. If this metric is a leaf node, this will be an empty iterator. TYPE:: BaseMetric

config property

config: dict[str, Any]

Get the runtime configuration of this metric instance as a dictionary.

RETURNS	DESCRIPTION
config	A dictionary containing the runtime configuration of this metric instance, validated against the metric's schema. TYPE: dict[str, Any]

Referenced by:

• API Reference evallm metrics  config Functions  config_schema

Functions

__init_subclass__

__init_subclass__(**kwargs)

Called when a class inherits from BaseMetric.

Sets the default name and config schema of the metric class and validates & stores the parametrized value type of the metric class (i.e. the T in BaseMetric[T]) as the value_type class variable and ensures that evaluate is implemented with the correct signature.

RAISES	DESCRIPTION
TypeError	If the metric class does not implement the evaluate method with the correct signature.
UnserializableValueTypeError	If the metric class's parametrized value type is not an instance of SerializableValueType.

Source code in src/evallm/metrics/base.py

def __init_subclass__(cls, **kwargs):
    """Called when a class inherits from `BaseMetric`.

    Sets the default name and config schema of the metric class and validates
    & stores the parametrized value type of the metric class (i.e.
    the `T` in `BaseMetric[T]`) as the
    [`value_type`][evallm.metrics.base.BaseMetric.value_type] class variable
    and ensures that `evaluate` is implemented with the correct signature.

    Raises:
        TypeError: If the metric class does not implement the `evaluate` method
            with the correct signature.
        UnserializableValueTypeError: If the metric class's parametrized value type
            is not an instance of
            [`SerializableValueType`][evallm.metrics.types.SerializableValueType].
    """
    super().__init_subclass__(**kwargs)

    # Set default class name if not already set
    if getattr(cls, "_name", None) is None:
        cls._name = cls.__name__

    # Set default config schema if not already set
    if getattr(cls, "config_schema", None) is None:
        cls.config_schema = DefaultSchema

    # Set the metric's parametrized value type as a class variable
    for base in get_original_bases(cls):
        if get_origin(base) is BaseMetric:
            value_type = get_args(base)[0]
            if not is_serializable_type(value_type):
                raise UnserializableValueTypeError(
                    f"Metric class '{cls.__name__}' has invalid value type annotation "
                    + f"{value_type!r}. Expected a SerializableValueType."
                )
            cls.value_type = value_type
            break

    try:  # Ensure evaluate method is implemented & has correct signature
        _ = check_evaluate_signature(cls)
    except TypeError:
        raise

__eq__

__eq__(other: object) -> bool

Check if this metric is equal to another metric.

Two metrics are considered equal iff they are the same instance.

PARAMETER	DESCRIPTION
other	The other metric to compare against. TYPE: object

RETURNS	DESCRIPTION
bool	True if this metric is the same instance as the other metric, False otherwise. TYPE: bool

Source code in src/evallm/metrics/base.py

def __eq__(self, other: object) -> bool:
    """Check if this metric is equal to another metric.

    Two metrics are considered equal iff they are the same instance.

    Args:
        other (object): The other metric to compare against.

    Returns:
        bool: `True` if this metric is the same instance as the other metric,
            `False` otherwise.
    """
    return self is other

__iter__

__iter__() -> Iterator[BaseMetric[SerializableValueType]]

Iterate over the subtree rooted at this metric in breadth-first order.

YIELDS	DESCRIPTION
metric	An iterator over the subtree rooted at this metric in breadth-first order, excluding this metric itself. TYPE:: BaseMetric

Source code in src/evallm/metrics/base.py

def __iter__(self) -> Iterator[BaseMetric[SerializableValueType]]:
    """Iterate over the subtree rooted at this metric in breadth-first order.

    Yields:
        metric (BaseMetric): An iterator over the subtree rooted at this metric in
            breadth-first order, excluding this metric itself.
    """
    yield from self.descendants(include_self=False, order="bfs")

__contains__

__contains__(metric: str | BaseMetric[SerializableValueType]) -> bool

Check if a metric is contained in the subtree rooted at this metric.

Note

Passing a metric instance checks for identity (i.e. whether exactly that instance is contained in this subtree), whereas passing a metric's name checks for membership (i.e. whether any metric with that name is contained in this subtree).

Example

leaf1: MetricA = MetricA()  # name='MetricA'
leaf2: MetricA = MetricA()  # name='MetricA'
composite: CompositeMetric = CompositeMetric()
composite.add(leaf1)

Identity-based membership check:

>>> leaf1 in composite
True
>>> leaf2 in composite
False

Name-based membership check:

>>> "MetricA" in composite
True
>>> "NonExistentMetric" in composite
False

PARAMETER	DESCRIPTION
metric	The name of the metric to check for, or the metric instance itself. TYPE: str | BaseMetric

RETURNS	DESCRIPTION
bool	True if the given metric is a descendant of this metric (excluding this metric itself), False otherwise. TYPE: bool

Source code in src/evallm/metrics/base.py

def __contains__(self, metric: str | BaseMetric[SerializableValueType]) -> bool:
    """Check if a metric is contained in the subtree rooted at this metric.

    Note:
        Passing a metric **instance** checks for **identity** (i.e. whether exactly
        that instance is contained in this subtree), whereas passing a metric's
        **name** checks for **membership** (i.e. whether any metric with that name
        is contained in this subtree).

    Example:
        ```python
        leaf1: MetricA = MetricA()  # name='MetricA'
        leaf2: MetricA = MetricA()  # name='MetricA'
        composite: CompositeMetric = CompositeMetric()
        composite.add(leaf1)
        ```
        Identity-based membership check:
        ```pycon
        >>> leaf1 in composite
        True
        >>> leaf2 in composite
        False
        ```
        Name-based membership check:
        ```pycon
        >>> "MetricA" in composite
        True
        >>> "NonExistentMetric" in composite
        False
        ```

    Args:
        metric (str | BaseMetric): The name of the metric to check for, or the
            metric instance itself.

    Returns:
        bool: `True` if the given metric is a descendant of this metric
            (excluding this metric itself), `False` otherwise.
    """
    if isinstance(metric, str):
        return any(node.name == metric for node in self.descendants())
    return any(node is metric for node in self.descendants())

__repr__

__repr__() -> str

String representation of the subtree, rooted in this metric instance.

RETURNS	DESCRIPTION
representation	A string representation of the metric instance, including its name and class. TYPE: str

Source code in src/evallm/metrics/base.py

def __repr__(self) -> str:
    """String representation of the subtree, rooted in this metric instance.

    Returns:
        representation (str): A string representation of the metric instance,
            including its name and class.
    """
    return self._repr_tree()

result

result(value: T) -> dict[str, T]

Helper method for wrapping the value of a metric's evaluation result in a dictionary to conform to the expected return type of evaluate.

Example

class CustomMetric(BaseMetric[list[int]]):
    def evaluate(self, text: str) -> dict[str, list[int]]:
        # Perform some evaluation logic and compute the metric's value
        res: list[int] = [len(text.split())]

        # Wrap the result in a dictionary keyed by the metric's result_key
        return self.result(res)

PARAMETER	DESCRIPTION
value	The result of computing the metric, where T is the metric's parametrized value type. TYPE: T

RETURNS	DESCRIPTION
result	A dictionary containing the given value, keyed by the metric's result_key. TYPE: dict[str, T]

RAISES	DESCRIPTION
TypeError	If the given value is not of the expected type T or is not JSON-serializable.

Referenced by:

• API Reference evallm metrics
  •  base Classes  BaseMetric
  •  metrics Extension Path for Custom Metrics 2. Implementing the evaluate method

Source code in src/evallm/metrics/base.py

def result(self, value: T) -> dict[str, T]:  # pragma: no cover
    """Helper method for wrapping the value of a metric's evaluation result in a
    dictionary to conform to the expected return type of `evaluate`.

    Example:
        ```python
        class CustomMetric(BaseMetric[list[int]]):
            def evaluate(self, text: str) -> dict[str, list[int]]:
                # Perform some evaluation logic and compute the metric's value
                res: list[int] = [len(text.split())]

                # Wrap the result in a dictionary keyed by the metric's result_key
                return self.result(res)
        ```

    Args:
        value (T): The result of computing the metric, where `T` is the metric's
            parametrized value type.

    Returns:
        result (dict[str, T]): A dictionary containing the given value, keyed by
            the metric's [`result_key`][..result_key].

    Raises:
        TypeError: If the given value is not of the expected type `T` or is not
            JSON-serializable.
    """
    if not matches_annotation(value, self.value_type):
        raise TypeError(
            f"Value {value!r} of type {type(value)!r} is not of expected type "
            + f"{self.value_type!r} for metric '{self.name}'."
        )
    return {self.result_key: value}

descendants

descendants(
    *, include_self: bool = False, order: Literal["dfs", "bfs"] = "dfs"
) -> Iterator[BaseMetric[SerializableValueType]]

Yield nodes in the subtree rooted at self (cycle-safe).

PARAMETER	DESCRIPTION
include_self	Whether to include self in the yielded nodes. TYPE: bool DEFAULT: False
order	The order to traverse the tree. "dfs" for depth-first search, "bfs" for breadth-first search. TYPE: 'dfs' | 'bfs' DEFAULT: 'dfs'

YIELDS	DESCRIPTION
descendants	An iterator over the descendant nodes of this metric, in the specified traversal order. TYPE:: Iterator[BaseMetric]

Source code in src/evallm/metrics/base.py

def descendants(
    self,
    *,
    include_self: bool = False,
    order: Literal["dfs", "bfs"] = "dfs",
) -> Iterator[BaseMetric[SerializableValueType]]:
    """Yield nodes in the subtree rooted at self (cycle-safe).

    Args:
        include_self (bool): Whether to include `self` in the yielded nodes.
        order ("dfs" | "bfs"): The order to traverse the tree. "dfs" for depth-first
            search, "bfs" for breadth-first search.

    Yields:
        descendants (Iterator[BaseMetric]): An iterator over the descendant nodes of this metric,
            in the specified traversal order.
    """
    visited: set[int] = set()

    # Populate the frontier with the initial node(s) based on traversal order
    frontier: deque[BaseMetric[SerializableValueType]] = deque()
    if include_self:
        frontier.append(cast(BaseMetric[SerializableValueType], self))
    else:  # keep order of children consistent across DFS/BFS
        if order == "dfs":
            frontier.extend(reversed(list(self.children)))
        else:
            frontier.extend(list(self.children))

    while frontier:
        # Get the next node to visit, depending on traversal order
        node: BaseMetric[SerializableValueType] = (
            frontier.pop() if order == "dfs" else frontier.popleft()
        )

        # Skip visited nodes to prevent infinite loops
        if (node_id := id(node)) in visited:
            continue
        visited.add(node_id)

        yield node

        # Expand neighbors
        if order == "dfs":
            frontier.extend(  # reverse for correct DFS order
                reversed(list(node.children))
            )
        else:
            frontier.extend(list(node.children))

from_config classmethod

from_config(*, json_payload: str | None = None, **kwargs: Any) -> BaseMetric[T]

Factory method to create a metric instance from kwargs or JSON payload.

Info

This method can be used to create any metric instance, so long as the provided config parameters conform to the metric's schema. If the metric class doesn't define a config schema (the default), calling this method without kwargs is equivalent to calling the metric class constructor directly. The expected parameters depend on the metric's declared config_schema (see also @config_schema decorator) and the provided parameters must strictly conform to it.

PARAMETER	DESCRIPTION
json_payload	A JSON string containing the runtime parameters for this metric instance. TYPE: str | None DEFAULT: None

PARAMETER	DESCRIPTION
**kwargs	The runtime parameters to use for this metric instance. TYPE: Any

RETURNS	DESCRIPTION
metric	An instance of this metric class created using the provided configuration parameters. TYPE: BaseMetric[T]

RAISES	DESCRIPTION
TypeError	If json_payload is not a string or None.
ValueError	If trying to pass runtime parameters to a metric class that doesn't define a config_schema, or trying to instantiate a metric class with invalid runtime parameters (i.e. missing required parameters, extra unknown parameters, or parameters of the wrong type), or passing an invalid JSON payload, or passing both json_payload and kwargs at the same time.

Referenced by:

• API Reference evallm
  • application
    •  application Classes  MetricSpec Attributes  config
    •  models Classes  MetricSpec Attributes  config
  • metrics  base Classes  DefaultSchema

Source code in src/evallm/metrics/base.py

@classmethod
def from_config(
    cls, *, json_payload: str | None = None, **kwargs: Any
) -> BaseMetric[T]:
    """Factory method to create a metric instance from kwargs or JSON payload.

    !!! info
        This method can be used to create any metric instance, so long as the
        provided config parameters conform to the metric's schema. If the metric
        class doesn't define a config schema (the default), calling this method
        without kwargs is equivalent to calling the metric class constructor
        directly.
        The expected parameters depend on the metric's declared config_schema
        (see also [`@config_schema` decorator][evallm.metrics.config.config_schema])
        and the provided parameters must strictly conform to it.

    Args:
        json_payload (str | None): A JSON string containing the runtime parameters
            for this metric instance.

    Keyword Args:
        **kwargs (Any): The runtime parameters to use for this metric instance.

    Returns:
        metric (BaseMetric[T]): An instance of this metric class created using the
            provided configuration parameters.

    Raises:
        TypeError: If `json_payload` is not a string or `None`.
        ValueError: If

            - trying to pass runtime parameters to a metric class that doesn't
                define a `config_schema`, or
            - trying to instantiate a metric class with invalid runtime parameters
                (i.e. missing required parameters, extra unknown parameters, or
                parameters of the wrong type), or
            - passing an invalid JSON payload, or
            - passing both `json_payload` and `kwargs` at the same time.
    """
    runtime_params: dict[str, Any]
    if json_payload is not None:
        if kwargs:
            raise ValueError("Cannot mix json_payload with kwargs.")
        if not isinstance(json_payload, str):
            raise TypeError("json_payload must be a string or None.")
        try:
            runtime_params = orjson.loads(json_payload)
        except orjson.JSONDecodeError:
            raise ValueError(
                "Invalid json_payload: not a valid JSON string."
            ) from None
    else:
        runtime_params = kwargs

    return cls(**runtime_params)

evaluate

evaluate(text: str) -> dict[str, dict[str, bool]]

Evaluate whether the input text matches the expected JSON format.

This method attempts to parse the input text as JSON and checks whether it contains the expected fields as specified in the configuration.

PARAMETER	DESCRIPTION
text	The input text to evaluate. TYPE: str

RETURNS	DESCRIPTION
dict[str, dict[str, bool]]	dict[str, dict[str, bool]]: A dictionary mapping this metric instance's result_key to the evaluation results, where each expected field is mapped to a boolean, indicating whether that field is valid. Note that all fields will be False if the input text is not valid JSON.

Source code in src/evallm/metrics/builtin/json_format.py

def evaluate(self, text: str) -> dict[str, dict[str, bool]]:
    """Evaluate whether the input text matches the expected JSON format.

    This method attempts to parse the input text as JSON and checks whether it
    contains the expected fields as specified in the configuration.

    Args:
        text (str): The input text to evaluate.

    Returns:
        dict[str, dict[str, bool]]: A dictionary mapping this metric instance's
            [`result_key`][evallm.metrics.base.BaseMetric.result_key] to the
            evaluation results, where each expected field is mapped to a boolean,
            indicating whether that field is valid. Note that all fields will be
            `False` if the input text is not valid JSON.
    """
    expected_schema: dict[str, Any] = self.config["expected_schema"]
    res: dict[str, bool] = {field: False for field in expected_schema["properties"]}

    # handle empty inputs
    if text is None or text.strip() == "":
        return self.result(res)

    try:  # parse input text as JSON
        parsed: dict[str, Any] = orjson.loads(text)
    except orjson.JSONDecodeError:
        return self.result(res)

    # validate fields according to the expected schema
    for field, field_schema in expected_schema["properties"].items():
        if self.config["check_formats"]:
            validator: Draft202012Validator = Draft202012Validator(
                schema=field_schema,
                format_checker=Draft202012Validator.FORMAT_CHECKER,
            )
        else:
            validator = Draft202012Validator(schema=field_schema)

        field_value: Any = parsed.get(field)
        required: bool = field in expected_schema.get("required", [])
        if field in parsed:
            if not required:
                if field_value is None:  # optional field with null value -> valid
                    res[field] = True
                else:  # optional field is present -> check validity
                    res[field] = validator.is_valid(parsed[field])
            else:  # check validity of required field
                res[field] = validator.is_valid(parsed[field])
        elif not required:  # missing optional fields are valid
            res[field] = True

    return self.result(res)