config

Configuration schema & runtime parameters for metrics

Use mouse to pan and zoom

TYPE ALIAS	DESCRIPTION
`PlainDictSchema`	A plain dictionary mapping configuration parameters to their types.
`PydanticSchema`	A `pydantic.BaseModel` subclass.
`TypedDictSchema`	A `TypedDict` subclass defining configuration parameters and their types.

CLASS	DESCRIPTION
`InvalidConfigSchema`	Exception raised when an invalid schema is provided to the `@config_schema`

FUNCTION	DESCRIPTION
`config_schema`	Decorator for defining the expected schema of a metric's runtime parameters.

Type Aliases

PlainDictSchema

PlainDictSchema = dict[str, object]

A plain dictionary mapping configuration parameters to their types.

All parameters defined in a plain dictionary schema are considered required.

Info

The dictionary's values must be type objects that serve as runtime annotations. Make sure you're passing the expected types rather than the parameter values themselves.

TypedDictSchema

TypedDictSchema = type[TypedDictType]

A TypedDict subclass defining configuration parameters and their types.

PydanticSchema

PydanticSchema = type[BaseModel]

A pydantic.BaseModel subclass. defining configuration parameters, their types, and optionally validation logic and default values.

Classes

InvalidConfigSchema


              flowchart TD
              evallm.metrics.config.InvalidConfigSchema[InvalidConfigSchema]

              

              click evallm.metrics.config.InvalidConfigSchema href "" "evallm.metrics.config.InvalidConfigSchema"

Use mouse to pan and zoom

Exception raised when an invalid schema is provided to the @config_schema decorator.

Functions

config_schema

config_schema(
    *, schema: PlainDictSchema | TypedDictSchema | PydanticSchema
) -> Callable[[type[BaseMetric[Any]]], type[BaseMetric[Any]]]

Decorator for defining the expected schema of a metric's runtime parameters.

The specified schema is converted to a pydantic model and set as the value of the decorated classes' config_schema class variable. This can subsequently be used for validating runtime parameters that are provided on instantiation. (see also: BaseMetric.config)

Example

Specifying a config schema

...as a plain dictionary:

schema: PlainDictSchema = {"threshold": int}  # (1)!

...as a TypedDict subclass:

from typing import TypedDict, NotRequired


class CustomConfig(TypedDict):
    # required parameter
    keywords: list[str]

    # optional parameter with default value of None
    min_keyword_counts: NotRequired[dict[str, int]]


schema: TypedDictSchema = CustomConfig  # (2)!

...as a pydantic.BaseModel subclass:

from pydantic import BaseModel, Field, field_validator


class ConfigWithValidation(BaseModel):
    # required parameter with validation
    keywords: list[str] = Field(..., min_length=1)

    # optional parameter with validation
    threshold: float | None = Field(default=None, ge=0, le=1)

    @field_validator("keywords")
    @classmethod
    def validate_keywords(cls, keywords: list[str]) -> list[str]:
        if not all(k.startswith("foo") for k in keywords):
            raise ValueError("All keywords must start with 'foo'.")
        return keywords


schema: PydanticSchema = ConfigWithValidation  # (3)!

Decorating a metric class with a config schema

@register_metric
@config_schema(schema=schema)  # (4)!
class KeywordRatios(BaseMetric[dict[str, float]]):
    def evaluate(self, text: str) -> dict[str, float]:
        counts: dict[str, int] = {}
        for keyword in self.config.get("keywords"):  # (5)!
            counts[keyword] = text.count(keyword)

        ratios: dict[str, float] = {}
        for keyword, count in counts.items():
            kw_ratio: float = counts[keyword] / sum(counts.values())
            if (threshold := self.config.get("threshold")) is not None:  # (6)!
                if kw_ratio < threshold:
                    kw_ratio = 0.0
            ratios[keyword] = kw_ratio

        return self.result(ratios)

A plain dictionary is the simplest way to define a schema, it will however assume all parameters as required.
A TypedDict subclass allows you to specify optional parameters using typing.NotRequired and provides better readability.
A pydantic.BaseModel subclass provides the most flexibility, allowing you to define complex validation logic and default values. The defined model will be used as-is.
The @config_schema decorator should be applied before the @register_metric decorator.
You can access validated configuration values in the metric's evaluate(...) method via self.config.get(<parameter_name>).
Remember to appropriately handle parameters that you defined as optional in the schema!

PARAMETER	DESCRIPTION
`schema`	The schema to enforce for the decorated metric class. Must be one of the following: A plain dictionary mapping parameter names to their types A `TypedDict` subclass A `pydantic.BaseModel` subclass TYPE: `PlainDictSchema \| TypedDictSchema \| PydanticSchema`

RETURNS	DESCRIPTION
`metric_cls`	The decorated metric class. TYPE: `Callable[[type[BaseMetric[Any]]], type[BaseMetric[Any]]]`

Referenced by:

API Reference evallm metrics

Source code in src/evallm/metrics/config.py

def config_schema(
    *, schema: PlainDictSchema | TypedDictSchema | PydanticSchema
) -> Callable[[type[BaseMetric[Any]]], type[BaseMetric[Any]]]:
    """Decorator for defining the expected schema of a metric's runtime parameters.

    The specified schema is converted to a pydantic model and set as the value of the
    decorated classes'
    [`config_schema`][evallm.metrics.base.BaseMetric.config_schema] class
    variable. This can subsequently be used for validating runtime parameters that are
    provided on instantiation. (see also:
    [`BaseMetric.config`][evallm.metrics.base.BaseMetric.config])

    ??? example annotate

        **Specifying a config schema**

        ...as a *plain dictionary*:
        ```python
        schema: PlainDictSchema = {"threshold": int}  # (1)!
        ```

        ...as a *`TypedDict` subclass*:
        ```python
        from typing import TypedDict, NotRequired


        class CustomConfig(TypedDict):
            # required parameter
            keywords: list[str]

            # optional parameter with default value of None
            min_keyword_counts: NotRequired[dict[str, int]]


        schema: TypedDictSchema = CustomConfig  # (2)!
        ```

        ...as a *`pydantic.BaseModel` subclass*:
        ```python
        from pydantic import BaseModel, Field, field_validator


        class ConfigWithValidation(BaseModel):
            # required parameter with validation
            keywords: list[str] = Field(..., min_length=1)

            # optional parameter with validation
            threshold: float | None = Field(default=None, ge=0, le=1)

            @field_validator("keywords")
            @classmethod
            def validate_keywords(cls, keywords: list[str]) -> list[str]:
                if not all(k.startswith("foo") for k in keywords):
                    raise ValueError("All keywords must start with 'foo'.")
                return keywords


        schema: PydanticSchema = ConfigWithValidation  # (3)!
        ```

        **Decorating a metric class with a config schema**

        ```python
        @register_metric
        @config_schema(schema=schema)  # (4)!
        class KeywordRatios(BaseMetric[dict[str, float]]):
            def evaluate(self, text: str) -> dict[str, float]:
                counts: dict[str, int] = {}
                for keyword in self.config.get("keywords"):  # (5)!
                    counts[keyword] = text.count(keyword)

                ratios: dict[str, float] = {}
                for keyword, count in counts.items():
                    kw_ratio: float = counts[keyword] / sum(counts.values())
                    if (threshold := self.config.get("threshold")) is not None:  # (6)!
                        if kw_ratio < threshold:
                            kw_ratio = 0.0
                    ratios[keyword] = kw_ratio

                return self.result(ratios)
        ```

    1.  A plain dictionary is the simplest way to define a schema, it will however
        assume all parameters as required.
    2.  A `TypedDict` subclass allows you to specify optional parameters using
        `typing.NotRequired` and provides better readability.
    3.  A [`pydantic.BaseModel`](https://docs.pydantic.dev/2.12/api/base_model/)
        subclass provides the most flexibility, allowing you to define complex
        validation logic and default values. The defined model will be used as-is.
    4.  The `@config_schema` decorator should be applied *before* the
        `@register_metric` decorator.
    5.  You can access validated configuration *values* in the metric's `evaluate(...)`
        method via `self.config.get(<parameter_name>)`.
    6.  Remember to appropriately handle parameters that you defined as optional in the
        schema!

    Args:
        schema (PlainDictSchema | TypedDictSchema | PydanticSchema): The schema to
            enforce for the decorated metric class. Must be one of the following:

            - A plain dictionary mapping parameter names to their types
            - A `TypedDict` subclass
            - A `pydantic.BaseModel` subclass

    Returns:
        metric_cls (Callable[[type[BaseMetric[Any]]], type[BaseMetric[Any]]]): The decorated metric class.
    """

    def decorate(
        metric_cls: type[BaseMetric[Any]],
    ) -> type[BaseMetric[Any]]:
        """Decorate a metric class by setting its `_config_schema` class variable to
        the provided schema.

        Args:
            metric_cls (type[BaseMetric[Any]]): The metric class to
                decorate.

        Returns:
            type[BaseMetric[Any]]: The decorated metric class with
                its `_config_schema` attribute set to the provided schema as a
                pydantic model.

        Raises:
            TypeError: If the provided `schema` argument is not one of:

                - a plain dictionary, mapping parameter names to their types, or
                - a `TypedDict` subclass, or
                - a `pydantic.BaseModel` subclass.
            InvalidConfigSchema: If

                - the field name is not a non-empty string, or
                - the annotation is not a valid type hint, or
                - the annotation can't be resolved to a pydantic field type.
        """
        if isinstance(schema, type) and issubclass(schema, BaseModel):
            schema_model: PydanticSchema = _ensure_no_extras_and_frozen(schema)
            metric_cls.config_schema = schema_model
            return metric_cls
        elif _is_typeddict_type(schema) or isinstance(schema, dict):  # pragma: no cover
            try:  # convert plain dict or TypedDict schema to a pydantic model
                schema_model = _convert_to_pydantic_model(metric_cls, schema)
                metric_cls.config_schema = _ensure_no_extras_and_frozen(schema_model)
                return metric_cls
            except InvalidConfigSchema:
                raise
        else:  # pragma: no cover
            raise TypeError(
                "Schema must be one of: a plain dictionary, a TypedDict subclass, or "
                + "a pydantic BaseModel subclass."
            )

    return decorate