models

Use mouse to pan and zoom

Canonical internal data contracts as DTOs

DTOs are implemented as immutable BaseModels that perform structural validation only. While this allows them to stay decoupled from other layers, it also means that no semantic/ stateful validation is performed.

CLASS	DESCRIPTION
`BaseDTO`	Abstract base class for all DTOs in the application layer.
`MetricSpec`	DTO specifying a metric and its configuration
`BaseDatasetSpec`	Abstract base class for dataset specifications.
`JsonlDatasetSpec`	DTO specifying a structured JSONL dataset to evaluate
`EvaluationRequest`	DTO specifying which configuration of metrics to evaluate a given dataset on
`DatasetRecord`	DTO representing a single record to evaluate in a dataset
`EvaluationResult`	DTO representing the evaluated metrics' results for a single record

Classes

BaseDTO `pydantic-model`

Abstract base class for all DTOs in the application layer.

Config:

extra: forbid
frozen: True
json_schema_extra: make_crossref_resolver(docs_base_url='https://psandhaas.github.io/evaLLM/reference')
json_schema_serialization_defaults_required: True
revalidate_instances: subclass-instances
ser_json_inf_nan: constants
use_attribute_docstrings: True

Functions

create `classmethod`

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
`obj`	The DTO's fields as a native Python object (e.g. a dict). TYPE: `Any \| None` DEFAULT: `None`
`json_payload`	The DTO's fields as a JSON string- or bytes-payload. TYPE: `str \| bytes \| bytearray \| None` DEFAULT: `None`
`**kwargs`	Keyword arguments corresponding to the DTO's fields. TYPE: `Any` DEFAULT: `{}`

RETURNS	DESCRIPTION
`Self`	An instance of the DTO created from the provided input. TYPE: `Self`

RAISES	DESCRIPTION
`ValueError`	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

MetricSpec `pydantic-model`

DTO specifying a metric and its configuration

Fields:

name (str)
config (dict[str, JsonValue] | None)
result_key (str | None)

Attributes

name `pydantic-field`

name: str

Non-empty name under which a metric is registered.

config `pydantic-field`

config: dict[str, JsonValue] | None = None

Runtime configuration for the metric or None (default), if not needed.

result_key `pydantic-field`

result_key: str | None = None

Optional key to use in the evaluation result dictionary for this metric instance's value.

Referenced by:

API Reference evallm application
- application Classes EvaluationResult Attributes results
- models Classes EvaluationResult Attributes results

Functions

resolve_metric

resolve_metric(catalog: type[Catalog]) -> BaseMetric

Resolve the metric specification to a BaseMetric instance.

Performs contextual validation of the DTO by:

looking up the specified name in the provided catalog and
trying to instantiate the metric with the provided config.

If either of these steps fail, an appropriate error is raised.

Metric instance's result_key

The resolved metric instance's result_key attribute is not yet set to the value of this DTO's result_key field. This is because at this point there's no guarantee that the provided result_key is unique among all resolved metrics for a given evaluation request.

PARAMETER	DESCRIPTION
`catalog`	A catalog class to use for looking up the metric. TYPE: `type[Catalog]`

RETURNS	DESCRIPTION
`BaseMetric`	An instance of the specified metric, if resolution and validation succeed. TYPE: `BaseMetric`

RAISES	DESCRIPTION
`TypeError`	If the provided catalog does not implement the `Catalog` interface.
`UnknownMetricError`	If the specified metric name is not registered in the catalog.
`MetricConfigurationError`	If the metric cannot be instantiated from the provided config.

Referenced by:

API Reference evallm application
- api Functions metric
- application Functions metric

Source code in src/evallm/application/models.py

def resolve_metric(self, catalog: type[Catalog]) -> BaseMetric:
    """Resolve the metric specification to a
    [`BaseMetric`][evallm.metrics.base.BaseMetric] instance.

    Performs contextual validation of the DTO by:

    1. looking up the specified [`name`][..name] in the provided
    [`catalog`][evallm.metrics.registry.Catalog] and
    2. trying to instantiate the metric with the provided [`config`][..config].

    If either of these steps fail, an appropriate error is raised.

    !!! note "Metric instance's `result_key`"
        The resolved metric instance's `result_key` attribute is **not** yet set to
        the value of this DTO's `result_key` field. This is because at this point
        there's no guarantee that the provided `result_key` is unique among all
        resolved metrics for a given evaluation request.

    Args:
        catalog (type[Catalog]): A catalog class to use for looking up the metric.

    Returns:
        BaseMetric: An instance of the specified metric, if resolution and
            validation succeed.

    Raises:
        TypeError: If the provided catalog does not implement the
            [`Catalog`][evallm.metrics.registry.Catalog] interface.
        UnknownMetricError: If the specified metric name is not registered in the
            catalog.
        MetricConfigurationError: If the metric cannot be instantiated from the
            provided config.
    """
    if not issubclass(catalog, Catalog):
        raise TypeError(
            f"Provided catalog {catalog.__name__} doesn't implement the Catalog "
            + "interface"
        )
    if self.name not in (available := catalog.available()):
        raise UnknownMetricError(metric_name=self.name, available=available)
    metric_cls: type[BaseMetric] = catalog.get_metric(self.name)
    try:
        return metric_cls.from_config(**cast(dict[str, Any], self.config or {}))
    except ValueError as exc:
        raise MetricConfigurationError(
            metric_name=self.name,
            config=self.config,
            reason=str(exc),
        ) from exc

create `classmethod`

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
`obj`	The DTO's fields as a native Python object (e.g. a dict). TYPE: `Any \| None` DEFAULT: `None`
`json_payload`	The DTO's fields as a JSON string- or bytes-payload. TYPE: `str \| bytes \| bytearray \| None` DEFAULT: `None`
`**kwargs`	Keyword arguments corresponding to the DTO's fields. TYPE: `Any` DEFAULT: `{}`

RETURNS	DESCRIPTION
`Self`	An instance of the DTO created from the provided input. TYPE: `Self`

RAISES	DESCRIPTION
`ValueError`	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

BaseDatasetSpec `pydantic-model`

Abstract base class for dataset specifications.

Fields:

data_format (str)

Attributes

data_format `pydantic-field`

data_format: str

Dataset format discriminator used for dataset spec resolution.

Functions

infer_data_format `classmethod`

infer_data_format(data: Mapping[str, Any]) -> str | None

Infer dataset format from the input-file suffix, if possible.

PARAMETER	DESCRIPTION
`data`	Raw dataset payload. TYPE: `Mapping[str, Any]`

RETURNS	DESCRIPTION
`str \| None`	str \| None: Inferred dataset format, or `None` when no inference is possible.

Source code in src/evallm/application/models.py

@classmethod
def infer_data_format(cls, data: Mapping[str, Any]) -> str | None:
    """Infer dataset format from the input-file suffix, if possible.

    Args:
        data (Mapping[str, Any]): Raw dataset payload.

    Returns:
        str | None: Inferred dataset format, or `None` when no inference is
            possible.
    """
    path_like = data.get("input_file")
    if path_like is None:  # pragma: no cover
        return None
    suffix = Path(str(path_like)).suffix.strip().lower().lstrip(".")
    if not suffix:
        return None
    return suffix

resolve `classmethod`

resolve(data: BaseDatasetSpec | Mapping[str, Any]) -> BaseDatasetSpec

Resolve a raw or already-validated dataset payload to a concrete dataset DTO.

PARAMETER	DESCRIPTION
`data`	Raw or typed dataset payload. TYPE: `BaseDatasetSpec \| Mapping[str, Any]`

RETURNS	DESCRIPTION
`BaseDatasetSpec`	Resolved dataset spec instance. TYPE: `BaseDatasetSpec`

RAISES	DESCRIPTION
`ValueError`	If the dataset format is missing, unsupported, or conflicts with the inferred format.

Source code in src/evallm/application/models.py

@classmethod
def resolve(
    cls,
    data: BaseDatasetSpec | Mapping[str, Any],
) -> BaseDatasetSpec:
    """Resolve a raw or already-validated dataset payload to a concrete dataset DTO.

    Args:
        data (BaseDatasetSpec | Mapping[str, Any]): Raw or typed dataset payload.

    Returns:
        BaseDatasetSpec: Resolved dataset spec instance.

    Raises:
        ValueError: If the dataset format is missing, unsupported, or conflicts
            with the inferred format.
    """
    if isinstance(data, BaseDatasetSpec):  # pragma: no cover
        return data

    payload: dict[str, Any] = dict(data)
    explicit_value = payload.get("data_format")
    explicit: str | None = None
    if explicit_value is not None:
        if (
            not isinstance(explicit_value, str) or not explicit_value.strip()
        ):  # pragma: no cover
            raise ValueError("dataset.data_format must be a non-empty string")
        explicit = explicit_value.strip().lower()

    inferred = cls.infer_data_format(payload)
    if explicit is not None and inferred is not None and explicit != inferred:
        raise ValueError(
            "dataset.data_format does not match inferred format from input_file "
            + f"(explicit={explicit!r}, inferred={inferred!r})"
        )

    resolved_format = explicit or inferred
    if resolved_format is None:
        raise ValueError(
            "Could not infer dataset.data_format from input_file. "
            + "Provide dataset.data_format explicitly."
        )

    if resolved_format not in cls._specs_by_format:
        available = ", ".join(sorted(cls._specs_by_format))
        raise ValueError(
            f"Unsupported dataset format {resolved_format!r}. "
            + f"Available formats: {available}."
        )

    payload["data_format"] = resolved_format
    dataset_cls = cls._specs_by_format[resolved_format]
    return dataset_cls.model_validate(payload)

create `classmethod`

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
`obj`	The DTO's fields as a native Python object (e.g. a dict). TYPE: `Any \| None` DEFAULT: `None`
`json_payload`	The DTO's fields as a JSON string- or bytes-payload. TYPE: `str \| bytes \| bytearray \| None` DEFAULT: `None`
`**kwargs`	Keyword arguments corresponding to the DTO's fields. TYPE: `Any` DEFAULT: `{}`

RETURNS	DESCRIPTION
`Self`	An instance of the DTO created from the provided input. TYPE: `Self`

RAISES	DESCRIPTION
`ValueError`	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

JsonlDatasetSpec `pydantic-model`

DTO specifying a structured JSONL dataset to evaluate

Fields:

data_format (Literal['jsonl'])
input_file (FilePath)
record_id_key (str)
text_key (str)

Validators:

_validate_jsonl_path → input_file
_validate_keys

Attributes

data_format `pydantic-field`

data_format: Literal['jsonl'] = 'jsonl'

Dataset format discriminator.

input_file `pydantic-field`

input_file: FilePath

Path to a local JSONL-file containing the dataset to evaluate.

record_id_key `pydantic-field`

record_id_key: str = '#IDX#'

The dataset field/ column name that contains the unique identifier for each record, specified as a non-empty string. Defaults to "#IDX#", which indicates that record IDs should be generated based on the record's (0-indexed) position in the dataset.

Referenced by:

API Reference evallm application
- application Classes
- models Classes
  - DatasetRecord Attributes record_id
  - EvaluationResult Attributes record_id

text_key `pydantic-field`

text_key: str

The dataset field/ column name that contains the text to evaluate, specified as a non-empty string.

Referenced by:

API Reference evallm application
- application Classes
  - DatasetRecord Attributes text
  - Evaluator
- models Classes DatasetRecord Attributes text

Functions

create `classmethod`

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
`obj`	The DTO's fields as a native Python object (e.g. a dict). TYPE: `Any \| None` DEFAULT: `None`
`json_payload`	The DTO's fields as a JSON string- or bytes-payload. TYPE: `str \| bytes \| bytearray \| None` DEFAULT: `None`
`**kwargs`	Keyword arguments corresponding to the DTO's fields. TYPE: `Any` DEFAULT: `{}`

RETURNS	DESCRIPTION
`Self`	An instance of the DTO created from the provided input. TYPE: `Self`

RAISES	DESCRIPTION
`ValueError`	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

infer_data_format `classmethod`

infer_data_format(data: Mapping[str, Any]) -> str | None

Infer dataset format from the input-file suffix, if possible.

PARAMETER	DESCRIPTION
`data`	Raw dataset payload. TYPE: `Mapping[str, Any]`

RETURNS	DESCRIPTION
`str \| None`	str \| None: Inferred dataset format, or `None` when no inference is possible.

Source code in src/evallm/application/models.py

@classmethod
def infer_data_format(cls, data: Mapping[str, Any]) -> str | None:
    """Infer dataset format from the input-file suffix, if possible.

    Args:
        data (Mapping[str, Any]): Raw dataset payload.

    Returns:
        str | None: Inferred dataset format, or `None` when no inference is
            possible.
    """
    path_like = data.get("input_file")
    if path_like is None:  # pragma: no cover
        return None
    suffix = Path(str(path_like)).suffix.strip().lower().lstrip(".")
    if not suffix:
        return None
    return suffix

resolve `classmethod`

resolve(data: BaseDatasetSpec | Mapping[str, Any]) -> BaseDatasetSpec

Resolve a raw or already-validated dataset payload to a concrete dataset DTO.

PARAMETER	DESCRIPTION
`data`	Raw or typed dataset payload. TYPE: `BaseDatasetSpec \| Mapping[str, Any]`

RETURNS	DESCRIPTION
`BaseDatasetSpec`	Resolved dataset spec instance. TYPE: `BaseDatasetSpec`

RAISES	DESCRIPTION
`ValueError`	If the dataset format is missing, unsupported, or conflicts with the inferred format.

Source code in src/evallm/application/models.py

@classmethod
def resolve(
    cls,
    data: BaseDatasetSpec | Mapping[str, Any],
) -> BaseDatasetSpec:
    """Resolve a raw or already-validated dataset payload to a concrete dataset DTO.

    Args:
        data (BaseDatasetSpec | Mapping[str, Any]): Raw or typed dataset payload.

    Returns:
        BaseDatasetSpec: Resolved dataset spec instance.

    Raises:
        ValueError: If the dataset format is missing, unsupported, or conflicts
            with the inferred format.
    """
    if isinstance(data, BaseDatasetSpec):  # pragma: no cover
        return data

    payload: dict[str, Any] = dict(data)
    explicit_value = payload.get("data_format")
    explicit: str | None = None
    if explicit_value is not None:
        if (
            not isinstance(explicit_value, str) or not explicit_value.strip()
        ):  # pragma: no cover
            raise ValueError("dataset.data_format must be a non-empty string")
        explicit = explicit_value.strip().lower()

    inferred = cls.infer_data_format(payload)
    if explicit is not None and inferred is not None and explicit != inferred:
        raise ValueError(
            "dataset.data_format does not match inferred format from input_file "
            + f"(explicit={explicit!r}, inferred={inferred!r})"
        )

    resolved_format = explicit or inferred
    if resolved_format is None:
        raise ValueError(
            "Could not infer dataset.data_format from input_file. "
            + "Provide dataset.data_format explicitly."
        )

    if resolved_format not in cls._specs_by_format:
        available = ", ".join(sorted(cls._specs_by_format))
        raise ValueError(
            f"Unsupported dataset format {resolved_format!r}. "
            + f"Available formats: {available}."
        )

    payload["data_format"] = resolved_format
    dataset_cls = cls._specs_by_format[resolved_format]
    return dataset_cls.model_validate(payload)

EvaluationRequest `pydantic-model`

DTO specifying which configuration of metrics to evaluate a given dataset on

Fields:

dataset (BaseDatasetSpec)
metrics (list[MetricSpec])

Validators:

_validate_dataset_spec → dataset

Attributes

dataset `pydantic-field`

dataset: BaseDatasetSpec

The dataset to evaluate, specified as a concrete BaseDatasetSpec subtype.

Referenced by:

API Reference evallm application application Classes Evaluator

metrics `pydantic-field`

metrics: list[MetricSpec]

List of one or more metrics to use for evaluation, each specified as a MetricSpec.

Functions

create `classmethod`

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
`obj`	The DTO's fields as a native Python object (e.g. a dict). TYPE: `Any \| None` DEFAULT: `None`
`json_payload`	The DTO's fields as a JSON string- or bytes-payload. TYPE: `str \| bytes \| bytearray \| None` DEFAULT: `None`
`**kwargs`	Keyword arguments corresponding to the DTO's fields. TYPE: `Any` DEFAULT: `{}`

RETURNS	DESCRIPTION
`Self`	An instance of the DTO created from the provided input. TYPE: `Self`

RAISES	DESCRIPTION
`ValueError`	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

DatasetRecord `pydantic-model`

DTO representing a single record to evaluate in a dataset

Fields:

record_id (str)
text (str)
metadata (dict[str, JsonValue] | None)

Validators:

_stringify_text → text

Attributes

record_id `pydantic-field`

record_id: str

The record's unique identifier, specified as a string. The value of this field corresponds to the dataset field specified by JsonlDatasetSpec.record_id_key.

Referenced by:

API Reference evallm application
- application Classes
  - DatasetRecord Attributes text
  - EvaluationResult Attributes record_id
- models Classes
  - DatasetRecord Attributes text
  - EvaluationResult Attributes record_id

text `pydantic-field`

text: str

The record's value to evaluate. The value of this field corresponds to the dataset field specified by JsonlDatasetSpec.text_key, and indexed by record_id.

Type coercion for text field

During validation, None values will be converted to the empty string (i.e. ""), values of JSON-serializable types that aren't strings will be converted to their string representation (e.g. {'key': 123} will become "{'key': 123}"), and string values are kept as-is (e.g. " " will remain " ").

metadata `pydantic-field`

metadata: dict[str, JsonValue] | None = None

Optional dictionary containing any additional fields from the dataset record.

Functions

create `classmethod`

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
`obj`	The DTO's fields as a native Python object (e.g. a dict). TYPE: `Any \| None` DEFAULT: `None`
`json_payload`	The DTO's fields as a JSON string- or bytes-payload. TYPE: `str \| bytes \| bytearray \| None` DEFAULT: `None`
`**kwargs`	Keyword arguments corresponding to the DTO's fields. TYPE: `Any` DEFAULT: `{}`

RETURNS	DESCRIPTION
`Self`	An instance of the DTO created from the provided input. TYPE: `Self`

RAISES	DESCRIPTION
`ValueError`	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

EvaluationResult `pydantic-model`

DTO representing the evaluated metrics' results for a single record

Fields:

record_id (str)
results (dict[str, JsonValue])

Attributes

record_id `pydantic-field`

record_id: str

The unique identifier for the evaluated record, specified as a string. The value of this field corresponds to the dataset field specified by JsonlDatasetSpec.record_id_key and mirrors the value of the corresponding DatasetRecord.record_id.

results `pydantic-field`

results: dict[str, JsonValue]

A dictionary mapping metric result keys to their computed results.

Each key corresponds to the resolved metric instance's result_key, which defaults to the metric's name unless explicitly overridden.

Functions

create `classmethod`

create(
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self

Convenience factory to create DTOs from various input sources.

PARAMETER	DESCRIPTION
`obj`	The DTO's fields as a native Python object (e.g. a dict). TYPE: `Any \| None` DEFAULT: `None`
`json_payload`	The DTO's fields as a JSON string- or bytes-payload. TYPE: `str \| bytes \| bytearray \| None` DEFAULT: `None`
`**kwargs`	Keyword arguments corresponding to the DTO's fields. TYPE: `Any` DEFAULT: `{}`

RETURNS	DESCRIPTION
`Self`	An instance of the DTO created from the provided input. TYPE: `Self`

RAISES	DESCRIPTION
`ValueError`	If 0 or more than 1 of the input sources are provided.

Source code in src/evallm/application/models.py

@classmethod
def create(
    cls,
    obj: Any | None = None,
    *,
    json_payload: str | bytes | bytearray | None = None,
    **kwargs: Any,
) -> Self:
    """Convenience factory to create DTOs from various input sources.

    Args:
        obj (Any | None): The DTO's fields as a native Python object (e.g. a dict).
        json_payload (str | bytes | bytearray | None): The DTO's fields as a JSON
            string- or bytes-payload.
        **kwargs: Keyword arguments corresponding to the DTO's fields.

    Returns:
        Self: An instance of the DTO created from the provided input.

    Raises:
        ValueError: If 0 or more than 1 of the input sources are provided.
    """
    if (
        sum(
            [
                obj is not None,
                json_payload is not None,
                bool(kwargs),
            ]
        )
        != 1
    ):
        raise ValueError(
            "Provide exactly one of: obj, json_payload, or keyword arguments."
        )

    if json_payload is not None:
        return cls.model_validate_json(json_payload)

    if obj is not None:
        return cls.model_validate(obj)

    return cls.model_validate(kwargs)

models

Classes

BaseDTO pydantic-model

Functions

create classmethod

MetricSpec pydantic-model

Attributes

name pydantic-field

config pydantic-field

result_key pydantic-field

Functions

resolve_metric

create classmethod

BaseDatasetSpec pydantic-model

Attributes

data_format pydantic-field

Functions

infer_data_format classmethod

resolve classmethod

create classmethod

JsonlDatasetSpec pydantic-model

Attributes

data_format pydantic-field

input_file pydantic-field

record_id_key pydantic-field

text_key pydantic-field

Functions

create classmethod

infer_data_format classmethod

resolve classmethod

EvaluationRequest pydantic-model

Attributes

dataset pydantic-field

metrics pydantic-field

Functions

create classmethod

DatasetRecord pydantic-model

Attributes

record_id pydantic-field

text pydantic-field

metadata pydantic-field

Functions

create classmethod

EvaluationResult pydantic-model

Attributes

record_id pydantic-field

results pydantic-field

Functions

create classmethod

Functions

BaseDTO `pydantic-model`

create `classmethod`

MetricSpec `pydantic-model`

name `pydantic-field`

config `pydantic-field`

result_key `pydantic-field`

create `classmethod`

BaseDatasetSpec `pydantic-model`

data_format `pydantic-field`

infer_data_format `classmethod`

resolve `classmethod`

create `classmethod`

JsonlDatasetSpec `pydantic-model`

data_format `pydantic-field`

input_file `pydantic-field`

record_id_key `pydantic-field`

text_key `pydantic-field`

create `classmethod`

infer_data_format `classmethod`

resolve `classmethod`

EvaluationRequest `pydantic-model`

dataset `pydantic-field`

metrics `pydantic-field`

create `classmethod`

DatasetRecord `pydantic-model`

record_id `pydantic-field`

text `pydantic-field`

metadata `pydantic-field`

create `classmethod`

EvaluationResult `pydantic-model`

record_id `pydantic-field`

results `pydantic-field`

create `classmethod`