errors

Use mouse to pan and zoom

Use mouse to pan and zoom

Application-layer exceptions

CLASS	DESCRIPTION
EvaluationError	Base exception for application-layer evaluation failures.
RequestValidationError	Base exception for contextual validation of evaluation requests.
UnknownMetricError	Raised when a requested metric name is not registered.
MetricConfigurationError	Raised when a metric cannot be instantiated from the provided config.
DuplicateMetricResultKeyError	Raised when multiple metrics are specified with the same result key in a
UnsupportedDatasetError	Raised when no reader can handle a dataset spec.
DatasetAccessError	Raised when a dataset cannot be accessed or read.
DatasetRecordError	Base exception for invalid or unusable dataset records.
MissingTextFieldError	Raised when a record does not contain the configured text field.
MissingRecordIdFieldError	Raised when a record does not contain the configured record-id field.

Classes

EvaluationError

              flowchart TD
              evallm.application.errors.EvaluationError[EvaluationError]

              

              click evallm.application.errors.EvaluationError href "" "evallm.application.errors.EvaluationError"
            

Use mouse to pan and zoom

Base exception for application-layer evaluation failures.

RequestValidationError

              flowchart TD
              evallm.application.errors.RequestValidationError[RequestValidationError]
              evallm.application.errors.EvaluationError[EvaluationError]

                              evallm.application.errors.EvaluationError --> evallm.application.errors.RequestValidationError
                


              click evallm.application.errors.RequestValidationError href "" "evallm.application.errors.RequestValidationError"
              click evallm.application.errors.EvaluationError href "" "evallm.application.errors.EvaluationError"
            

Use mouse to pan and zoom

Base exception for contextual validation of evaluation requests.

UnknownMetricError

UnknownMetricError(metric_name: str, *, available: list[str] | None = None)

              flowchart TD
              evallm.application.errors.UnknownMetricError[UnknownMetricError]
              evallm.application.errors.RequestValidationError[RequestValidationError]
              evallm.application.errors.EvaluationError[EvaluationError]

                              evallm.application.errors.RequestValidationError --> evallm.application.errors.UnknownMetricError
                                evallm.application.errors.EvaluationError --> evallm.application.errors.RequestValidationError
                



              click evallm.application.errors.UnknownMetricError href "" "evallm.application.errors.UnknownMetricError"
              click evallm.application.errors.RequestValidationError href "" "evallm.application.errors.RequestValidationError"
              click evallm.application.errors.EvaluationError href "" "evallm.application.errors.EvaluationError"
            

Use mouse to pan and zoom

Raised when a requested metric name is not registered.

Raised by:

• API Reference evallm application
  •  application Classes  MetricSpec Functions  resolve_metric
  •  models Classes  MetricSpec Functions  resolve_metric

Source code in src/evallm/application/errors.py

def __init__(self, metric_name: str, *, available: list[str] | None = None) -> None:
    self.metric_name = metric_name
    self.available = available or []
    message = f"Unknown metric: {metric_name!r}."
    if self.available:  # pragma: no cover
        message += f" Available metrics: {', '.join(sorted(self.available))}."
    super().__init__(message)

MetricConfigurationError

MetricConfigurationError(
    metric_name: str,
    *,
    config: dict[str, JsonValue] | None = None,
    reason: str | None = None,
)

              flowchart TD
              evallm.application.errors.MetricConfigurationError[MetricConfigurationError]
              evallm.application.errors.RequestValidationError[RequestValidationError]
              evallm.application.errors.EvaluationError[EvaluationError]

                              evallm.application.errors.RequestValidationError --> evallm.application.errors.MetricConfigurationError
                                evallm.application.errors.EvaluationError --> evallm.application.errors.RequestValidationError
                



              click evallm.application.errors.MetricConfigurationError href "" "evallm.application.errors.MetricConfigurationError"
              click evallm.application.errors.RequestValidationError href "" "evallm.application.errors.RequestValidationError"
              click evallm.application.errors.EvaluationError href "" "evallm.application.errors.EvaluationError"
            

Use mouse to pan and zoom

Raised when a metric cannot be instantiated from the provided config.

Raised by:

• API Reference evallm application
  •  application Classes  MetricSpec Functions  resolve_metric
  •  models Classes  MetricSpec Functions  resolve_metric

Source code in src/evallm/application/errors.py

def __init__(
    self,
    metric_name: str,
    *,
    config: dict[str, JsonValue] | None = None,
    reason: str | None = None,
) -> None:
    self.metric_name = metric_name
    self.config = config
    self.reason = reason
    message = f"Invalid configuration for metric {metric_name!r}."
    if reason:  # pragma: no cover
        message += f" {reason}"
    super().__init__(message)

DuplicateMetricResultKeyError

DuplicateMetricResultKeyError(duplicate_keys: str | list[str], metric_names: list[str])

              flowchart TD
              evallm.application.errors.DuplicateMetricResultKeyError[DuplicateMetricResultKeyError]
              evallm.application.errors.RequestValidationError[RequestValidationError]
              evallm.application.errors.EvaluationError[EvaluationError]

                              evallm.application.errors.RequestValidationError --> evallm.application.errors.DuplicateMetricResultKeyError
                                evallm.application.errors.EvaluationError --> evallm.application.errors.RequestValidationError
                



              click evallm.application.errors.DuplicateMetricResultKeyError href "" "evallm.application.errors.DuplicateMetricResultKeyError"
              click evallm.application.errors.RequestValidationError href "" "evallm.application.errors.RequestValidationError"
              click evallm.application.errors.EvaluationError href "" "evallm.application.errors.EvaluationError"
            

Use mouse to pan and zoom

Raised when multiple metrics are specified with the same result key in a request.

Raised by:

• API Reference evallm application
  •  api Classes
    •  BaseEvaluationBuilder Functions  use
    •  JsonlEvaluationBuilder Functions  use
  •  application Classes
    •  BaseEvaluationBuilder Functions  use
    •  JsonlEvaluationBuilder Functions  use
  •  validation Functions  ensure_unique_metric_result_keys

Source code in src/evallm/application/errors.py

def __init__(
    self, duplicate_keys: str | list[str], metric_names: list[str]
) -> None:
    self.duplicate_keys: list[str] = (
        duplicate_keys if isinstance(duplicate_keys, list) else [duplicate_keys]
    )  # pragma: no cover
    self.metric_names: list[str] = metric_names
    duplicates: str = (
        f"'{duplicate_keys}'"
        if isinstance(duplicate_keys, str)
        else f"'{', '.join(self.duplicate_keys)}'"
    )  # pragma: no cover
    names: str = f"'{', '.join(self.metric_names)}'"
    super().__init__(f"Duplicate result keys {duplicates} for metrics: {names}.")

UnsupportedDatasetError

UnsupportedDatasetError(dataset_type: type[object])

              flowchart TD
              evallm.application.errors.UnsupportedDatasetError[UnsupportedDatasetError]
              evallm.application.errors.RequestValidationError[RequestValidationError]
              evallm.application.errors.EvaluationError[EvaluationError]

                              evallm.application.errors.RequestValidationError --> evallm.application.errors.UnsupportedDatasetError
                                evallm.application.errors.EvaluationError --> evallm.application.errors.RequestValidationError
                



              click evallm.application.errors.UnsupportedDatasetError href "" "evallm.application.errors.UnsupportedDatasetError"
              click evallm.application.errors.RequestValidationError href "" "evallm.application.errors.RequestValidationError"
              click evallm.application.errors.EvaluationError href "" "evallm.application.errors.EvaluationError"
            

Use mouse to pan and zoom

Raised when no reader can handle a dataset spec.

Source code in src/evallm/application/errors.py

def __init__(self, dataset_type: type[object]) -> None:
    self.dataset_type = dataset_type
    super().__init__(f"Unsupported dataset spec: {dataset_type.__name__}.")

DatasetAccessError

DatasetAccessError(*, path: str, reason: str)

              flowchart TD
              evallm.application.errors.DatasetAccessError[DatasetAccessError]
              evallm.application.errors.EvaluationError[EvaluationError]

                              evallm.application.errors.EvaluationError --> evallm.application.errors.DatasetAccessError
                


              click evallm.application.errors.DatasetAccessError href "" "evallm.application.errors.DatasetAccessError"
              click evallm.application.errors.EvaluationError href "" "evallm.application.errors.EvaluationError"
            

Use mouse to pan and zoom

Raised when a dataset cannot be accessed or read.

Source code in src/evallm/application/errors.py

def __init__(self, *, path: str, reason: str) -> None:
    self.path = path
    self.reason = reason
    super().__init__(f"Failed to access dataset at {path!r}: {reason}")

DatasetRecordError

DatasetRecordError(*, index: int, reason: str)

              flowchart TD
              evallm.application.errors.DatasetRecordError[DatasetRecordError]
              evallm.application.errors.EvaluationError[EvaluationError]

                              evallm.application.errors.EvaluationError --> evallm.application.errors.DatasetRecordError
                


              click evallm.application.errors.DatasetRecordError href "" "evallm.application.errors.DatasetRecordError"
              click evallm.application.errors.EvaluationError href "" "evallm.application.errors.EvaluationError"
            

Use mouse to pan and zoom

Base exception for invalid or unusable dataset records.

Source code in src/evallm/application/errors.py

def __init__(self, *, index: int, reason: str) -> None:
    self.index = index
    self.reason = reason
    super().__init__(f"Invalid dataset record at index {index}: {reason}")

MissingTextFieldError

MissingTextFieldError(*, index: int, text_key: str, record: dict[str, Any])

              flowchart TD
              evallm.application.errors.MissingTextFieldError[MissingTextFieldError]
              evallm.application.errors.DatasetRecordError[DatasetRecordError]
              evallm.application.errors.EvaluationError[EvaluationError]

                              evallm.application.errors.DatasetRecordError --> evallm.application.errors.MissingTextFieldError
                                evallm.application.errors.EvaluationError --> evallm.application.errors.DatasetRecordError
                



              click evallm.application.errors.MissingTextFieldError href "" "evallm.application.errors.MissingTextFieldError"
              click evallm.application.errors.DatasetRecordError href "" "evallm.application.errors.DatasetRecordError"
              click evallm.application.errors.EvaluationError href "" "evallm.application.errors.EvaluationError"
            

Use mouse to pan and zoom

Raised when a record does not contain the configured text field.

Source code in src/evallm/application/errors.py

def __init__(self, *, index: int, text_key: str, record: dict[str, Any]) -> None:
    self.text_key = text_key
    self.record = record
    super().__init__(
        index=index,
        reason=f"missing text field {text_key!r}",
    )

MissingRecordIdFieldError

MissingRecordIdFieldError(*, index: int, record_id_key: str, record: dict[str, Any])

              flowchart TD
              evallm.application.errors.MissingRecordIdFieldError[MissingRecordIdFieldError]
              evallm.application.errors.DatasetRecordError[DatasetRecordError]
              evallm.application.errors.EvaluationError[EvaluationError]

                              evallm.application.errors.DatasetRecordError --> evallm.application.errors.MissingRecordIdFieldError
                                evallm.application.errors.EvaluationError --> evallm.application.errors.DatasetRecordError
                



              click evallm.application.errors.MissingRecordIdFieldError href "" "evallm.application.errors.MissingRecordIdFieldError"
              click evallm.application.errors.DatasetRecordError href "" "evallm.application.errors.DatasetRecordError"
              click evallm.application.errors.EvaluationError href "" "evallm.application.errors.EvaluationError"
            

Use mouse to pan and zoom

Raised when a record does not contain the configured record-id field.

Source code in src/evallm/application/errors.py

def __init__(
    self,
    *,
    index: int,
    record_id_key: str,
    record: dict[str, Any],
) -> None:
    self.record_id_key = record_id_key
    self.record = record
    super().__init__(
        index=index,
        reason=f"missing record id field {record_id_key!r}",
    )