base
Abstract generic base class BaseMetric[T] and its DefaultSchema configuration
| CLASS | DESCRIPTION |
|---|---|
DefaultSchema |
Default empty schema as a |
BaseMetric |
Abstract base class for all metrics, parameterized by a serializable value type. |
Type Aliases
Classes
DefaultSchema
pydantic-model
Default empty schema as a pydantic.BaseModel.
Used as the default value for the config_schema class variable in BaseMetric
to match the shared interface and indicate that a metric doesn't take any
configuration parameters. Schema enforces that no configuration parameters can be
passed at instantiation by forbidding extra fields and having no defined fields. (
see also: BaseMetric.from_config(**kwargs))
Config:
extra:forbidfrozen:True
BaseMetric
flowchart TD
evallm.metrics.base.BaseMetric[BaseMetric]
click evallm.metrics.base.BaseMetric href "" "evallm.metrics.base.BaseMetric"
Abstract base class for all metrics, parameterized by a serializable value type.
This generic class defines the shared interface implemented by all metrics. It
includes convenience methods and properties for tree traversal, as well as an
abstract evaluate method that must be implemented by all concrete metric
classes.
Metric interface
- The
evaluatemethod is expected to return a dictionary of the formdict[str, T], where the key is this metric instance'sresult_keyand the value is ofSerializableValueType. - A metric's
nameidentifies its class in aCatalogand serves as the defaultresult_keyunless a custom result key is set on the instance. Theresult_keyon the other hand, is an instance-level property that determines the key under which this metric's value is stored in the evaluation results. - The (invariant) generic type parameter
Tdeclares the expected serializable value shape for a metric. At class-definition time,BaseMetricvalidates thatTis compatible withSerializableValueType, andevaluateis validated againstdict[str, T]. - At runtime, calling
result(value)enforces thatvaluematchesTand returns{self.result_key: value}. Implementations that bypassresult(value)are expected, by convention, to preserve the same contract.
| PARAMETER | DESCRIPTION |
|---|---|
**kwargs |
The runtime configuration parameters for this metric instance, as
a dictionary. The expected parameters depend on the metric class' defined
|
| RAISES | DESCRIPTION |
|---|---|
ValidationError
|
If trying to instantiate a metric class with invalid runtime parameters (i.e. missing required parameters, extra unknown parameters, or parameters of the wrong type). |
ValueError
|
If trying to pass runtime parameters to a metric class that doesn't
define a |
-
API Reference
evallm
-
API Reference
evallm-
metrics -
application
-
-
API Reference
evallmmetrics
-
API Reference
evallmmetrics
| METHOD | DESCRIPTION |
|---|---|
__init_subclass__ |
Called when a class inherits from |
__eq__ |
Check if this metric is equal to another metric. |
__iter__ |
Iterate over the subtree rooted at this metric in breadth-first order. |
__contains__ |
Check if a metric is contained in the subtree rooted at this metric. |
__repr__ |
String representation of the subtree, rooted in this metric instance. |
result |
Helper method for wrapping the value of a metric's evaluation result in a |
evaluate |
Evaluate the metric on the given text. |
descendants |
Yield nodes in the subtree rooted at self (cycle-safe). |
from_config |
Factory method to create a metric instance from kwargs or JSON payload. |
| ATTRIBUTE | DESCRIPTION |
|---|---|
value_type |
The metric class' parametrized type, binding the generic type parameter
TYPE:
|
config_schema |
A schema defining the expected runtime configuration parameters for this metric. |
parent |
Get the parent of this metric.
TYPE:
|
result_key |
The key to use for this metric's value in the evaluation result dictionary.
TYPE:
|
name |
The name of this metric's class.
TYPE:
|
is_composite |
Check if this metric is a composite metric (i.e. can have child metrics).
TYPE:
|
is_root |
Check if this metric is a root metric (i.e. has no parent).
TYPE:
|
root |
Get the root metric of the tree this metric belongs to.
TYPE:
|
depth |
Get the depth of this metric in the tree (i.e. the number of edges from this
TYPE:
|
children |
Get an iterator over the children of this metric.
TYPE:
|
config |
Get the runtime configuration of this metric instance as a dictionary. |
Source code in src/evallm/metrics/base.py
Attributes
value_type
class-attribute
value_type: object
The metric class' parametrized type, binding the generic type parameter T.
-
API Reference
evallmmetrics
config_schema
class-attribute
A schema defining the expected runtime configuration parameters for this metric.
If the metric class does not explicitly define a config schema, this defaults to an
empty pydantic model (see
DefaultSchema) that takes no
parameters.
Info
A class' config schema is always configured such that:
- all declared fields must be provided unless they are declared as optional
- no additional fields beyond those declared in the schema are allowed (see
also
pydantic.ConfigDict.extra) - the schema is frozen/immutable once defined (see also
pydantic.ConfigDict.frozen) - configuration parameters provided at runtime are strictly validated against
the schema, raising a
ValidationErrorif the provided parameters violate it (see also pydantic Conversion Table)
-
API Reference
evallm
parent
property
writable
parent: CompositeMetric | None
Get the parent of this metric.
| RETURNS | DESCRIPTION |
|---|---|
CompositeMetric | None
|
CompositeMetric | None: The parent metric, or |
result_key
property
writable
result_key: str
The key to use for this metric's value in the evaluation result dictionary.
By default, this is the same as the metric's name, but it can be
overridden on a per-instance basis. This is useful when multiple instances of
the same metric class are used in a single evaluation request with different
runtime parameters and therefore need distinct keys in the output.
| RETURNS | DESCRIPTION |
|---|---|
result_key
|
The key of this metric instance's value in the evaluation
result dictionary. Defaults to the metric's
TYPE:
|
-
API Reference
evallm
name
property
name: str
The name of this metric's class.
This is a class-, not an instance-level property that's used for identifying
this metric in the catalog. It also serves as the default
result_key for metric instances unless explicitly overridden.
Metric class registration and name updates
When a metric class is registered in the catalog under a custom name using
@register_metric(name=...), the name-property of any pre-existing
instances of that class will be updated to reflect the new name in the
catalog.
| RETURNS | DESCRIPTION |
|---|---|
name
|
The name of this metric. Defaults to the class name if not
overwritten by the
TYPE:
|
-
API Reference
evallm-
application -
metrics
-
is_composite
property
is_composite: bool
Check if this metric is a composite metric (i.e. can have child metrics).
| RETURNS | DESCRIPTION |
|---|---|
bool
|
TYPE:
|
is_root
property
is_root: bool
Check if this metric is a root metric (i.e. has no parent).
| RETURNS | DESCRIPTION |
|---|---|
bool
|
TYPE:
|
root
property
root: BaseMetric[SerializableValueType]
Get the root metric of the tree this metric belongs to.
| RETURNS | DESCRIPTION |
|---|---|
BaseMetric
|
The root metric of the tree this metric belongs to.
TYPE:
|
depth
property
depth: int
Get the depth of this metric in the tree (i.e. the number of edges from this metric to the root).
| RETURNS | DESCRIPTION |
|---|---|
depth
|
The depth of this metric in the tree. Root metrics have depth 0, their children have depth 1, and so on.
TYPE:
|
children
property
children: Iterator[BaseMetric[SerializableValueType]]
Get an iterator over the children of this metric.
| YIELDS | DESCRIPTION |
|---|---|
children
|
An iterator over the child metrics (i.e. direct descendants) of this metric. If this metric is a leaf node, this will be an empty iterator.
TYPE::
|
config
property
Get the runtime configuration of this metric instance as a dictionary.
| RETURNS | DESCRIPTION |
|---|---|
config
|
A dictionary containing the runtime configuration of this metric instance, validated against the metric's schema. |
-
API Reference
evallmmetricsconfig Functionsconfig_schema
Functions
__init_subclass__
Called when a class inherits from BaseMetric.
Sets the default name and config schema of the metric class and validates
& stores the parametrized value type of the metric class (i.e.
the T in BaseMetric[T]) as the
value_type class variable
and ensures that evaluate is implemented with the correct signature.
| RAISES | DESCRIPTION |
|---|---|
TypeError
|
If the metric class does not implement the |
UnserializableValueTypeError
|
If the metric class's parametrized value type
is not an instance of
|
Source code in src/evallm/metrics/base.py
__eq__
Check if this metric is equal to another metric.
Two metrics are considered equal iff they are the same instance.
| PARAMETER | DESCRIPTION |
|---|---|
other
|
The other metric to compare against.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
bool
|
TYPE:
|
Source code in src/evallm/metrics/base.py
__iter__
__iter__() -> Iterator[BaseMetric[SerializableValueType]]
Iterate over the subtree rooted at this metric in breadth-first order.
| YIELDS | DESCRIPTION |
|---|---|
metric
|
An iterator over the subtree rooted at this metric in breadth-first order, excluding this metric itself.
TYPE::
|
Source code in src/evallm/metrics/base.py
__contains__
__contains__(metric: str | BaseMetric[SerializableValueType]) -> bool
Check if a metric is contained in the subtree rooted at this metric.
Note
Passing a metric instance checks for identity (i.e. whether exactly that instance is contained in this subtree), whereas passing a metric's name checks for membership (i.e. whether any metric with that name is contained in this subtree).
Example
Identity-based membership check: Name-based membership check:| PARAMETER | DESCRIPTION |
|---|---|
metric
|
The name of the metric to check for, or the metric instance itself.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
bool
|
TYPE:
|
Source code in src/evallm/metrics/base.py
__repr__
__repr__() -> str
String representation of the subtree, rooted in this metric instance.
| RETURNS | DESCRIPTION |
|---|---|
representation
|
A string representation of the metric instance, including its name and class.
TYPE:
|
Source code in src/evallm/metrics/base.py
result
Helper method for wrapping the value of a metric's evaluation result in a
dictionary to conform to the expected return type of evaluate.
Example
| PARAMETER | DESCRIPTION |
|---|---|
value
|
The result of computing the metric, where
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
result
|
A dictionary containing the given value, keyed by
the metric's |
| RAISES | DESCRIPTION |
|---|---|
TypeError
|
If the given value is not of the expected type |
-
API Reference
evallmmetrics
Source code in src/evallm/metrics/base.py
evaluate
abstractmethod
Evaluate the metric on the given text.
| PARAMETER | DESCRIPTION |
|---|---|
text
|
The text to evaluate the metric on.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
value
|
The evaluation result keyed by the metric's
|
Source code in src/evallm/metrics/base.py
descendants
descendants(
*, include_self: bool = False, order: Literal["dfs", "bfs"] = "dfs"
) -> Iterator[BaseMetric[SerializableValueType]]
Yield nodes in the subtree rooted at self (cycle-safe).
| PARAMETER | DESCRIPTION |
|---|---|
include_self
|
Whether to include
TYPE:
|
order
|
The order to traverse the tree. "dfs" for depth-first search, "bfs" for breadth-first search.
TYPE:
|
| YIELDS | DESCRIPTION |
|---|---|
descendants
|
An iterator over the descendant nodes of this metric, in the specified traversal order.
TYPE::
|
Source code in src/evallm/metrics/base.py
from_config
classmethod
from_config(*, json_payload: str | None = None, **kwargs: Any) -> BaseMetric[T]
Factory method to create a metric instance from kwargs or JSON payload.
Info
This method can be used to create any metric instance, so long as the
provided config parameters conform to the metric's schema. If the metric
class doesn't define a config schema (the default), calling this method
without kwargs is equivalent to calling the metric class constructor
directly.
The expected parameters depend on the metric's declared config_schema
(see also @config_schema decorator)
and the provided parameters must strictly conform to it.
| PARAMETER | DESCRIPTION |
|---|---|
json_payload
|
A JSON string containing the runtime parameters for this metric instance.
TYPE:
|
| PARAMETER | DESCRIPTION |
|---|---|
**kwargs |
The runtime parameters to use for this metric instance.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
metric
|
An instance of this metric class created using the provided configuration parameters.
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
TypeError
|
If |
ValueError
|
If
|
-
API Reference
evallm-
application -
metricsbase ClassesDefaultSchema
-