Skip to content

evaLLM Architecture Overview

This document describes evaLLM at a high level, focusing on interfaces, major flows, and boundary-observable behaviour.

System Context

evaLLM exposes two ingress paths:

  • CLI ingress through evallm.cli.main.
  • Library ingress through evallm.application.api and evallm.application.services.Evaluator.

Both paths converge on the same canonical contracts (EvaluationRequest, MetricSpec, dataset specs), then flow through the same orchestration pipeline. Request validation is designed to fail fast, surfacing errors before execution.

Overall Architecture

flowchart LR
    accTitle: evaLLM overall architecture
    accDescr: High-level component architecture for evaLLM showing boundaries and data flow from CLI or Python API to results output.

    subgraph Consumers
        CLI["CLI User"]
        PY["Python Consumer"]
    end

    subgraph Ingress
        CLIMain["evallm.cli.main"]
        API["evallm.application.api"]
    end

    subgraph AppLayer["Application Layer"]
        DTO["models: EvaluationRequest and MetricSpec"]
        Resolver["services.RequestResolver"]
        Eval["services.Evaluator"]
        Adapter["adapters.resolved_records"]
        Val["validation.ensure_unique_metric_result_keys"]
    end

    subgraph ReaderLayer["Reader Layer"]
        BR["readers.BaseReader"]
        JR["readers.JsonlReader"]
        DS["(JSONL Dataset File)"]
    end

    subgraph MetricLayer["Metric Layer"]
        Catalog["registry.Catalog and MetricsCatalog"]
        BM["base.BaseMetric"]
        CM["composite.CompositeMetric"]
        Builtin["builtin metrics"]
        CFG["config_schema decorator"]
        Reg["@register_metric decorator"]
    end

    subgraph Egress
        Out["(JSONL Evaluation Results)"]
    end

    CLI --> CLIMain
    PY --> API

    CLIMain --> DTO
    API --> DTO

    DTO --> Resolver
    Resolver --> Val
    Resolver --> Catalog
    Resolver --> CM
    Catalog --> Builtin
    Builtin --> BM
    CFG -. attaches schema .-> BM
    Reg -. registers metric class .-> Catalog

    Resolver --> JR
    JR -. implements .-> BR
    JR --> DS
    Resolver --> Adapter
    Adapter --> JR

    Resolver --> Eval
    Eval --> CM
    Eval --> Out
Use mouse to pan and zoom

Abstract

  • A single orchestration path utilizing canonical DTOs keeps behaviour consistent between CLI and programmatic usage.
  • Reader and metric subsystems are decoupled from user-facing logic and remain replaceable behind stable interfaces (BaseReader, Catalog, BaseMetric).
  • Configuration and registration are explicit extension points (config_schema, register_metric); together they allow for flexible adaptation of the framework to custom use cases.
  • The metric interface (specifically, the parametrization of BaseMetric subclasses) enables generic downstream processing of evaluation results. For example, adding pluggable strategies for aggregating the results of individual records into a single dataset evaluation score, would be as straightforward as designing an aggregation layer that implements strategies for generic types.

CLI and Programmatic Interaction Flow

sequenceDiagram
    accTitle: CLI and programmatic interaction flow
    accDescr: End-to-end sequence for both CLI and Python API usage, including request resolution, dataset reading, metric execution, and result emission.

    actor U as User
    participant C as CLI main
    participant A as API builder/helpers
    participant E as Evaluator
    participant R as RequestResolver
    participant Cat as MetricsCatalog
    participant Ad as resolved_records adapter
    participant J as JsonlReader
    participant M as CompositeMetric

    rect rgb(245, 250, 255)
        Note over U,A: Programmatic path
        U->>A: Build EvaluationRequest (or JsonlEvaluationBuilder)
        A->>E: Instantiate Evaluator(request)
    end

    rect rgb(255, 249, 245)
        Note over U,C: CLI path
        U->>C: evallm run --input --metric ...
        C->>A: Parse presets/specs and construct EvaluationRequest
        A->>E: Instantiate Evaluator(request)
    end

    E->>R: Resolve request
    R->>R: Validate unique metric result keys
    R->>Cat: Resolve each MetricSpec by name
    Cat-->>R: Metric classes
    R->>M: Build CompositeMetric with metric instances

    R->>J: Create reader for dataset spec
    J->>Ad: Iterate raw JSONL records
    Ad-->>R: DatasetRecord iterator
    R-->>E: (records context, composite metric)

    loop for each dataset record
        E->>M: evaluate(record.text)
        M-->>E: results by result_key
        E-->>U: EvaluationResult (streamed)
    end

    alt CLI output mode
        C-->>U: Write JSONL line(s) to stdout or file
    else programmatic mode
        E-->>U: Yield EvaluationResult iterator
    end
Use mouse to pan and zoom

Object Relationships

Application Layer

---
config:
    class:
        hideEmptyMembersBox: true
---
classDiagram
    accTitle: Application package relationships
    accDescr: Application layer DTOs, builders, and services.
    direction LR

    namespace models {
        class pydantic.BaseModel
        class BaseDTO {
            <<abstract>>
            #model_config: pydantic.ConfigDict
            +create(obj: Any | None = None, json_payload: str | bytes | bytearray | None = None, **kwargs) BaseDTO
        }
        class BaseDatasetSpec {
            <<abstract>>
            +data_format: str
            -_specs_by_format: ClassVar~dict~
            +infer_data_format(data: Mapping[str, Any]) str | None
            +resolve(data: BaseDatasetSpec | Mapping[str, Any]) BaseDatasetSpec
        }
        class JsonlDatasetSpec {
            +data_format: Literal['jsonl']
            +input_file: FilePath
            +record_id_key: str
            +text_key: str
            +...()
        }
        class MetricSpec {
            +name: str
            +config: dict[str, JsonValue] | None = None
            +result_key: str | None = None
            +resolve_metric(catalog: type[Catalog]): BaseMetric
        }
        class EvaluationRequest {
            +dataset: BaseDatasetSpec
            +metrics: list[MetricSpec]
            +...()
        }
        class EvaluationResult {
            +record_id: str
            +results: dict[str, JsonValue]
            +...()
        }
        class DatasetRecord {
            +record_id: str
            +text: str
            +metadata: dict[str, JsonValue] | None = None
            +...()
        }
    }

    class BaseEvaluationBuilder~T~ {
        <<abstract>>
        #_catalog: type[Catalog]
        #_metrics: list[MetricSpec]
        +catalog: type[Catalog]*
        +metrics: tuple[MetricSpec, ...]*
        +use(metric_spec: MetricSpec) Self
        +build_dataset()* T
        +build_request() EvaluationRequest
        +build_evaluator() Evaluator
        +run() Iterator[EvaluationResult]
        +from_request(request: EvaluationRequest, catalog: type[Catalog] = MetricsCatalog)$ BaseEvaluationBuilder~T~*
    }
    class JsonlEvaluationBuilder~JsonlDatasetSpec~ {
        -_input_file: Path
        -_text_key: str
        -_record_id_key: str
        +input_file: Path
        +text_key: str
        +record_id_key: str
        +JsonlEvaluationBuilder(input_file: str | os.PathLike, text_key: str, record_id_key: str = "#IDX#", catalog: type[Catalog] = MetricsCatalog)
        +build_dataset() JsonlDatasetSpec
        +from_request(request: EvaluationRequest, catalog: type[Catalog] = MetricsCatalog)$ JsonlEvaluationBuilder~JsonlDatasetSpec~
    }

    namespace services {
        class RequestResolver {
            -_catalog: type[Catalog]
            -_create_reader(spec: BaseDatasetSpec) BaseReader
            -_resolved_dataset(request: EvaluationRequest) ContextManager
            -_ensure_unique_result_keys(request: EvaluationRequest) None
            -_compose_metrics(request: EvaluationRequest) CompositeMetric
            +RequestResolver(catalog: type[Catalog] = MetricsCatalog)
            +\_\_call\_\_(request: EvaluationRequest) tuple[ContextManager[Iterator[DatasetRecord]], CompositeMetric]
        }
        class Evaluator {
            -_resolved_records: ContextManager[Iterator[DatasetRecord]]
            -_composite_metric: CompositeMetric
            -_evaluate_record(record: DatasetRecord): dict[str, JsonValue]
            +run() Iterator[EvaluationResult]
        }
    }

    BaseEvaluationBuilder <|-- JsonlEvaluationBuilder
    pydantic.BaseModel <|-- BaseDTO
    BaseDTO <|-- BaseDatasetSpec
    BaseDTO <|-- MetricSpec
    BaseDTO <|-- EvaluationRequest
    BaseDTO <|-- EvaluationResult
    BaseDTO <|-- DatasetRecord
    BaseDatasetSpec <|-- JsonlDatasetSpec

    EvaluationRequest --* BaseDatasetSpec: 1
    EvaluationRequest --* "*" MetricSpec: metrics
    EvaluationRequest <-- RequestResolver: validates
    EvaluationResult <-- Evaluator: yields
    DatasetRecord <-- RequestResolver: creates iterator of
    Evaluator --> RequestResolver: uses
    Evaluator --> CompositeMetric: evaluates
    RequestResolver --> Catalog: resolves metrics via
    MetricSpec --> BaseMetric: resolved to
Use mouse to pan and zoom

Metrics Layer

---
config:
    class:
        hideEmptyMembersBox: true
---
classDiagram
    accTitle: Metrics package relationships
    accDescr: Metric base class, composite container, built-in implementations, registration, and configuration system.
    direction LR

    %% note for BaseMetric "Component-interface in the Composite pattern and simultaneously abstract base class"
    namespace Composite Pattern {
        class BaseMetric~T: SerializableValueType~ {
            <<abstract>>
            value_type: ClassVar[object]
            config_schema: ClassVar[type[BaseModel]] = DefaultSchema
            -_name: ClassVar[str]
            #name: str
            #result_key: str
            #config: dict[str, Any]
            #parent: BaseMetric | None
            #+is_composite: bool
            #+is_root: bool
            #+depth: int
            #+root: BaseMetric
            #+children: Iterator~BaseMetric~
            +BaseMetric(**kwargs)
            +from_config(json_payload: str | None = None, **kwargs)$ BaseMetric~T~
            -\_\_init_subclass\_\_(**kwargs)
            -\_\_eq\_\_(other: object) bool
            -\_\_iter\_\_() Iterator[BaseMetric]
            -\_\_contains\_\_(metric: BaseMetric) bool
            -\_\_repr\_\_() str
            +descendants(include_self: bool = False, order: Literal["dfs", "bfs"] = "dfs") Iterator[BaseMetric]
            +result(value: T) dict[str, T]
            +evaluate(text: str)* dict[str, T]
        }
        class CompositeMetric~dict[str, SerializableValueType]~ {
            -_children: list[BaseMetric]
            +add(metric: BaseMetric, move: bool = False) None
            +remove(metric: BaseMetric) BaseMetric
            +evaluate(text: str) dict[str, dict[str, SerializableValueType]]
        }
    }

    %% note "DefaultSchema is not explicitly shown in this diagram since it only serves as a default"
    class DefaultSchema {
        #model_config: ConfigDict
        #...()
    }

    class pydantic.BaseModel {
        #model_config: ConfigDict
        +model_dump() dict[str, Any]
        +model_validate(dict[str, Any]) BaseModel
        +model_validate_json(str) BaseModel
    }

    namespace builtin {
        class CountsMetric~int~ {
            <<metric>>
            +config_schema: type[CountsConfig]
            +evaluate(text: str) dict[str, int]
        }
        class CountsConfig {
            +segments: Literal[characters|tokens|encodings]
            +segmentation: str | None
            +hf_token: str | None
            #+tokenizer: Callable[str, list[str] | list[int]]*
            #...()
        }
        class JsonFormatMetric~dict[str, bool]~ {
            <<metric>>
            +config_schema: type[JsonFormatConfig]
            +evaluate(text) dict[str, dict[str, bool]]
        }
        class JsonFormatConfig {
            +expected_schema: dict[str, Any]
            +check_formats: bool
            #...()
        }
    }

    namespace registry {
        class Catalog {
            <<interface>>
            +available()$ list[str]
            +get_metric(identifier)$ type[BaseMetric]
            +is_registered(metric)$ bool
            +register(metric)$ None
            +rename_metric(old, new)$ None
        }
        class MetricsCatalog {
            #_registry: ClassVar[dict[str, type[BaseMetric]]]
            +available()$ list[str]
            +get_metric(name)$ type[BaseMetric]
            +is_registered(metric)$ bool
            +register(metric)$ None
            +rename_metric(old, new)$ None
        }
    }

    BaseMetric <|-- CompositeMetric
    BaseMetric <|.. CompositeMetric
    BaseMetric <|-- CountsMetric
    BaseMetric <|-- JsonFormatMetric
    pydantic.BaseModel <|-- DefaultSchema
    pydantic.BaseModel <|-- CountsConfig
    pydantic.BaseModel <|-- JsonFormatConfig
    Catalog <|.. MetricsCatalog
    BaseMetric~T: SerializableValueType~ --> DefaultSchema: uses
    CountsMetric --> CountsConfig: @config_schema
    JsonFormatMetric --> JsonFormatConfig: @config_schema
    CompositeMetric o--> "*" BaseMetric
    MetricsCatalog <-- BaseMetric: @register_metric
Use mouse to pan and zoom

Readers Layer

---
config:
        class:
            hideEmptyMembersBox: true
---
classDiagram
    accTitle: Readers package relationships
    accDescr: Lazy file reader abstraction, JSONL implementation, and errors.

    namespace abstractions {
        class BaseReader~T~{
            <<abstract>>
            -_path: pathlib.Path
            -_file: TextIOBase | None
            -_closed: bool
            +__init__(path)
            +__enter__() BaseReader[T]
            +__exit__(exc_type, exc_value, exc_tb) None
            +__iter__() BaseReader[T]
            +__next__() T
            +send() T
            +throw(exc_type, exc_value, exc_tb) NoReturn
            +close() None
            -_parse_line(line: str)* T
        }
    }

    namespace concrete {
        class JsonlReader~dict[str, Any]~ {
            ...
            -_parse_line(line: str) dict[str, Any]
        }
    }

    namespace exceptions {
        class ReaderError
        class InvalidMessageFormatError
    }

    BaseReader <|-- JsonlReader
    ReaderError <|-- InvalidMessageFormatError
Use mouse to pan and zoom