Skip to content

readers

Input ingestion layer for evaLLM datasets.

status
Use mouse to pan and zoom
Use mouse to pan and zoom

This package isolates file-reading concerns from metric and orchestration logic.

Abstract

evallm.readers is the ingestion boundary. reader provides a reusable, lazy streaming contract plus concrete file-format readers, while utils offers optional post-parse helpers for callers that need structured message conversion.

Design Decisions

Decision: Use lazy, line-by-line iteration.
Motivation: dataset inputs can be large; BaseReader parses records on demand instead of loading the full file into memory.
Decision: Combine iterator and context-manager contracts.
Motivation: consumers can stream naturally with for while still getting deterministic resource cleanup via with.
Decision: Keep readers format-specific but output-shape-neutral.
Motivation: concrete readers (for example JsonlReader) produce raw dictionaries; normalization into canonical DTOs is an application-layer adapter responsibility.
Decision: Expose message-conversion helpers outside the core reader abstraction.
Motivation: optional helper utilities should not couple the base reading contract to one message representation.
Referenced by:
MODULE DESCRIPTION
reader

Abstract base class for lazy file reading and an implementation for JSONL files
utils

Utilities for file reading and parsing
CLASS DESCRIPTION
BaseReader

Abstract base class implementing Contextmanager & Iterator protocols for lazy
JsonlReader

Reader for JSON Lines (JSONL) files that lazily yields records as dictionaries.
InvalidMessageFormatError

Exception that is raised when a dictionary cannot be parsed into a valid
FUNCTION DESCRIPTION
messages_from_dicts

Convert a sequence of dictionaries to a list of BaseMessage objects.

Classes

BaseReader 

BaseReader(path: PathLike)


              flowchart TD
              evallm.readers.BaseReader[BaseReader]

              

              click evallm.readers.BaseReader href "" "evallm.readers.BaseReader"
Use mouse to pan and zoom

Abstract base class implementing Contextmanager & Iterator protocols for lazy file reading.

Subclasses must implement the _parse_line method to define how each line of the file should be parsed into the desired type T. The base class handles file opening, closing, and iteration, allowing subclasses to focus solely on the parsing logic.

PARAMETER DESCRIPTION
path

The path to the file to be read.

TYPE: PathLike

RAISES DESCRIPTION
FileNotFoundError

If the specified file does not exist at the given path.
Returned by:
METHOD DESCRIPTION
__enter__

Enter the runtime context related to this object. The file is not opened
__exit__

Ensure that the file is properly closed when exiting the context. Exceptions
__iter__

Iterator over the lines of the file, yielding parsed objects of type T.
__next__

Read the next line from the file, parse it into type T, and return it. If the
send

Implemented to satisfy the Generator protocol, but not used in this context.
throw

Pause the generator and ensure that the file is closed before propagating
close

Close the file if it is open and mark the reader as closed to prevent
Source code in src/evallm/readers/reader.py 
38
39
40
41
42
43
44
45
46
47
48
49
50
51
def __init__(self, path: os.PathLike):
    """Initialize the BaseReader with the path to a file.

    Args:
        path (os.PathLike): The path to the file to be read.

    Raises:
        FileNotFoundError: If the specified file does not exist at the given path.
    """
    if not pathlib.Path.exists(path := pathlib.Path(path)):
        raise FileNotFoundError(f"File not found: {path}")
    self._path: pathlib.Path = path
    self._file: TextIOBase | None = None
    self._closed: bool = False

Functions

__enter__ 
__enter__() -> BaseReader[T]

Enter the runtime context related to this object. The file is not opened until the first line is read, so this method simply returns the reader instance.

RETURNS DESCRIPTION
BaseReader[T]

BaseReader[T]: The reader instance itself, ready to be used as a context
BaseReader[T]

manager for lazy file reading.
Source code in src/evallm/readers/reader.py 
53
54
55
56
57
58
59
60
61
def __enter__(self) -> BaseReader[T]:
    """Enter the runtime context related to this object. The file is not opened
    until the first line is read, so this method simply returns the reader instance.

    Returns:
        BaseReader[T]: The reader instance itself, ready to be used as a context
        manager for lazy file reading.
    """
    return self
__exit__ 
__exit__(
    exc_type: type[BaseException] | None,
    exc_value: BaseException | None,
    exc_tb: TracebackType | None,
) -> None

Ensure that the file is properly closed when exiting the context. Exceptions are propagated after closing the file.

Source code in src/evallm/readers/reader.py 
63
64
65
66
67
68
69
70
71
72
def __exit__(
    self,
    exc_type: type[BaseException] | None,
    exc_value: BaseException | None,
    exc_tb: TracebackType | None,
) -> None:
    """Ensure that the file is properly closed when exiting the context. Exceptions
    are propagated after closing the file."""
    self.close()
    return None
__iter__ 
__iter__() -> BaseReader[T]

Iterator over the lines of the file, yielding parsed objects of type T.

RETURNS DESCRIPTION
BaseReader[T]

BaseReader[T]: The reader instance itself, which is an iterator over the
BaseReader[T]

parsed lines of the file.
Source code in src/evallm/readers/reader.py 
74
75
76
77
78
79
80
81
def __iter__(self) -> BaseReader[T]:
    """Iterator over the lines of the file, yielding parsed objects of type `T`.

    Returns:
        BaseReader[T]: The reader instance itself, which is an iterator over the
        parsed lines of the file.
    """
    return self
__next__ 
__next__() -> T

Read the next line from the file, parse it into type T, and return it. If the end of the file is reached, the file is closed and StopIteration is raised.

RETURNS DESCRIPTION
T

The next parsed object from the file.

TYPE: T

RAISES DESCRIPTION
StopIteration

If the end of the file is reached, indicating that there are no more lines to read.
Source code in src/evallm/readers/reader.py 
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
def __next__(self) -> T:
    """Read the next line from the file, parse it into type `T`, and return it. If the
    end of the file is reached, the file is closed and `StopIteration` is raised.

    Returns:
        T: The next parsed object from the file.

    Raises:
        StopIteration: If the end of the file is reached, indicating that there are
            no more lines to read.
    """
    if self._closed:
        raise StopIteration
    if self._file is None:
        # enter self as context manager to ensure file is closed on exceptions
        self._file = self._path.open("r", encoding="utf-8")
    if not (line := self._file.readline()):
        self.close()
        raise StopIteration
    return self._parse_line(line.strip("\n"))
send 
send() -> T

Implemented to satisfy the Generator protocol, but not used in this context.

Always returns the next parsed object from the file, since BaseReader does not support receiving values via send(). This method is effectively an alias for __next__().

RETURNS DESCRIPTION
T

The next parsed object from the file.

TYPE: T

Source code in src/evallm/readers/reader.py 
104
105
106
107
108
109
110
111
112
113
114
def send(self) -> T:
    """Implemented to satisfy the Generator protocol, but not used in this context.

    Always returns the next parsed object from the file, since `BaseReader` does
    not support receiving values via `send()`. This method is effectively an alias
    for `__next__()`.

    Returns:
        T: The next parsed object from the file.
    """
    return self.__next__()
throw 
throw(
    exc_type: type[BaseException] | None,
    exc_value: BaseException | None = None,
    exc_tb: TracebackType | None = None,
) -> NoReturn

Pause the generator and ensure that the file is closed before propagating either the specified exception or a default StopIteration.

PARAMETER DESCRIPTION
exc_type

The type of the exception to be raised. If None, a StopIteration is raised.

TYPE: type[BaseException] | None

exc_value

The value of the exception to be raised.

TYPE: BaseException | None DEFAULT: None

exc_tb

The traceback associated with the exception.

TYPE: TracebackType | None DEFAULT: None

RAISES DESCRIPTION
StopIteration

If the end of the file is reached or if exc_type is None, indicating that the generator should be stopped.
Source code in src/evallm/readers/reader.py 
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
def throw(
    self,
    exc_type: type[BaseException] | None,
    exc_value: BaseException | None = None,
    exc_tb: TracebackType | None = None,
) -> NoReturn:
    """Pause the generator and ensure that the file is closed before propagating
    either the specified exception or a default `StopIteration`.

    Args:
        exc_type (type[BaseException] | None): The type of the exception
            to be raised. If `None`, a `StopIteration` is raised.
        exc_value (BaseException | None): The value of the exception to be raised.
        exc_tb (TracebackType | None): The traceback associated with the exception.

    Raises:
        StopIteration: If the end of the file is reached or if `exc_type` is
            `None`, indicating that the generator should be stopped.
    """
    self.close()  # ensure file is closed before propagating exception

    if exc_type is None:
        raise StopIteration

    exc: BaseException = exc_type() if exc_value is None else exc_value
    if exc_tb is not None:
        exc = exc.with_traceback(exc_tb)

    raise exc
close 
close() -> None

Close the file if it is open and mark the reader as closed to prevent further access.

Source code in src/evallm/readers/reader.py 
146
147
148
149
150
151
152
def close(self) -> None:
    """Close the file if it is open and mark the reader as closed to prevent
    further access."""
    if self._file:
        self._file.close()
        self._file = None
    self._closed = True

JsonlReader 

JsonlReader(path: PathLike)


              flowchart TD
              evallm.readers.JsonlReader[JsonlReader]
              evallm.readers.reader.BaseReader[BaseReader]

                              evallm.readers.reader.BaseReader --> evallm.readers.JsonlReader
                


              click evallm.readers.JsonlReader href "" "evallm.readers.JsonlReader"
              click evallm.readers.reader.BaseReader href "" "evallm.readers.reader.BaseReader"
Use mouse to pan and zoom

Reader for JSON Lines (JSONL) files that lazily yields records as dictionaries.

Acts as a context manager for safe file handling; the file is opened on the first read and automatically closed when done or if an exception occurs. Each line of the JSONL file is parsed into a dictionary using orjson for efficient JSON parsing.

Tip

Once instantiated, a JsonlReader can be reused across multiple with blocks to re-read the same file.

Usage 
with (lines := JsonlReader("path/to/gpt-5.1-codex-max-1000x.jsonl")):
    for ln in lines:
        parsed_dict: dict[str, Any] = (
            ln  # each line is parsed into a dictionary
        )
        try:
            # if the jsonl file is a dump of LLM-outputs in a format known
            # to langchain (e.g. OpenAI chat completions), we can convert the
            # dictionaries into canonical BaseMessage objects for easier
            # downstream processing
            messages: list[BaseMessage] = JsonlReader.messages_from_dicts(ln)
            pretty_ln: str = "\n".join([msg.pretty_repr() for msg in messages])
        except InvalidMessageFormatError:
            # an error is thrown if the parsed dictionary doesn't conform
            # to a valid BaseMessage format
            continue
print(pretty_ln)

>>> ================================ Human Message =================================

    Build a Python web scraper for a site with heavy anti-bot protection like Cloudflare.
    Use Playwright with stealth plugins and implement human-like interactions with
    randomized mouse movements and delays. Include CAPTCHA solving using Tesseract or
    a third-party API.
    ================================== Ai Message ==================================

    Sorry, I can’t help with that.
PARAMETER DESCRIPTION
path

The path to the file to be read.

TYPE: PathLike

RAISES DESCRIPTION
FileNotFoundError

If the specified file does not exist at the given path.
METHOD DESCRIPTION
__enter__

Enter the runtime context related to this object. The file is not opened
__exit__

Ensure that the file is properly closed when exiting the context. Exceptions
__iter__

Iterator over the lines of the file, yielding parsed objects of type T.
__next__

Read the next line from the file, parse it into type T, and return it. If the
send

Implemented to satisfy the Generator protocol, but not used in this context.
throw

Pause the generator and ensure that the file is closed before propagating
close

Close the file if it is open and mark the reader as closed to prevent
Source code in src/evallm/readers/reader.py 
38
39
40
41
42
43
44
45
46
47
48
49
50
51
def __init__(self, path: os.PathLike):
    """Initialize the BaseReader with the path to a file.

    Args:
        path (os.PathLike): The path to the file to be read.

    Raises:
        FileNotFoundError: If the specified file does not exist at the given path.
    """
    if not pathlib.Path.exists(path := pathlib.Path(path)):
        raise FileNotFoundError(f"File not found: {path}")
    self._path: pathlib.Path = path
    self._file: TextIOBase | None = None
    self._closed: bool = False

Functions

__enter__ 
__enter__() -> BaseReader[T]

Enter the runtime context related to this object. The file is not opened until the first line is read, so this method simply returns the reader instance.

RETURNS DESCRIPTION
BaseReader[T]

BaseReader[T]: The reader instance itself, ready to be used as a context
BaseReader[T]

manager for lazy file reading.
Source code in src/evallm/readers/reader.py 
53
54
55
56
57
58
59
60
61
def __enter__(self) -> BaseReader[T]:
    """Enter the runtime context related to this object. The file is not opened
    until the first line is read, so this method simply returns the reader instance.

    Returns:
        BaseReader[T]: The reader instance itself, ready to be used as a context
        manager for lazy file reading.
    """
    return self
__exit__ 
__exit__(
    exc_type: type[BaseException] | None,
    exc_value: BaseException | None,
    exc_tb: TracebackType | None,
) -> None

Ensure that the file is properly closed when exiting the context. Exceptions are propagated after closing the file.

Source code in src/evallm/readers/reader.py 
63
64
65
66
67
68
69
70
71
72
def __exit__(
    self,
    exc_type: type[BaseException] | None,
    exc_value: BaseException | None,
    exc_tb: TracebackType | None,
) -> None:
    """Ensure that the file is properly closed when exiting the context. Exceptions
    are propagated after closing the file."""
    self.close()
    return None
__iter__ 
__iter__() -> BaseReader[T]

Iterator over the lines of the file, yielding parsed objects of type T.

RETURNS DESCRIPTION
BaseReader[T]

BaseReader[T]: The reader instance itself, which is an iterator over the
BaseReader[T]

parsed lines of the file.
Source code in src/evallm/readers/reader.py 
74
75
76
77
78
79
80
81
def __iter__(self) -> BaseReader[T]:
    """Iterator over the lines of the file, yielding parsed objects of type `T`.

    Returns:
        BaseReader[T]: The reader instance itself, which is an iterator over the
        parsed lines of the file.
    """
    return self
__next__ 
__next__() -> T

Read the next line from the file, parse it into type T, and return it. If the end of the file is reached, the file is closed and StopIteration is raised.

RETURNS DESCRIPTION
T

The next parsed object from the file.

TYPE: T

RAISES DESCRIPTION
StopIteration

If the end of the file is reached, indicating that there are no more lines to read.
Source code in src/evallm/readers/reader.py 
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
def __next__(self) -> T:
    """Read the next line from the file, parse it into type `T`, and return it. If the
    end of the file is reached, the file is closed and `StopIteration` is raised.

    Returns:
        T: The next parsed object from the file.

    Raises:
        StopIteration: If the end of the file is reached, indicating that there are
            no more lines to read.
    """
    if self._closed:
        raise StopIteration
    if self._file is None:
        # enter self as context manager to ensure file is closed on exceptions
        self._file = self._path.open("r", encoding="utf-8")
    if not (line := self._file.readline()):
        self.close()
        raise StopIteration
    return self._parse_line(line.strip("\n"))
send 
send() -> T

Implemented to satisfy the Generator protocol, but not used in this context.

Always returns the next parsed object from the file, since BaseReader does not support receiving values via send(). This method is effectively an alias for __next__().

RETURNS DESCRIPTION
T

The next parsed object from the file.

TYPE: T

Source code in src/evallm/readers/reader.py 
104
105
106
107
108
109
110
111
112
113
114
def send(self) -> T:
    """Implemented to satisfy the Generator protocol, but not used in this context.

    Always returns the next parsed object from the file, since `BaseReader` does
    not support receiving values via `send()`. This method is effectively an alias
    for `__next__()`.

    Returns:
        T: The next parsed object from the file.
    """
    return self.__next__()
throw 
throw(
    exc_type: type[BaseException] | None,
    exc_value: BaseException | None = None,
    exc_tb: TracebackType | None = None,
) -> NoReturn

Pause the generator and ensure that the file is closed before propagating either the specified exception or a default StopIteration.

PARAMETER DESCRIPTION
exc_type

The type of the exception to be raised. If None, a StopIteration is raised.

TYPE: type[BaseException] | None

exc_value

The value of the exception to be raised.

TYPE: BaseException | None DEFAULT: None

exc_tb

The traceback associated with the exception.

TYPE: TracebackType | None DEFAULT: None

RAISES DESCRIPTION
StopIteration

If the end of the file is reached or if exc_type is None, indicating that the generator should be stopped.
Source code in src/evallm/readers/reader.py 
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
def throw(
    self,
    exc_type: type[BaseException] | None,
    exc_value: BaseException | None = None,
    exc_tb: TracebackType | None = None,
) -> NoReturn:
    """Pause the generator and ensure that the file is closed before propagating
    either the specified exception or a default `StopIteration`.

    Args:
        exc_type (type[BaseException] | None): The type of the exception
            to be raised. If `None`, a `StopIteration` is raised.
        exc_value (BaseException | None): The value of the exception to be raised.
        exc_tb (TracebackType | None): The traceback associated with the exception.

    Raises:
        StopIteration: If the end of the file is reached or if `exc_type` is
            `None`, indicating that the generator should be stopped.
    """
    self.close()  # ensure file is closed before propagating exception

    if exc_type is None:
        raise StopIteration

    exc: BaseException = exc_type() if exc_value is None else exc_value
    if exc_tb is not None:
        exc = exc.with_traceback(exc_tb)

    raise exc
close 
close() -> None

Close the file if it is open and mark the reader as closed to prevent further access.

Source code in src/evallm/readers/reader.py 
146
147
148
149
150
151
152
def close(self) -> None:
    """Close the file if it is open and mark the reader as closed to prevent
    further access."""
    if self._file:
        self._file.close()
        self._file = None
    self._closed = True

InvalidMessageFormatError 


              flowchart TD
              evallm.readers.InvalidMessageFormatError[InvalidMessageFormatError]

              

              click evallm.readers.InvalidMessageFormatError href "" "evallm.readers.InvalidMessageFormatError"
Use mouse to pan and zoom

Exception that is raised when a dictionary cannot be parsed into a valid BaseMessage, either because it is missing a required key or because the message type is not supported.

Raised by:

Functions

messages_from_dicts 

messages_from_dicts(dicts: dict | Sequence[dict]) -> list[BaseMessage]

Convert a sequence of dictionaries to a list of BaseMessage objects.

PARAMETER DESCRIPTION
dicts

A single or a sequence of dictionaries to be converted into BaseMessage objects.

  • If a single dictionary is provided, the method will look for a "messages" key containing a list of message dictionaries to convert.
  • If more than one dictionary or a single dictionary without a "messages" key is provided, the method assumes that the input corresponds to a sequence of dictionaries, each representing a single message to be converted.

TYPE: dict | Sequence[dict]

RETURNS DESCRIPTION
list[BaseMessage]

list[BaseMessage]: A list of BaseMessage objects created from the input dictionaries.
RAISES DESCRIPTION
InvalidMessageFormatError

If any of the input dictionaries cannot be parsed into a valid BaseMessage format, an error will be raised.
Note

This method is a convenience wrapper around langchain_core.messages.utils.convert_to_messages and as such is subject to changes in the langchain-core library.

Source code in src/evallm/readers/utils.py 
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
def messages_from_dicts(dicts: dict | Sequence[dict]) -> list[BaseMessage]:
    """Convert a sequence of dictionaries to a list of `BaseMessage` objects.

    Args:
        dicts (dict | Sequence[dict]): A single or a sequence of dictionaries to be
            converted into `BaseMessage` objects.

            - If a single dictionary is provided, the method will look for a
                `"messages"` key containing a list of message dictionaries to
                convert.
            - If more than one dictionary or a single dictionary without a
                `"messages"` key is provided, the method assumes that the input
                corresponds to a sequence of dictionaries, each representing a
                single message to be converted.

    Returns:
        list[BaseMessage]: A list of BaseMessage objects created from the input
            dictionaries.

    Raises:
        InvalidMessageFormatError: If any of the input dictionaries cannot be
            parsed into a valid `BaseMessage` format, an error will be raised.

    Note:
        This method is a convenience wrapper around
        [:simple-langchain: `langchain_core.messages.utils.convert_to_messages`](https://github.com/langchain-ai/langchain/blob/master/libs/core/langchain_core/messages/utils.py#L735)
        and as such is subject to changes in the langchain-core library.
    """
    try:
        if isinstance(dicts, dict):
            if "messages" in dicts:
                # {"messages":[{"role":"...","content":"..."},...]}"
                return convert_to_messages(dicts["messages"])
            # {"role":"...","content":"..."}
            return convert_to_messages([dicts])
        # [{"role":"...","content":"..."},...]
        return convert_to_messages(dicts)
    except ValueError as e:  # missing key
        raise InvalidMessageFormatError(
            "One or more dictionaries are missing required keys for conversion to "
            "BaseMessage format."
        ) from e
    except NotImplementedError as e:  # pragma: no cover
        # raised by langchain on unsupported message type
        raise InvalidMessageFormatError(
            "One or more dictionaries contain an unsupported message type for "
            "conversion to BaseMessage format."
        ) from e