Type Hints That Help

The last lesson ended on an assumption that lived only in someone’s head. ShareSection.render calls len(records) and reads the records twice, so it needs a list, not a stream. Nothing in the code said so. The previous bug hunt happened because nothing said so.

Type hints are how Python code says so. You’ve seen them on dataclass fields since the beginner course. Today they go on the analyzer’s functions.

Hints are for readers, and for tools

Here’s count with hints:

def count(
    records: Iterable[LogRecord],
    key: Callable[[LogRecord], Hashable],
    keep: Predicate | None = None,
) -> Counter[Hashable]:

Without reading its body, you know it takes any iterable of records, a function from a record to something it can count by, and an optional filter, and that it gives back a Counter. That’s the job of a signature, and hints make it complete.

Python itself doesn’t check them at all:

Hints are not checked when the code runs Python

from collections import Counter
from collections.abc import Callable, Hashable, Iterable


def count(
    records: Iterable[str], key: Callable[[str], Hashable], keep: Callable[[str], bool] | None = None
) -> Counter[Hashable]:
    """Count records by key(record), counting only those where keep(record) is true."""
    return Counter(key(r) for r in records if keep is None or keep(r))


print(count.__annotations__)

# Python doesn't check hints when it runs. This call breaks every one of them...
print(count([1, 2, 3], key=str))
# ...and this one fails, but only when count gets round to calling the "function":
print(count(["a"], key="not a function"))

The hints are stored on the function, and that’s all Python does with them. They’re checked before the program runs, by a type checker such as mypy or pyright, and by your editor, which uses them to autocomplete, underline mistakes and show you a function’s signature as you type.

The types you’ll use most

Hint	Means
`int`, `str`, `bool`, `LogRecord`	an instance of that class
`list[str]`, `dict[str, int]`, `set[str]`	a container, and what’s inside it
`tuple[str, int]`	exactly two items: a `str`, then an `int`
`X \| None`	an `X`, or `None`. Say it whenever `None` is possible.
`Iterable[X]`	anything a `for` loop can read once: a list, a generator, a file…
`Sequence[X]`	something with a length and indexes that can be read many times: a list, a tuple
`Iterator[X]`	what a generator function returns
`Callable[[A, B], R]`	a function taking an `A` and a `B`, returning an `R`

Iterable, Sequence, Iterator and Callable come from collections.abc. They’re abstract: they describe what an object can do, not which class it is.

Choosing the types

A hint is a promise in both directions, so choose it deliberately:

Accept the widest type that works. count only loops over records once, so Iterable[LogRecord] is right. list[LogRecord] would make a type checker reject a generator, for no reason.
Return the most specific type you can. count returns a Counter, so say Counter, and callers can use .most_common() without a checker complaining.
Say when you need more. ShareSection.render reads its records twice, so it takes Sequence[LogRecord]. Now last lesson’s assumption is written down, and a checker rejects the generator before the bug can happen.

That last point is the payoff. Report.render takes an Iterable and turns it into a list. Each section’s render takes a Sequence. The hints now describe the design: one place reads the stream, and everything after it can re-read.

Naming a type

When the same hint appears in many places, give it a name. Callable[[LogRecord], bool] is every filter from yesterday, so the analyzer names it once:

type Predicate = Callable[[LogRecord], bool]


def level_at_least(minimum: Level | str) -> Predicate: ...
def all_of(*predicates: Predicate) -> Predicate: ...

The type statement (Python 3.12 and later) creates a type alias. It’s purely a name for readers and checkers, and it makes the signatures read in the analyzer’s own vocabulary.

Generic functions

The Day 5 reference goes one step further and writes def count[K: Hashable](records, key: Callable[[LogRecord], K], ...) -> Counter[K]. The [K: Hashable] makes count generic. K is whatever type key returns, so count(records, key=lambda r: r.timestamp.hour) is known to return a Counter[int]. You don’t need to write generics to benefit from them. Most of the standard library already is generic, which is how list[str] works.

What a checker catches

Here’s a file with three mistakes that would all run without complaint until the moment they failed, or worse, didn’t:

from analyzer import read_lines
from filters import level_at_least
from report import CountSection, Report

daily = Report("Daily", [CountSection("By source", key=lambda r: r.sorce)])
print(daily.render(read_lines("messy.log")))
keep = level_at_least(30)

With the Day 5 reference beside it, mypy reports each one without running anything:

mistakes.py:5: error: "LogRecord" has no attribute "sorce"; maybe "source"?  [attr-defined]
mistakes.py:6: error: Argument 1 to "render" of "Report" has incompatible type "Iterator[str]"; expected "Iterable[LogRecord]"  [arg-type]
mistakes.py:7: error: Argument 1 to "level_at_least" has incompatible type "int"; expected "Level | str"  [arg-type]
Found 3 errors in 1 file (checked 1 source file)

The typo inside a lambda is caught because the checker knows a CountSection’s key receives a LogRecord. The second mistake passes lines where records belong, which would otherwise have failed inside a section, far from here. The third passes a number where a level name belongs.

And here’s what it can’t catch: code whose types are all correct and whose meaning is wrong. record.level >= "WARNING" compares a str with a str, which is perfectly well typed. Whether it means what you think is the subject of the next lesson.

Try it on your machine (optional)

With Python installed, put the Day 5 reference files in a folder, then run:

python -m pip install mypy

python -m mypy analyzer.py filters.py report.py

Add the three-mistake file above as mistakes.py and run python -m mypy mistakes.py to see the output for yourself. mypy’s --strict flag also insists that every function has hints. It’s a good setting for new code.

Your turn

Exercise

Hint five functions

Add type hints to the five functions below, without changing what they do. Decide on each hint yourself before reading the tests, using this lesson’s rules: the widest type that works for parameters, the most specific for return values, and | None wherever None can come back.

Then run the tests. They compare your hints with the interface this exercise asks for, and a failure shows yours next to the expected ones. Equivalent spellings count as the same, like typing.List[str] for list[str], or None | LogRecord for LogRecord | None.

Be clear about what these tests check: the shape of your annotations, nothing more. They can’t tell whether the code actually keeps the promises the hints make. That’s what a type checker does, by reading the code as well as the hints, and it’s what the optional mypy exercise above is for.

import sys
import types
import typing
import unittest
from collections.abc import Callable, Generator, Iterable, Iterator
from dataclasses import dataclass
from datetime import datetime
from typing import get_args, get_origin, get_type_hints


@dataclass(frozen=True)
class LogRecord:
    timestamp: datetime
    level: str
    source: str


# Add type hints to these five functions. Don't change what they do.

def sources(records):
    """The distinct sources, sorted."""
    return sorted({r.source for r in records})


def first_error(records):
    """The first ERROR record, or None if there isn't one."""
    for record in records:
        if record.level == "ERROR":
            return record
    return None


def by_hour(records):
    """Records grouped by the hour they happened in."""
    groups = {}
    for record in records:
        groups.setdefault(record.timestamp.hour, []).append(record)
    return groups


def only(level):
    """A filter that keeps records at exactly this level."""
    def keep(record):
        return record.level == level
    return keep


def errors(records):
    """Yield the ERROR records, one at a time."""
    for record in records:
        if record.level == "ERROR":
            yield record


def shape(hint):
    """A hint's structure, so typing.List[str] and list[str] compare equal,
    and so do X | None and None | X."""
    args = get_args(hint)
    if not args:
        return hint
    if get_origin(hint) in (typing.Union, types.UnionType):
        return frozenset(map(shape, args))
    return get_origin(hint), tuple(tuple(map(shape, a)) if isinstance(a, list) else shape(a)
                                   for a in args)


class TestHints(unittest.TestCase):
    def check(self, function, *allowed):
        yours = get_type_hints(function)
        shapes = [{name: shape(hint) for name, hint in hints.items()} for hints in allowed]
        if {name: shape(hint) for name, hint in yours.items()} not in shapes:
            self.fail(f"\n  yours:    {yours}\n  expected: {allowed[0]}")

    def test_sources(self):
        self.check(sources, {"records": Iterable[LogRecord], "return": list[str]})

    def test_first_error(self):
        self.check(first_error, {"records": Iterable[LogRecord], "return": LogRecord | None})

    def test_by_hour(self):
        self.check(by_hour, {"records": Iterable[LogRecord],
                             "return": dict[int, list[LogRecord]]})

    def test_only(self):
        self.check(only, {"level": str, "return": Callable[[LogRecord], bool]})

    def test_errors(self):
        self.check(errors, {"records": Iterable[LogRecord], "return": Iterator[LogRecord]},
                   {"records": Iterable[LogRecord], "return": Generator[LogRecord, None, None]})


suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestHints)
unittest.TextTestRunner(stream=sys.stdout, verbosity=2).run(suite)

Hint 1: the idea

Read each function twice. Once for what it accepts: the widest type that works — does it need a list, or only something it can loop over? Once for what it returns: the most specific type, with | None if None can come back. only returns a function, so its return type is a Callable; errors yields, so it’s a generator.

Hint 2: a first step

Start with sources. It loops over records exactly once, so an Iterable[LogRecord] is enough for the parameter; and sorted always returns a list — of what? Then first_error, whose return has two possibilities.

Show one solution

def sources(records: Iterable[LogRecord]) -> list[str]:
    """The distinct sources, sorted."""
    return sorted({r.source for r in records})


def first_error(records: Iterable[LogRecord]) -> LogRecord | None:
    """The first ERROR record, or None if there isn't one."""
    for record in records:
        if record.level == "ERROR":
            return record
    return None


def by_hour(records: Iterable[LogRecord]) -> dict[int, list[LogRecord]]:
    """Records grouped by the hour they happened in."""
    groups: dict[int, list[LogRecord]] = {}
    for record in records:
        groups.setdefault(record.timestamp.hour, []).append(record)
    return groups


def only(level: str) -> Callable[[LogRecord], bool]:
    """A filter that keeps records at exactly this level."""
    def keep(record: LogRecord) -> bool:
        return record.level == level
    return keep


def errors(records: Iterable[LogRecord]) -> Iterator[LogRecord]:
    """Yield the ERROR records, one at a time."""
    for record in records:
        if record.level == "ERROR":
            yield record

Every function reads its records once, so every one takes Iterable. If you wrote list[LogRecord], the code still works, but a checker would reject every caller who passes a generator.

first_error is the one people most often get wrong: -> LogRecord looks right until you notice the return None. | None makes every caller handle the empty case. A checker will reject first_error(records).source until you do.

by_hour also needs a hint inside it. groups = {} gives the checker nothing to go on, so groups: dict[int, list[LogRecord]] = {} says what it will hold. That’s the one place a local variable needs a hint: when it starts empty.

sorted(...) always returns a list, whatever it’s given, so sources promises exactly that. And a generator function’s return type describes what calling it gives you, an Iterator, not what each yield produces. (Generator[LogRecord, None, None] is also correct. It additionally says the generator accepts nothing through send() and returns nothing at the end.)

Going further: Protocols

Optional

Nothing later in the course depends on this section.

Last lesson, Report accepted any object with a title and a render. That’s duck typing, and a Protocol writes it down as a type:

from typing import Protocol


class Section(Protocol):
    """Anything with a title and a render method can be part of a Report."""

    @property
    def title(self) -> str: ...     # read-only, so frozen dataclasses qualify

    def render(self, records: Sequence[LogRecord]) -> list[str]: ...


@dataclass(frozen=True)
class Report:
    title: str
    sections: Sequence[Section]

CountSection never mentions Section, and doesn’t need to. A checker compares its shape with the protocol: a title, and a render with the right signature. That’s structural typing, duck typing a checker can verify.

The @property is there for a reason. Declared as plain title: str, the protocol would require a title you can assign to, and a frozen dataclass’s title is read-only, so mypy rejects every section with “Protocol member Section.title expected settable variable, got read-only attribute”. A read-only property asks only for what Report actually does, which is read it.

What you can now do

Add hints to function signatures with built-in generics, | None and collections.abc types.
Choose Iterable or Sequence according to how the function really uses its argument, and return the most specific type you can.
Name a repeated type with a type alias.
Know what a type checker catches, and what it can’t: code that’s well typed and still wrong.

Next: a report that undercounts by half, and every type in it is correct.