Benchmarks You Can Keep

Yesterday’s measurements were one-offs: run the benchmark, read the number, change the code. A library that will be maintained needs those measurements to survive — and to complain when somebody undoes them.

A benchmark is part of the source

benchmark.py ships in the repository, next to the tests. That matters for three reasons: anyone can reproduce your claims, the numbers get re-measured on hardware you don’t own, and a change that regresses performance can be demonstrated rather than argued about.

The library’s benchmark is thirty lines and prints this:

2000 rows, cache_size=128

in-memory SQLite (no latency):
  one by one     2000 selects       3.7 ms
  get_many          1 selects       3.5 ms
  ratio: 1.0x

with 0.2 ms of latency per query:
  one by one     2000 selects     636.2 ms
  get_many          1 selects       8.2 ms
  ratio: 77x

Four properties make that output useful rather than decorative.

It states the workload. Rows and cache size are printed, because a number without its inputs is meaningless — and because the next person will change them.

It reports counts as well as times. The selects column is exact and machine-independent; the milliseconds are neither. When someone reports different timings, the counts tell you whether they ran the same thing.

It compares, rather than reporting an absolute. “77x” survives being run on a slower laptop; “8.2 ms” doesn’t.

It models the condition that matters. Without simulated latency this benchmark says the optimisation is pointless. That’s the honest result for in-process SQLite, and the wrong conclusion for a library people will point at a server.

Warm-up, repetition and what to report

import time


def work(size=50_000):
    return sum(index * index for index in range(size))


timings = []
for run in range(7):
    start = time.perf_counter()
    work()
    timings.append((time.perf_counter() - start) * 1000)

print("each run (ms):", " ".join(f"{value:6.2f}" for value in timings))
print(f"\nfirst:   {timings[0]:6.2f} ms")
print(f"best:    {min(timings):6.2f} ms")
print(f"mean:    {sum(timings) / len(timings):6.2f} ms")
print(f"worst:   {max(timings):6.2f} ms")

Here the seven runs agree closely, and the first is not the slowest — a tight loop over integers has little to warm up. Add imports, a connection, a cold file cache or a JIT and the first run can be many times the others, which is often exactly what a user experiences. Which number to report depends on the question:

• Best of N for “how fast can this code be”, the comparison you want when optimising. Interruptions only add time, so the minimum is the least noisy estimate.
• Mean or median of N for “what will a user see”, once the system is warm.
• First run when startup cost is the thing you’re measuring.

Report which one you chose. A number without that is not reproducible.

Turn the important ones into tests

A benchmark tells you the number today. A test tells you when it changed.

The rule from Day 6 applies: assert on counts, not milliseconds. The query counts in this library are its performance contract, and they’re exact:

    def test_get_many_costs_one_query(self) -> None:
        Post.get_many(self.connection, list(range(1, 11)))
        self.assertEqual(self.connection.selects, 1)

    def test_a_partially_cached_request_only_fetches_the_rest(self) -> None:
        Post.get_many(self.connection, [1, 2])
        self.connection.reset()
        Post.get_many(self.connection, [1, 2, 3, 4])
        self.assertEqual(self.connection.selects, 1)
        self.assertEqual(self.connection.queries[-1].count("?"), 2, "only the missing ids")

Those two fail on any change that reintroduces N+1, on any machine, in milliseconds. No wall clock anywhere.

The same trick works for more than queries: count allocations of a class, calls to a mocked dependency, bytes written, HTTP requests made. Whatever the expensive thing is, count it.

import unittest


class Store:
    """Rows, a cache, and a counter of how often it really queried."""

    def __init__(self):
        self.rows = {id: f"row {id}" for id in range(1, 101)}
        self.cache = {}
        self.queries = 0
        self.fetched = []          # the ids each real query asked the database for

    def get(self, id):
        if id in self.cache:
            return self.cache[id]
        self.queries += 1
        self.fetched.append([id])
        self.cache[id] = self.rows[id]
        return self.cache[id]

    def get_many(self, ids):
        missing = [id for id in dict.fromkeys(ids) if id not in self.cache]
        if missing:
            self.queries += 1
            self.fetched.append(missing)
            for id in missing:
                self.cache[id] = self.rows[id]
        return {id: self.cache[id] for id in dict.fromkeys(ids)}

    def invalidate(self, id):
        self.cache.pop(id, None)


class CacheContract(unittest.TestCase):
    def setUp(self):
        self.store = Store()

    def test_a_second_get_is_free(self):
        # TODO: two gets of the same id cost one query
        ...

    def test_get_many_is_one_query(self):
        # TODO: ten uncached ids cost one query
        ...

    def test_a_partially_cached_batch_only_fetches_the_rest(self):
        # TODO: after caching two, a batch of four costs one more query
        ...

    def test_invalidating_forces_a_reread(self):
        # TODO: your own — what does invalidate() owe the caller?
        ...


unittest.TextTestRunner(verbosity=2).run(
    unittest.defaultTestLoader.loadTestsFromTestCase(CacheContract)
)

    def test_a_second_get_is_free(self):
        self.store.get(1)
        self.store.get(1)
        self.assertEqual(self.store.queries, 1)

    def test_get_many_is_one_query(self):
        self.store.get_many(list(range(1, 11)))
        self.assertEqual(self.store.queries, 1)

    def test_a_partially_cached_batch_only_fetches_the_rest(self):
        self.store.get_many([1, 2])
        self.store.get_many([1, 2, 3, 4])
        self.assertEqual(self.store.queries, 2)
        self.assertEqual(self.store.fetched[-1], [3, 4])   # only the missing ids

    def test_invalidating_forces_a_reread(self):
        self.store.get(1)
        self.store.invalidate(1)
        self.store.get(1)
        self.assertEqual(self.store.queries, 2)

What you learned

• Ship the benchmark with the library, print its workload, and report counts beside times.
• Compare ratios rather than absolutes, so numbers survive different hardware.
• Say whether you report best, mean or first — they answer different questions.
• Turn the performance properties that matter into counting tests; keep wall-clock assertions out of the suite.