Summaries as JSON

The analyzer’s report is for people. But the numbers are also wanted by programs: a dashboard that draws the error rate, or an alert that fires when payments errors spike. Those programs shouldn’t have to pick numbers out of a printout. They need data, and on the web and between programs, data usually means JSON.

You met the json module in the beginner course. Today is about what happens at its edges.

What JSON can hold

JSON has six kinds of value, and each maps onto a Python type:

Anything else, like a datetime, a set or your own class, has no JSON equivalent, and json refuses it:

import json
from collections import Counter
from datetime import datetime

summary = {
    "records": 6,
    "levels": Counter({"ERROR": 3, "INFO": 2, "WARNING": 1}),   # a dict subclass: fine
    "sources": ("auth", "payments"),                            # a tuple: becomes a list
    "complete": True,
    "notes": None,
}
print(json.dumps(summary, indent=2))

try:
    json.dumps({"first": datetime(2026, 9, 1, 9, 14, 2)})
except TypeError as error:
    print("TypeError:", error)

The refusal is the helpful case. You find out straight away, and you convert the value yourself, usually a datetime to ISO text with .isoformat(), just as you did for SQLite. The quiet changes are the dangerous ones. The tuple came back as a list, and nothing said so.

Bug hunt: the numbers that stopped matching

The summary gains an errors-per-hour count, a dict from hour to count. A test checks the summary survives being saved and loaded:

import json
import sys
import unittest
from collections import Counter


def errors_per_hour(hours):
    """How many errors started in each hour of the day."""
    return dict(Counter(hours))


def save_summary(summary, path):
    with open(path, "w") as file:
        json.dump(summary, file, indent=2)


def load_summary(path):
    with open(path) as file:
        return json.load(file)


class TestSummaryFile(unittest.TestCase):
    def test_summary_survives_a_round_trip(self):
        summary = {"records": 6, "errors_per_hour": errors_per_hour([9, 9, 10])}
        save_summary(summary, "summary.json")
        self.assertEqual(load_summary("summary.json"), summary)


suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestSummaryFile)
unittest.TextTestRunner(stream=sys.stdout, verbosity=2).run(suite)

Reproduce. One small summary, and the round trip fails.

Inspect. Compare the two sides of the failure: {'9': 2, '10': 1} against {9: 2, 10: 1}. The counts are right. The keys changed type. To see where, look at the file itself. Add this at the end of the playground:

print(open("summary.json").read())

In the file the keys are already "9" and "10", in quotes.

Before you open this, make the round trip pass without changing the test. There’s more than one way. Then compare with ours, which is about choosing the file’s shape, not just converting values.

def errors_per_hour(hours):
    """How many errors started in each hour of the day, in hour order."""
    return [{"hour": hour, "errors": count} for hour, count in sorted(Counter(hours).items())]

"errors_per_hour": [
  {"hour": 9, "errors": 2},
  {"hour": 10, "errors": 1}
]

The Day 3 summary

store.py builds the whole summary straight from the database, in the shape above. SQL does the counting, and substr(timestamp, 12, 2) picks the hour out of the ISO text:

def summary(conn):
    """Everything the daily report needs, as plain JSON-ready values."""
    total, first, last = conn.execute(
        "SELECT COUNT(*), MIN(timestamp), MAX(timestamp) FROM records").fetchone()
    hours = conn.execute(
        "SELECT CAST(substr(timestamp, 12, 2) AS INTEGER) AS hour, COUNT(*) AS n "
        "FROM records WHERE level = 'ERROR' GROUP BY hour ORDER BY hour")
    return {
        "records": total,
        "first": first,
        "last": last,
        "levels": count_by(conn, "level"),
        "errors_by_source": count_by(conn, "source", level="ERROR"),
        "errors_by_hour": [{"hour": row["hour"], "errors": row["n"]} for row in hours],
    }

first and last come out of the database as ISO text already, so they go straight into JSON. levels and errors_by_source stay dicts: their keys are names, which are strings already, so they survive the round trip unchanged. By the callout’s advice they could be lists of objects too. Keeping them as dicts is a judgement call, made on purpose because every reader so far is a small script. That’s the difference between a choice and an accident.

Day 3 checkpoint

def save_requests(conn, requests):
    """Store requests, skipping line numbers already stored. Return how many were new."""
    rows = ((r.line_number, r.ip, r.timestamp.isoformat(), r.method, r.path, r.status, r.size)
            for r in requests)
    with conn:
        cursor = conn.executemany(
            "INSERT INTO requests VALUES (?, ?, ?, ?, ?, ?, ?) "
            "ON CONFLICT (line_number) DO NOTHING", rows)
    return cursor.rowcount


def status_counts(conn):
    """{status: number of requests}, in status order."""
    rows = conn.execute(
        "SELECT status, COUNT(*) AS n FROM requests GROUP BY status ORDER BY status")
    return {row["status"]: row["n"] for row in rows}


def error_paths(conn, min_status=400):
    """Distinct paths that got a status of min_status or above, sorted."""
    rows = conn.execute(
        "SELECT DISTINCT path FROM requests WHERE status >= ? ORDER BY path", (min_status,))
    return [row["path"] for row in rows]


def export_summary(conn, path):
    """Write the summary the tests describe as JSON to path."""
    total = conn.execute("SELECT COUNT(*) FROM requests").fetchone()[0]
    data = {
        "requests": total,
        "statuses": [{"status": status, "count": count}
                     for status, count in status_counts(conn).items()],
        "error_paths": error_paths(conn),
    }
    with open(path, "w") as file:
        json.dump(data, file, indent=2)

The Day 3 reference solution

The analyzer at the end of Day 3, and where Day 4 starts. main.py parses messy.log, stores the records in an in-memory database, prints a count straight from SQL, writes summary.json and shows it, then runs all nineteen tests.

Working on your own machine? Download the Day 3 files and carry on there. You need Python 3.12 or later, and nothing else.

Going further: teaching json new types

If you write the same kinds of value to JSON in many places, you can teach json to convert them. Subclass JSONEncoder and override default, which json calls for any value it doesn’t know:

import json
from datetime import datetime


class SummaryEncoder(json.JSONEncoder):
    """Also knows how to write datetimes and sets."""

    def default(self, value):
        if isinstance(value, datetime):
            return value.isoformat()
        if isinstance(value, set):
            return sorted(value)
        return super().default(value)       # anything else: the usual TypeError


data = {"first": datetime(2026, 9, 1, 9, 14, 2), "sources": {"search", "auth"}}
print(json.dumps(data, cls=SummaryEncoder))
print(json.dumps(data, default=str))       # the quick version: str() on anything unknown

Compare the two lines of output. default=str is tempting because it turns almost anything into something, and that’s the problem: the set comes out as one string that looks like Python, which no other program can use. The encoder converts only the types it knows, and still refuses everything else. The same rule as allow-lists in SQL: say what’s allowed, and reject the rest.

Going the other way is harder. JSON has no way to mark “this string is a date”, so turning ISO strings back into datetimes is up to the reader, which is one more reason to keep file formats simple.

What you can now do

• Predict which Python values JSON keeps, changes, or refuses.
• Test that saved data comes back equal, and find what didn’t.
• Design a JSON shape with fixed keys and typed values, rather than data as keys.
• Build a report from SQL and export it for other programs.

Tomorrow the analyzer becomes configurable: which lines to keep and how to group them become arguments, and those arguments are functions.