A Safety Net for a Script

You’re about to change every line of report.py. You’ll make mistakes while you do it, because everyone does. What matters is how quickly you find out. The answer should be “within a second, every time”, and that takes a test.

But the script has no functions to test. It’s one block of code that reads a file and prints. So the first test checks the only thing the script offers: its output.

A characterization test

A characterization test records what code does now and fails if that changes. It doesn’t judge whether the behaviour is right. If the script has a bug, the test pins the bug too, and that’s deliberate. While you restructure, “unchanged” is the only thing you’re checking.

Writing one takes three steps: run the script, copy what it prints, and assert that it keeps printing exactly that.

The safety net Python

# report.py - counts what happened in a log. It works. Nobody dares touch it. f = open("sample.log") data = f.read() f.close() c = {} e = {} n = 0 for l in data.split("\n"): if l.strip() == "": continue n = n + 1 w = l.split() if w[2] in c: c[w[2]] = c[w[2]] + 1 else: c[w[2]] = 1 if w[2] == "ERROR": s = w[3][:-1] if s in e: e[s] = e[s] + 1 else: e[s] = 1 print("Lines read:", n) print("Levels:", c) print("Errors by source:", e)

import io import runpy import unittest from contextlib import redirect_stdout # What report.py printed on the day this test was written. Not what it *should* # print - what it *does* print. The test's job is to notice any change. EXPECTED = """\ Lines read: 6 Levels: {'INFO': 3, 'WARNING': 1, 'ERROR': 2} Errors by source: {'payments': 1, 'search': 1} """ def run_script(path): """Run a script as if from the command line, and return what it printed.""" output = io.StringIO() with redirect_stdout(output): runpy.run_path(path, run_name="__main__") return output.getvalue() class TestReport(unittest.TestCase): def test_output_is_unchanged(self): self.assertEqual(run_script("report.py"), EXPECTED)

import sys import unittest import test_report suite = unittest.defaultTestLoader.loadTestsFromModule(test_report) unittest.TextTestRunner(stream=sys.stdout, verbosity=2).run(suite)

2026-09-01 09:14:02 INFO auth: user=ada action=login 2026-09-01 09:14:07 WARNING payments: order=1042 amount=19.99 retry=1 2026-09-01 09:15:30 ERROR payments: order=1043 msg="card declined" 2026-09-01 09:16:11 INFO auth: user=alan action=logout 2026-09-01 09:17:45 ERROR search: query="python books" msg="index offline" 2026-09-01 09:18:02 INFO search: query=dune results=4

Open test_report.py. Two standard-library tools do the work:

runpy.run_path(path, run_name="__main__") runs a Python file exactly as python report.py would. run_name="__main__" matters later today, once the script has an if __name__ == "__main__": guard.
redirect_stdout sends everything print writes into a StringIO, an in-memory text file, instead of the screen. getvalue() hands it back as one string.

EXPECTED uses """\ so the string starts on the next line without a leading newline. It’s the output copied character for character, trailing newline included.

Why the tests end with a runner

On your own machine you’d run a test file with python -m unittest or pytest. In this browser runtime unittest.main() can’t find the tests, so every multi-file playground in this course has a small run_tests.py that loads the test module and runs it. The tests themselves are exactly what you’d write at home.

A test you haven’t seen fail

Here’s a question worth asking about any new test: would it notice if something broke? A test that has only ever passed hasn’t proved anything yet.

Try it. In the playground above, open report.py, change "Levels:" to "Level:", and run the tests again. The test fails, and the diff underneath points straight at the missing s:

- Level: {'INFO': 3, 'WARNING': 1, 'ERROR': 2}
+ Levels: {'INFO': 3, 'WARNING': 1, 'ERROR': 2}
?      +

That’s what you want from a safety net. Now change it back. Deliberately breaking the code to check that a test catches it is a habit worth keeping. It takes ten seconds, and it’s the only way to know that a test is really testing.

Bug hunt: the net with a hole in it

Every day this course includes a bug hunt, done the same way every time: reproduce the problem with a small input, inspect the failure and the relevant state, narrow it down to the cause, pin it with a test, then fix it.

A colleague wrote their own safety net for the same script. Yesterday, someone deleted the last line of report.py by accident, and the report went out without its errors. The safety net passed the whole time:

A safety net that passes Python

import sys import unittest import test_report suite = unittest.defaultTestLoader.loadTestsFromModule(test_report) unittest.TextTestRunner(stream=sys.stdout, verbosity=2).run(suite)

Reproduce. Run it: OK. Open report.py: the Errors by source line is gone. The script is broken and the test is green. That’s the whole failure.

Inspect. When a test passes and it shouldn’t, the test is the bug. Read the one line that matters in test_report.py:

self.assertTrue(run_script("report.py"), EXPECTED)

Narrow. What does assertTrue do with two arguments? Ask it, in the smallest possible experiment:

What assertTrue checks Python

import inspect
import unittest

check = unittest.TestCase()
check.assertTrue("any text at all", "the expected output")
print("passed")

print(inspect.signature(unittest.TestCase.assertTrue))
print(unittest.TestCase.assertTrue.__doc__)

assertTrue(expr, msg=None) checks that its first argument is truthy. The second is the message to show if it fails. Any non-empty string is truthy, so this test passes whatever the script prints, as long as it prints something. It never compared the two at all.

Pin it. This time, the regression test is the test itself, and “pinned” means you’ve seen it fail. Change assertTrue to assertEqual and run it. It now fails, showing the missing line.

Fix. Put print("Errors by source:", e) back at the end of report.py and run the tests again. They pass, and this time a pass means something.

Use the most specific assertion

assertEqual(a, b) fails with a diff showing exactly what differs. assertTrue(a == b) fails with False is not true, which tells you nothing. And assertTrue(a, b) doesn’t compare at all. Reach for assertEqual, assertIn, assertIsNone and assertRaises first.

What the net doesn’t catch

A characterization test is coarse. It checks one run on one input file. It can’t tell you which part broke, and it can’t check anything sample.log doesn’t contain. It never sees a line with no colon, for example.

That’s fine for its job, which is to catch you changing the output while restructuring. Finer-grained tests come once there are functions to call, in the next lesson.

Exercise

Try to sneak a change past the net

Make each of these changes to report.py in the first playground (the one with assertEqual), and predict before you run whether the safety net catches it:

Rename c to level_counts everywhere.
Count levels with collections.Counter instead of the if/else.
Remove the if l.strip() == "": continue check.
Change data.split("\n") to data.splitlines().

Hint 1: the idea

The net compares the printed text, character for character, with what the script printed before. So for each change the only question is: does the output text change? Getting the numbers right isn’t enough if they print differently. And a change can also make the script crash, which the net reports as an error rather than a failure.

Hint 2: a first step

Try each change on the smallest input you can before running the test. For change 2, compare print(Counter(["a", "a"])) with print({"a": 2}). For change 3, ask what "".split()[2] does.

Show one solution

Passes. Renaming is a pure refactor: the output is identical.
Fails. Counter counts correctly, but prints as Counter({'INFO': 3, ...}), not {'INFO': 3, ...}. It’s a real change in output, even though the numbers are the same. dict(counter) would print the same as before.
Fails, and not with a diff. The blank line has no third word, so w[2] raises IndexError, and the test reports an error rather than a failure.
Passes. Both split this file into the same lines. They’d differ on a file that ends lines with a bare \r, which splitlines() treats as a line break and split("\n") doesn’t. sample.log has none, so the net can’t tell. It’s a reminder that a net made from one input file only catches what that file exercises.

What you can now do

Put an untested script under a characterization test with runpy and redirect_stdout.
Check that a test can fail before you trust it, by breaking the code on purpose.
Run the bug-hunt routine: reproduce, inspect, narrow, pin, fix.
Choose assertions that fail informatively, and spot one that can’t fail.

Next: the actual refactor, one small step at a time.