Day 2 Checkpoint: Fields in the Library

Yesterday’s model worked and asked too much of its users: list your columns, and hope nobody assigns nonsense. Today’s does neither.

class Post(Model, table="post"):
    title = TextField(max_length=100)
    views = IntegerField(minimum=0)

That class body is now the whole specification. The fields validate, the columns are derived from them, the table name comes from the header, and the model registers itself with its own instance cache.

The checkpoint

models.py is written for you — it’s the __init_subclass__ from the last lesson, applied to the real library. fields.py is not: the descriptor protocol and the integer validator are yours to write.

Checkpoint

Day 2 checkpoint: make the fields work

Twelve tests, in three groups: the fields themselves, what __init_subclass__ set up, and the save/load round trip from Day 1 still working. Start with __set_name__ — until it runs, every field on a model shares one slot and they overwrite each other.

fields.py Python

"""Fields: one descriptor per column, validating what callers assign.""" class Field: """A column, and the rules for what may go in it. Assigned in a class body, so `__set_name__` learns the attribute name the developer chose. Values live in the *instance* dict under a private key; the descriptor itself is shared by every instance of the model. """ def __init__(self, *, null=False): self.null = null self.name = "<unnamed>" self.slot = "<unnamed>" def __set_name__(self, owner, name): """Python calls this when the class body is executed. TODO: remember the attribute name in self.name, and the key to store values under in self.slot (use "_" + name). Until you do, every field on a model shares one slot, and they overwrite each other. """ def __get__(self, instance, owner=None): """TODO: return the field itself when instance is None (Post.title), otherwise the value from instance.__dict__[self.slot]. A field that was never set should raise AttributeError naming the model and the field. """ raise NotImplementedError("__get__") def __set__(self, instance, value): """TODO: store self.check(value) in the instance dict, under self.slot.""" raise NotImplementedError("__set__") def check(self, value): """Return the value to store, or raise ValueError.""" if value is None and not self.null: raise ValueError(f"{self.name} may not be None") return value def __repr__(self): return f"{type(self).__name__}({self.name!r})" class TextField(Field): def __init__(self, *, max_length=None, **kwargs): super().__init__(**kwargs) self.max_length = max_length def check(self, value): value = super().check(value) if value is None: return None if not isinstance(value, str): raise ValueError(f"{self.name} must be a str, not {type(value).__name__}") if self.max_length is not None and len(value) > self.max_length: raise ValueError(f"{self.name} is longer than {self.max_length} characters") return value class IntegerField(Field): def __init__(self, *, minimum=None, **kwargs): super().__init__(**kwargs) self.minimum = minimum def check(self, value): """TODO: let None through when the field allows it (super().check does that), reject anything that isn't an int, reject a value below self.minimum, and mind that bool is a subclass of int. """ raise NotImplementedError("IntegerField.check")

"""The base model, now that fields describe themselves. Yesterday a subclass listed its columns by hand and shared one cache with every other model. Today `__init_subclass__` does both jobs at the moment the class is created, so a model can't be defined wrongly. """ from fields import Field, IntegerField, TextField class Model: #: Every model, by table name. Filled in by __init_subclass__. registry = {} def __init_subclass__(cls, table=None, **kwargs): super().__init_subclass__(**kwargs) cls.table = table or cls.__name__.lower() # Fields from base classes first, so a subclass can override one. cls.fields = { name: value for klass in reversed(cls.__mro__) for name, value in vars(klass).items() if isinstance(value, Field) } cls._instances = {} # one cache per model, not one for all of them Model.registry[cls.table] = cls def __init__(self, id=None, **values): self.id = id for name in self.fields: if name in values: setattr(self, name, values.pop(name)) if values: raise TypeError(f"{type(self).__name__} has no column {sorted(values)[0]!r}") @property def columns(self): return tuple(self.fields) def save(self, connection): columns = tuple(self.fields) values = [getattr(self, name) for name in columns] names = ", ".join(columns) if self.id is None: marks = ", ".join("?" for _ in columns) cursor = connection.execute( f"insert into {self.table} ({names}) values ({marks})", values ) self.id = cursor.lastrowid else: assignments = ", ".join(f"{name} = ?" for name in columns) connection.execute( f"update {self.table} set {assignments} where id = ?", [*values, self.id] ) connection.commit() type(self)._instances[self.id] = self return self @classmethod def get(cls, connection, id): cached = cls._instances.get(id) if cached is not None: return cached row = connection.execute(f"select * from {cls.table} where id = ?", (id,)).fetchone() if row is None: raise LookupError(f"no {cls.__name__} with id {id}") found = cls(**dict(row)) cls._instances[id] = found return found def __repr__(self): shown = ", ".join(f"{name}={getattr(self, name, None)!r}" for name in self.fields) return f"{type(self).__name__}(id={self.id!r}, {shown})" class Post(Model, table="post"): title = TextField(max_length=100) views = IntegerField(minimum=0) class User(Model, table="user"): email = TextField(max_length=200)

"""SQLite scaffolding, supplied by the course. Day 1 is about Python's object model, not about SQL, so this file is given to you: a connection with the library's two tables already created. The rest of the week replaces almost everything else, but never this. """ import sqlite3 SCHEMA = """ create table if not exists post ( id integer primary key autoincrement, title text not null, views integer not null default 0 ); create table if not exists user ( id integer primary key autoincrement, email text not null ); """ def connect(path=":memory:"): """A connection whose rows behave like dictionaries.""" connection = sqlite3.connect(path) connection.row_factory = sqlite3.Row connection.executescript(SCHEMA) return connection

import unittest from db import connect from fields import IntegerField, TextField from models import Model, Post, User class Fields(unittest.TestCase): def test_a_field_validates_on_assignment(self): post = Post(title="Hello", views=0) with self.assertRaises(ValueError): post.views = -1 with self.assertRaises(ValueError): post.title = 42 self.assertEqual(post.views, 0, "the rejected value was not stored") def test_true_is_not_an_integer(self): with self.assertRaises(ValueError): Post(title="Hello", views=True) def test_values_are_per_instance(self): first = Post(title="First", views=1) second = Post(title="Second", views=2) self.assertEqual((first.title, second.title), ("First", "Second")) def test_reading_an_unset_field_says_so(self): post = Post(title="Hello") with self.assertRaises(AttributeError) as caught: post.views self.assertIn("views", str(caught.exception)) def test_the_class_attribute_is_the_field_itself(self): self.assertIsInstance(Post.title, TextField) self.assertIsInstance(Post.views, IntegerField) class Subclassing(unittest.TestCase): def test_fields_are_collected_from_the_class_body(self): self.assertEqual(tuple(Post.fields), ("title", "views")) def test_the_table_name_comes_from_the_class_keyword(self): self.assertEqual(Post.table, "post") def test_every_model_is_registered(self): self.assertIs(Model.registry["post"], Post) self.assertIs(Model.registry["user"], User) def test_each_model_has_its_own_cache(self): self.assertIsNot(Post._instances, User._instances) class SaveAndLoad(unittest.TestCase): def setUp(self): self.connection = connect() for model in Model.registry.values(): model._instances.clear() def test_round_trip(self): post = Post(title="Hello", views=3).save(self.connection) loaded = Post.get(self.connection, post.id) self.assertEqual((loaded.title, loaded.views), ("Hello", 3)) def test_two_models_with_the_same_id_stay_apart(self): Post(title="Hello", views=0).save(self.connection) User(email="someone@example.com").save(self.connection) self.assertIsInstance(Post.get(self.connection, 1), Post) self.assertIsInstance(User.get(self.connection, 1), User) def test_saving_twice_updates(self): post = Post(title="Hello", views=3).save(self.connection) post.views = 4 post.save(self.connection) self.assertEqual(self.connection.execute("select count(*) from post").fetchone()[0], 1) self.assertEqual(self.connection.execute("select views from post").fetchone()[0], 4)

import sys import unittest import test_models suite = unittest.defaultTestLoader.loadTestsFromModule(test_models) unittest.TextTestRunner(stream=sys.stdout, verbosity=2).run(suite)

Hint 1: the idea

This morning’s descriptor, three methods over. __set_name__ gives each field its own storage slot. __get__ reads it — returning the field itself through the class, and raising AttributeError for a value that was never set. __set__ stores whatever check returns. The integer rule lives in IntegerField.check, which runs the parent’s None rule first.

Hint 2: a first step

Write __set_name__ and run: many field tests change at once, because the fields stop sharing a slot. Then __get__ and __set__. For the integer check, the starter’s TODO is the warning to heed: True is an int in Python, so a plain isinstance lets it through.

Show one solution

    def __set_name__(self, owner, name):
        self.name = name
        self.slot = f"_{name}"

    def __get__(self, instance, owner=None):
        if instance is None:
            return self
        try:
            return instance.__dict__[self.slot]
        except KeyError:
            raise AttributeError(f"{type(instance).__name__}.{self.name} is not set") from None

    def __set__(self, instance, value):
        instance.__dict__[self.slot] = self.check(value)

and the integer rule:

    def check(self, value):
        value = super().check(value)
        if value is None:
            return None
        if isinstance(value, bool) or not isinstance(value, int):
            raise ValueError(f"{self.name} must be an int, not {type(value).__name__}")
        if self.minimum is not None and value < self.minimum:
            raise ValueError(f"{self.name} must be at least {self.minimum}")
        return value

raise ... from None in __get__ is worth a moment. Without it, the AttributeError arrives with KeyError: '_views' chained underneath it — an internal detail of how the field stores things, in the traceback of someone who just forgot to set a value. from None drops the chain. Day 4 is about exception chaining in general, and this is the case where hiding the cause is right: it isn’t a cause, it’s an implementation.

Bug hunt: every post has the same title

A field stores its value somewhere. This version stores it in the obvious wrong place, and the tests notice immediately.

Reproduce Python

"""Fields: one descriptor per column, validating what callers assign.""" class Field: """A column, and the rules for what may go in it. Assigned in a class body, so `__set_name__` learns the attribute name the developer chose. Values live in the *instance* dict under a private key; the descriptor itself is shared by every instance of the model. """ def __init__(self, *, null=False): self.null = null self.name = "<unnamed>" self.slot = "<unnamed>" def __set_name__(self, owner, name): self.name = name self.slot = f"_{name}" def __get__(self, instance, owner=None): if instance is None: return self # Post.title is the field itself try: return self.value except AttributeError: raise AttributeError(f"{type(instance).__name__}.{self.name} is not set") from None def __set__(self, instance, value): self.value = self.check(value) def check(self, value): """Return the value to store, or raise ValueError.""" if value is None and not self.null: raise ValueError(f"{self.name} may not be None") return value def __repr__(self): return f"{type(self).__name__}({self.name!r})" class TextField(Field): def __init__(self, *, max_length=None, **kwargs): super().__init__(**kwargs) self.max_length = max_length def check(self, value): value = super().check(value) if value is None: return None if not isinstance(value, str): raise ValueError(f"{self.name} must be a str, not {type(value).__name__}") if self.max_length is not None and len(value) > self.max_length: raise ValueError(f"{self.name} is longer than {self.max_length} characters") return value class IntegerField(Field): def __init__(self, *, minimum=None, **kwargs): super().__init__(**kwargs) self.minimum = minimum def check(self, value): value = super().check(value) if value is None: return None # bool is a subclass of int, and True in an integer column is a mistake. if isinstance(value, bool) or not isinstance(value, int): raise ValueError(f"{self.name} must be an int, not {type(value).__name__}") if self.minimum is not None and value < self.minimum: raise ValueError(f"{self.name} must be at least {self.minimum}") return value

import sys import unittest import test_models suite = unittest.defaultTestLoader.loadTestsFromModule(test_models) unittest.TextTestRunner(stream=sys.stdout, verbosity=2).run(suite)

Reproduce. Two failures, and the clearer one is:

AssertionError: Tuples differ: ('Second', 'Second') != ('First', 'Second')

Two posts were created with different titles. Both report the second one.

Inspect. Add this to the failing test:

print(Post.title, id(Post.title))
print(vars(first), vars(second))

Each instance holds only {'id': None}, and the title is on the field object. That’s the whole bug in one line of output: the fields keep no per-instance state at all.

Narrow. Post.title is one TextField, created once when the class body ran. __set__ writes self.value, and self is that shared field. Every post writes to the same attribute, so the last assignment wins for everyone. The second failure — an unset field not raising AttributeError — has the same cause: once any post has set a title, self.value exists forever.

Pin. Both failing tests are the regression tests. They’re the two questions worth asking about any descriptor: are two instances independent, and does an unset value still look unset?

Fix. Put the value where the instance is:

    def __get__(self, instance, owner=None):
        if instance is None:
            return self
        try:
            return instance.__dict__[self.slot]
        except KeyError:
            raise AttributeError(f"{type(instance).__name__}.{self.name} is not set") from None

    def __set__(self, instance, value):
        instance.__dict__[self.slot] = self.check(value)

The same bug, three days running

Day 1: a mutable default shared by every call. Day 1’s bug hunt: one cache shared by every model. Today: one value shared by every instance. Each looks different at the call site and each is the same mistake — state living on something longer-lived than the thing it describes. This is why the object model came first.

Why `_title` and not `title`

The field stores under _title rather than title, and the reason is the data-descriptor rule from this morning. Post.title defines __set__, so it wins over the instance dict for the name title — reading post.title always calls __get__, even if post.__dict__["title"] exists. Writing the value under the same name would be harmless but invisible; writing it under _title keeps the storage separate from the attribute, which matters when you look at vars(post) while debugging.

It does mean _title is now part of the object’s shape. A caller who writes post._title = "anything" bypasses validation entirely — the underscore is the only thing stopping them, and in Python that’s usually enough.

Where Day 2 leaves the library

The finished Day 2 Python

"""Fields: one descriptor per column, validating what callers assign.""" class Field: """A column, and the rules for what may go in it. Assigned in a class body, so `__set_name__` learns the attribute name the developer chose. Values live in the *instance* dict under a private key; the descriptor itself is shared by every instance of the model. """ def __init__(self, *, null=False): self.null = null self.name = "<unnamed>" self.slot = "<unnamed>" def __set_name__(self, owner, name): self.name = name self.slot = f"_{name}" def __get__(self, instance, owner=None): if instance is None: return self # Post.title is the field itself try: return instance.__dict__[self.slot] except KeyError: raise AttributeError(f"{type(instance).__name__}.{self.name} is not set") from None def __set__(self, instance, value): instance.__dict__[self.slot] = self.check(value) def check(self, value): """Return the value to store, or raise ValueError.""" if value is None and not self.null: raise ValueError(f"{self.name} may not be None") return value def __repr__(self): return f"{type(self).__name__}({self.name!r})" class TextField(Field): def __init__(self, *, max_length=None, **kwargs): super().__init__(**kwargs) self.max_length = max_length def check(self, value): value = super().check(value) if value is None: return None if not isinstance(value, str): raise ValueError(f"{self.name} must be a str, not {type(value).__name__}") if self.max_length is not None and len(value) > self.max_length: raise ValueError(f"{self.name} is longer than {self.max_length} characters") return value class IntegerField(Field): def __init__(self, *, minimum=None, **kwargs): super().__init__(**kwargs) self.minimum = minimum def check(self, value): value = super().check(value) if value is None: return None # bool is a subclass of int, and True in an integer column is a mistake. if isinstance(value, bool) or not isinstance(value, int): raise ValueError(f"{self.name} must be an int, not {type(value).__name__}") if self.minimum is not None and value < self.minimum: raise ValueError(f"{self.name} must be at least {self.minimum}") return value

import sys import unittest import test_models suite = unittest.defaultTestLoader.loadTestsFromModule(test_models) unittest.TextTestRunner(stream=sys.stdout, verbosity=2).run(suite)

Working on your own machine? Download the Day 2 files.

Two things are worth noticing about this version.

Model.__init__ no longer sets every field: it only assigns what it was given, so a missing value stays missing and reading it raises AttributeError rather than silently being None. save() would then fail on the unset attribute, which is exactly when you want to hear about it.

And columns is now a property derived from fields, so the two can’t disagree. Removing the last place where a model repeated itself is what the day was for.

What you learned

The descriptor protocol, the check hook and __init_subclass__ fit together into an API where the class body is the entire specification.
Per-instance state belongs in the instance; a descriptor is shared.
raise ... from None hides an implementation detail that would otherwise surface in a user’s traceback.

Tomorrow: types. The fields know what they hold at run time; a type checker should know it before the code runs — and there is a limit to how much of this design can be expressed that way.

The checkpoint

Bug hunt: every post has the same title

The same bug, three days running

Why _title and not title

Where Day 2 leaves the library

What you learned

Was this article helpful?

Why `_title` and not `title`