When a Regex Is the Right Tool

The message part of each line holds key=value pairs: user=ada action=login. Turning that into a dictionary looks like a job for the string methods you already know. For most lines, it is.

def parse_fields(message):
    return dict(pair.split("=", 1) for pair in message.split())

print(parse_fields("user=ada action=login"))
print(parse_fields('order=1043 msg="card declined"'))

The first line works. The second fails, because split() cuts wherever there’s whitespace, and "card declined" has a space inside a value. The pieces msg="card and declined" are no longer pairs.

This is the moment to change tools, and it’s worth being precise about why:

The line’s prefix, the date, time and level, fits the first row, and it stays with split() from the previous lesson. Only the message needs a regex. Converting code that already works to a regex doesn’t make it better. It just makes it harder to read.

Describing the pattern

A regular expression describes what the text looks like, not where to cut it. Here, a field is a word, an =, then either a quoted value or a bare one:

import re

message = 'order=1043 msg="card declined" retry=1'

# A key is a word; a value is either "anything but a quote, in quotes" or a word.
pattern = re.compile(r'(\w+)=("[^"]*"|\S+)')
print(pattern.findall(message))

# Named groups say what each part is, and finditer yields one match at a time.
pattern = re.compile(r'(?P<key>\w+)=(?P<value>"[^"]*"|\S+)')
for match in pattern.finditer(message):
    print(match["key"], "->", match["value"].strip('"'))

Just five pieces are doing all of that:

Write patterns as raw strings, r'…'. A backslash means something to two readers: Python reads the string literal first, and re reads the pattern after. r tells Python to leave every backslash alone for re. Without it, some escapes change meaning before re sees them. "\bword\b" holds two backspace characters, not the word boundaries r"\bword\b" asks for, so it quietly matches nothing. Others, like "\w", survive by luck, but Python warns about them (SyntaxWarning: invalid escape sequence), because the luck may run out.

Once a pattern has more than a couple of parts, re.VERBOSE lets you lay it out and comment it. Whitespace in the pattern is then ignored, so you can use it for layout:

import re

FIELD = re.compile(r"""
    (?P<key>\w+)          # a key: letters, digits, underscore
    =
    (?P<value>
        "[^"]*"           # either a quoted value, which may contain spaces,
      | \S+               # or a bare value, up to the next whitespace
    )
""", re.VERBOSE)

print({m["key"]: m["value"].strip('"') for m in FIELD.finditer('a=1 b="two words"')})

Bug hunt: the regex that ate a line

A colleague’s first version used ".*" for quoted values. It passed their test. Then a line with two quoted values came in:

import re

FIELD = re.compile(r'(?P<key>\w+)=(?P<value>".*"|\S+)')


def parse_fields(message):
    return {m["key"]: m["value"].strip('"') for m in FIELD.finditer(message)}


print(parse_fields('msg="card declined"'))
print(parse_fields('query="python books" msg="index offline"'))

Reproduce. The second call is the small failing input: two fields, one line.

Inspect. There’s no exception, which makes this one dangerous. What’s wrong only shows in the state: msg has vanished, and query has swallowed everything up to the last quote on the line. When a bug produces wrong output instead of a traceback, inspect the output, and compare it with what you expected.

Narrow. Take the pattern out of the program and try it on the smallest text that shows the problem:

import re

text = '"a" "b"'
print(re.findall(r'"(.*)"', text))      # greedy: as much as possible
print(re.findall(r'"(.*?)"', text))     # lazy: as little as possible
print(re.findall(r'"([^"]*)"', text))   # no quotes allowed inside

.* is greedy: it takes as much as it can while still letting the pattern match. Here that means everything up to the last quote. .*? is its lazy cousin. [^"]* is the most precise choice, because it says what a quoted value actually is: characters that aren’t a quote.

Pin it. A test with two quoted values on one line. It’s the case the original test missed.

Fix. Use the precise version in FIELD, r'(?P<key>\w+)=(?P<value>"[^"]*"|\S+)', and run again.

Your turn

import re
import sys
import unittest


def parse_fields(message):
    """Turn 'a=1 b="two words"' into {'a': '1', 'b': 'two words'}."""
    raise NotImplementedError("your code here")


class TestParseFields(unittest.TestCase):
    def test_bare_values(self):
        self.assertEqual(parse_fields("user=ada action=login"),
                         {"user": "ada", "action": "login"})

    def test_quoted_value_with_spaces(self):
        self.assertEqual(parse_fields('msg="card declined"'), {"msg": "card declined"})

    def test_equals_sign_inside_quotes(self):
        self.assertEqual(parse_fields('msg="x=1 failed" retry=2'),
                         {"msg": "x=1 failed", "retry": "2"})

    def test_two_quoted_values(self):
        self.assertEqual(parse_fields('query="python books" msg="index offline"'),
                         {"query": "python books", "msg": "index offline"})

    def test_empty_message(self):
        self.assertEqual(parse_fields(""), {})


suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestParseFields)
unittest.TextTestRunner(stream=sys.stdout, verbosity=2).run(suite)

FIELD = re.compile(r'(?P<key>\w+)=(?P<value>"[^"]*"|\S+)')


def parse_fields(message):
    """Turn 'a=1 b="two words"' into {'a': '1', 'b': 'two words'}."""
    return {m["key"]: m["value"].strip('"') for m in FIELD.finditer(message)}

What you can now do

• Decide between string methods and a regex by the shape of the format, not by habit.
• Write a pattern with character classes, alternation and named groups, and lay it out with re.VERBOSE.
• Recognise greedy matching, and use a negated class like [^"]* to say exactly what a value may contain.

Next: the log files get big, and reading them all into memory stops being an option.