Matching Patterns with re

Compile patterns once, choose between match and search, name your groups, and keep greedy quantifiers in check.

Syntaxre.compile(pattern, flags=0).search(string) -> re.Match | None

The re module matches text against a pattern. Reach for it when the structure is genuinely irregular; for fixed delimiters str.split, str.startswith and str.removeprefix are faster to write and to read.

compile, and the module-level shorthand

re.compile(pattern, flags=0) returns a re.Pattern with .search(), .match(), .fullmatch(), .findall(), .finditer() and .sub() on it. The module-level re.search(pattern, string) is the shorthand for re.compile(pattern).search(string); it caches recent compiles, so the difference matters mostly for readability and for keeping a long pattern out of a loop body. Always write patterns as raw strings — r"\d+", not "\d+".

match, search, fullmatch

Per the docs, match() checks for a match "only at the beginning" of the string; search() "scans through string looking for the first location" that matches. Neither requires the whole string — that is fullmatch(), which is the one you want for validation. All three return a re.Match or None.

Groups, and what each function returns

Use named groups: (?P<name>...), read back as m.group("name") or the shorthand m["name"], with m.groupdict() for all of them at once. findall returns strings when there are no groups, the group when there is exactly one, and tuples when there are more. finditer yields Match objects instead, so you keep .span() and the group names.

Greedy quantifiers

.* takes the longest run that still allows a match, so "/(.*)=" runs to the last = on the line. .*? is the lazy form and stops at the first. Better still, match a character class that excludes the delimiter: [^=]*.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Pull method, path and status out of access-log lines into dicts.
  • Validate that a reference code matches a fixed format before writing it to the database.
  • Redact card numbers or tokens from a string before it reaches the logs.
  • Extract every key=value pair from a free-form settings string.

More examples

re.VERBOSE for a pattern you will read again

Use VERBOSE for anything longer than a line. It costs nothing at match time, and it is the difference between a pattern a colleague can change and one they will rewrite from scratch. Note that whitespace you want to match must then be written \s or escaped.

Example · python
import re

# re.VERBOSE (re.X) ignores whitespace and # comments inside the pattern, so a
# pattern can have one group per line with a note on each.
LOG = re.compile(r"""
    ^(?P<ip>\d{1,3}(?:\.\d{1,3}){3})\s    # client address
    (?P<method>[A-Z]+)\s                  # HTTP method
    (?P<path>\S+)\s                       # request target
    (?P<status>\d{3})$                    # response status
""", re.VERBOSE)

m = LOG.fullmatch("10.0.0.7 GET /health 200")
print(m.groupdict())
# {'ip': '10.0.0.7', 'method': 'GET', 'path': '/health', 'status': '200'}

# (?:...) is a NON-capturing group: it repeats without adding to groupdict().

sub with named backreferences

The usual job is redacting secrets before text reaches a log. Use \g<name> rather than \1 whenever the replacement sits far from the pattern, and prefer the function form of repl over building a replacement string with string formatting, which reintroduces escaping bugs.

Example · python
import re

CARD = re.compile(r"(?P<prefix>\d{4})[ -]?\d{4}[ -]?\d{4}[ -]?(?P<last4>\d{4})")

# In the replacement, \g<name> refers to a named group; \1 and \g<1> refer to
# numbered ones. count=0 (the default) replaces every occurrence.
redacted = CARD.sub(r"\g<prefix>********\g<last4>", "card 4111 1111 1111 1234 on file")
print(redacted)   # card 4111********1234 on file

# repl can also be a function taking the Match, for anything conditional:
#   CARD.sub(lambda m: mask(m["prefix"], m["last4"]), text)

fullmatch for validation

Every format check should use fullmatch, or match with an explicit $ anchor. This is the single most common regex bug in validation code: the pattern looks right, the tests with valid input pass, and anything with trailing characters sails through.

Example · python
import re

SKU = re.compile(r"[A-Z]{1,3}-\d{3,5}")

def is_sku(value: str) -> bool:
    # fullmatch, not match: match("A-100xx") SUCCEEDS, because match only
    # anchors the start of the string and ignores the trailing junk.
    return SKU.fullmatch(value) is not None

print(is_sku("A-100"), is_sku("A-100xx"), bool(SKU.match("A-100xx")))
# True False True

Related

py strings intropy string methodspy string escapepy data validatepy fstrings

Discussion

  • Be the first to comment on this lesson.