Basic Regular Expressions

The re module provides features to enable pattern matching in Python. The basic mode of operation is to call either the match() or search() function from that module with a regex as the first argument, and a string to match against as the second argument. If the regex matches the string, the module returns a match object, and analysis of the match object can give you information about (for example) the exact strings matched by various portions of the pattern.

>>> import re
>>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist")
>>> m
<re.Match object; span=(0, 12), match='Isaac Newton'>
>>> m.groups()
('Isaac', 'Newton')
>>> m.group(0)
'Isaac Newton'
>>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds")
>>> m.group('first_name')
'Malcolm'
>>> m.group('last_name')
'Reynolds'
>>> m.groupdict()
{'first_name': 'Malcolm', 'last_name': 'Reynolds'}
>>> m = re.match(r"(..)+", "a1b2c3")  # Matches 3 times.
>>> m.group(1)                        # Returns only the last match.
'c3'
>>> m = re.search("\\d+", "hello123extra")
>>> m.start()
5
>>> m.end()
8
>>> m.span()
(5, 8)
>>> m.group(0)
'123'
>>>

In the interactive session above, we used the re module's match() and search() functions to determine whether strings conformed to a specific pattern (provided as the first argument to the function call). match() requires the pattern to occur at the start of the target string, while search() will move through the target string looking for it. If the pattern is not present in the string, the function call returns None. Otherwise it returns a match object ("m," above) that can be queried for specific aspects of the matched string by calling its various methods.

A regular expression pattern is a way of describing a set of character strings. These descriptions can be relatively concise: the pattern "x" matches precisely one character, the lowercase letter "x." Some characters have special meanings, so the pattern "x+" matches any string of one or more lower case "x"s—the plus sign generates a more complex pattern from the pattern it follows.

Most characters can be used in patterns like the lower case "x" to "stand for themselves," so for example if you wanted to match the literal string "thing" you would do so with a pattern that reads "thing"—the "t" in the pattern matches a "t" in the string, and so on. But there are quite a few abbreviations: for example, '\d' matches any decimal digit (making it equivalent to the pattern "[0123456789]," as we will learn shortly). Here are some of the more common abbreviations.

As the last line above indicates, patterns can contain groups, indicated by parentheses. The strings matched by the groups are, under certain circumstances, available—again, by calling the match object's methods. The groups can be numbered (according to their relative positions in the pattern, and starting at one rather than Python's usual zero—group 0 refers to the match as a whole) and they can also be named if the group's opening parenthesis in the pattern is followed by a question mark, an upper case "P" and a name in angle brackets, as we saw with "(?P<first_name>\w+)" in the earlier interactive session.

Groupings in the pattern are the principal way of extracting required information from the match object. Strings matched by non-grouping portions of the pattern cannot be individually identified in the match object. When you are testing a new regular expression, it is often useful to interactively inspect the result of calling the match objects' groups() and groupdict() methods to verify that your pattern is matching as you expect.

When testing patterns, you can test for equality with those objects; but you should also remember to test that unacceptable strings are not, in fact, matched. This will usually involve the use of your test case's assertNone() method on the match result.

Besides the match() and search() functions, the re module provides functions that allow you to make replacements of patterns in the target string (these functions return new strings, of course, because strings are immutable in Python). The re.sub() function takes not only a pattern and a target string but also a replacement string, as shown below.

re.sub(pat, replacement, target[, count, flags])

This replaces each non-overlapping occurrence of the given pattern in the target string with the replacement element given as the third argument. If replacement is a string, any backslash escapes in it are processed down to individual characters (so, for example, "\n" is replaced by a newline character). Escapes of the form \n (where n is a decimal digit) allow replacement by one of the matched groups from the pattern. The replacement argument can also be a function, in which case it is called for each replacement with a single argument, which is the match object corresponding to the currently matched string that is to be replaced.

The optional argument count is the maximum number of pattern occurrences to be replaced; count must be a non-negative integer. If omitted or zero, all occurrences will be replaced. Empty matches for the pattern are replaced only when not adjacent to a previous match, so sub('x*', '-', 'abc') returns '-a-b-c-'. The flags, if present, are the usual regular expression matching flags, which we'll discuss a little later. Let's get an idea of what you can do with the replacement facilities.

This example simply shows straight pattern replacement: both calls to re.sub() replace all occurrences of the string "1" with a newline character. When the pattern contains characters like newline (normally represented by escape sequences), or special patterns (which also require backslashes), r(aw) strings can make patterns more readable and easier to type. The more complex the patterns become, the truer this is.

>>> import re
>>> re.sub("1", "\\n", "123123123123") # replace digit one with newline
'\n23\n23\n23\n23'
>>> re.sub("1", r"\n", "123123123123") # replace digit one with newline
'\n23\n23\n23\n23'

The next call to re.sub() uses a function to supply the replacement string: if the function is replacing a single minus sign it returns a space, but two minus signs are translated into a plus sign.

>>> def dashrepl(matchobj):
...     if matchobj.group(0) == "-": return " "
...     else: return "+"
...
>>> re.sub('-{1,2}', dashrepl, 'pro----gram-files')
'pro++gram files'

The next example finds the "#" marker and removes it and everything after it, then removes everything but the digits from the remaining string.

>>> s = "(123) 456-7890 # Commented phone number"
>>> nocomment = re.sub("#.*$", "", s)
>>> nocomment
'(123) 456-7890 '
>>> re.sub(r"\D", "", nocomment)
'1234567890'

These examples attempt to match any string beginning and ending in an at sign ("@") with zero or more sequences of "=+=" in the middle (the "+" must be escaped to make the matching code treat it as an ordinary character).

>>> re.sub("@(=+=)*@", "xxx", "@@")
'xxx'
>>> re.sub("@(=+=)*@", "xxx", "@=+=@")
'xxx'
>>> re.sub("@(=+=)*@", "xxx", "@=+==+=@")
'xxx'
>>> re.sub("@(=+=)*@", "xxx", "@=+=+=@")
'@=+=+=@'

The last example shows a pattern (a single vowel) being used to make many replacements—all vowels in the target string are replaced with a dash.

>>> re.sub("[aeiouAEIOU]", "-", "The Quick Brown Fox Jumps Over the Lazy Dog")
'Th- Q--ck Br-wn F-x J-mps -v-r th- L-zy D-g'
>>>

It's useful to be able to try out lots of patterns as you are learning how they are made up. See if you can understand the following patterns by trying them against various strings. To help you do that, we'll write a little program that allows you to see the results of searching and matching for a specific pattern against a number of strings. The program reads a pattern, and if it's the empty string, terminates. Otherwise, it reads target strings and applies matches and searches on the strings that are subsequently input until the user enters an empty string, in which case it goes back to requesting a new pattern. Create pattest.py as shown:

"""
pattest.py: Allows the checking of various patterns and target strings
"""
import re
while True:
    pat = input("Pattern: ")
    if not pat:
        break
    while True:
        s = input("Target : ")
        if not s:
            break
        mm = re.match(pat, s)
        if mm:
            print("Match : matched {0!r}".format(s[mm.start():mm.end()]))
            print("Match : groups:", mm.groups())
            print("Match : gdict :", mm.groupdict())
        else:
            print("Match : no match")
        ms = re.search(pat, s)
        if ms:
            print("Search: matched {0!r}".format(s[ms.start():ms.end()]))
            print("Search: groups:", ms.groups())
            print("Search: gdict :", ms.groupdict())
        else:
            print("Search: no match")

This lets you test many strings against the same pattern quite quickly. Run it and ensure that you can think of strings that both match and don't match the patterns given below.