More On Regular Expressions

We want to search a block of text for phone numbers in Python using Regular Expressions. As usual, first, we'll write a test to confirm that we're getting the behavior we want, and then we'll write the code. Create test_phone.py as shown:

import unittest
from phone import get_phone, text

class TestRegex(unittest.TestCase):

    def test_phone(self):
        numbers = get_phone(text)
        self.assertEqual(len(numbers), 5)

if __name__ == "__main__":
    unittest.main()

Our first code finds only the phone numbers whose area code is not surrounded by parentheses, and the test is satisfied as long as the function detects five phone numbers in the text—without verifying that it has the exact numbers right. It is, however, much better than not having any tests!

"""
Demonstrate use of re.findall().
"""
import re

text = """While I was at the store I tried to call 555-123-4567 on my mobile
but accidentally called 555-754-4321.  The person on the line redirected me to
999-999-9999 which I don't think is a real number. Neither is 000-000-0000 or 555-555-0000.
Well, I will try (555) 123-4567 again now.
"""

def get_phone(text):
    "Scan a text, locating telephone numbers."
    # Note the use of a "raw" string constant
    return re.findall(r"\d\d\d-\d\d\d-\d\d\d\d", text)

if __name__ == '__main__':
    print(get_phone(text))

['555-123-4567', '555-754-4321', '999-999-9999', '000-000-0000', '555-555-0000']

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Demonstrate use of re.findall().
"""
import re

text = """While I was at the store I tried to call 555-123-4567 on my mobile
but accidentally called 555-754-4321.  The person on the line redirected me to
999-999-9999 which I don't think is a real number. Neither is 000-000-0000 or 555-555-0000.
Well, I will try (555) 123-4567 again now.
"""

def get_phone(text):
    "Scan a text, locating telephone numbers."
    # Note the use of a "raw" string constant
    return re.findall(r"\d\d\d-\d\d\d-\d\d\d\d", text)

if __name__ == '__main__':
    print(get_phone(text))

The first thing this program does is import the Python regular expression library, re. The get_phone() function uses a regular expression as the first argument to the findall() function from that library. The pattern \d\d\d-\d\d\d-\d\d\d\d is applied to the text, and the result is a list of strings matched by the pattern.

The regex pattern "\d\d\d-\d\d\d-\d\d\d\d" is the central component of the code above. If you replace each "\d" with an "X," you get XXX-XXX-XXXX—the template for matching the phone numbers.

You probably noticed that your code does not find the phone number with the prefix in parentheses. We'll cover that later in this lesson.

Regular expressions often use the backslash (\) character. Mostly it indicates special meanings for the characters immediately following, but it can also be used to "escape" the standard meanings of certain characters in pattern strings, so that you can recognize these special characters too. You probably remember that the backslash also has a special meaning in string literals ("\n" means newline, "\t" means tab, and so on).

You probably remember that to represent a single backslash in a string, you normally need to use two backslashes—"\\." This would make regular expressions very difficult to read. Consequently, we have "raw" string constants (whose representations are preceded by the letter "r") to represent regex patterns. These let you represent the backslashes without escaping, which makes them much more readable.

The basic use case for regular expressions is finding occurrences of strings that conform to a pattern. The Python regular expression library gives you two ways to perform this action, re.match() and re.search(). The difference between them is as follows:

• match() checks at the start of a string and returns None if nothing is found.
• search() moves up the string, looking for the first occurrence of the given pattern, and returns None only if the pattern occurs nowhere in the string.

For example, suppose we have several paragraphs and want to see if they start with or contain a phone number. If a paragraph starts with a phone number, we'll assume that the paragraph is just a phone number and we want to return it. Otherwise, if a paragraph contains a phone number, we want to return the length of the paragraph. If a paragraph has no telephone numbers, we'll return None. Create test_match_vs_search.py as shown:

import unittest
from match_vs_search import check_number

p1 = """While I was at the store in Washington, DC 20001 I tried to call 555-123-4567 on my mobile
but accidentally called 555-754-4321.  The person on the line redirected me to
999-999-9999 which I don't think is a real number. Neither is 000-000-0000 or 555-555-0000.
Well, I will try (555) 123-4567 again now."""

p2 = "555-555-5555"

p3 = "What is the author's phone number?"

class TestRegex(unittest.TestCase):

    def test_match(self):
        result = check_number(p2)
        self.assertEqual("555-555-5555", result)

    def test_search(self):
        result = check_number(p1)
        self.assertEqual(305, result)

    def test_none(self):
        result = check_number(p3)
        self.assertIsNone(result)

if __name__ == "__main__":
    unittest.main()

Save it and create match_vs_search.py in the same folder:

"""
Demonstrate the difference between match() and search().
"""

import re

def check_number(text):
    regex = r"\d\d\d-\d\d\d-\d\d\d\d"
    match = re.match(regex, text)
    if match:
        return match.group()
    match = re.search(regex, text)
    if match:
        return len(text)

Save it and run the test program:

...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

Now, let's try our match_vs_search program.

>>> from match_vs_search import *
>>> check_number("707-867-5309")
'707-867-5309'
>>> check_number("Jenny's number is 707-867-5309")
30

Let's look at how it works.

def check_number(text):
    regex = r"\d\d\d-\d\d\d-\d\d\d\d"
    match = re.match(regex, text)
    if match:
        return match.group()
    match = re.search(regex, text)
    if match:
        return len(text)

The check_number() function first attempts to match a telephone number at the beginning of the text. If that succeeds, it returns a match object match, described below.

If the re.match() call fails to find the pattern, it returns None, and the function then calls the re.search() function to try and find a number somewhere in the interior of the text. If the search succeeds, then the function returns the length of the paragraph. Otherwise it "falls off the bottom" and returns None (as is standard in Python).

Let's continue our session to explore the difference between match() and search():

>>> import re
>>> target = "This is a string"
>>> def t(p, t):
...     if re.match(p, t):
...         print("match")
...     if re.search(p, t):
...         print("search")
...
>>> t("is", target)
search
>>> t("This", target)
match
search
>>> t("Th", target)
match
search
>>> t("ing", target)
search
>>> t("^ing", target)
>>>

The last two examples show that a pattern for which search() is successful becomes unsuccessful if changed to require with ^ that the match occurs at the start of the string.

Any successful application of matching or searching returns a match object. This match object includes a number of useful methods, the most important of which are:

The match object returned from an re.match() call always has a start() value of 0 and the span() method also always returns 0 as the first element of the tuple. This is because, as noted earlier, the match() function only returns patterns found at the start of a string.

On the other hand, the search() function finds strings anywhere. The test_search() function in the tests calls check_number(p1), which calls search(). This also returns a match object, although it isn't returned to the caller. If we apply search() to paragraph 1, we see:

As you can see, match() and search() are two very similar functions with a single important difference.

The code you wrote found numbers of the form XXX-XXX-XXXX, because the re module's functions recognize "\d" as requiring a digit in the scanned string. (The backslash tells the functions that the "d" is to be specially interpreted—without it, they would only match the literal character "d"). But what about (555)-123-4567? That is a phone number, but it doesn't follow the same pattern.

You could write a second regular expression for this, and then try matching the first and only try the second if the first one did not match. This could become clumsy quite rapidly in the case of complex patterns. Fortunately, regular expressions can model complex patterns to handle this sort of problem. Regular expressions can specify using alternate patterns using the "|" special character, which means a pattern like the one below will find phone numbers following either the XXX-XXX-XXXX or (XXX)-XXX-XXXX patterns.

r"\d\d\d-\d\d\d-\d\d\d\d|\(\d\d\d\)(-| )\d\d\d-\d\d\d\d"

By now you are probably thinking that every character in a regular expression must be preceded by a backslash! This is not the case, but as we've learned, the parentheses have a specific meaning to the regular expression matching routines, so they need to be escaped to tell the routines to look for them just as regular characters.

One of the difficulties of the pattern above is that both alternate patterns have the same ending but different beginnings. We can overcome this by using parentheses to group portions of our pattern. So an equivalent pattern (ignoring complexities we haven't yet covered) would be

r"(\d\d\d|\(\d\d\d\))(-| )\d\d\d-\d\d\d\d"

In this pattern the alternation is restricted to the portions inside the parentheses—that is, the parentheses that are not preceded by backslashes. So the part of the pattern in parentheses will match either three digits or three digits surrounded by parentheses. Then it will match either a dash (-) or a space. In either case the rest of the pattern is the same. Now modify match_vs_search.py to use this extended pattern.

"""
Demonstrate the difference between match() and search().
"""

import re

def check_number(text):
    regex = r"\d\d\d-\d\d\d-\d\d\d\d"
    
    regex = r"(\d\d\d|\(\d\d\d\))(-| )\d\d\d-\d\d\d\d"
    match = re.match(regex, text)
    if match:
        return match.group()

    match = re.search(regex, text)
    if match:
        return len(text)

To correctly test this update, we also need to modify the test routine by adding tests that require correct matching of numbers whose area codes are in parentheses and followed by a dash or a space. You will see there is also some simplification of the test code, since there is no need to store the result in a variable before testing it.

import unittest
from match_vs_search import check_number

p1 = """While I was at the store in Washington, DC 20001 I tried to call 555-123-4567 on my mobile
but accidentally called 555-754-4321.  The person on the line redirected me to
999-999-9999 which I don't think is a real number. Neither is 000-000-0000 or 555-555-0000.
Well, I will try (555) 123-4567 again now."""

p1a = """While I was at the store in Washington, DC 20001 I tried to call (555) 123-4567 on my mobile
but accidentally called (555)-754-4321.  The person on the line redirected me to
(999)-999-9999 which I don't think is a real number. Neither is (000)-000-0000 or (555) 555-0000.
Well, I will try (555) 123-4567 again now."""

p2 = "555-555-5555"
p2a = "(555)-555-5555"

p3 = "What is the author's phone number?"

class TestRegex(unittest.TestCase):

    def test_match(self):
        
        self.assertEqual("555-555-5555", check_number(p2))
        self.assertEqual("(555)-555-5555", check_number(p2a))

    def test_search(self):
        
        self.assertEqual(305, check_number(p1))
        self.assertEqual(315, check_number(p1a))

    def test_none(self):
        result = check_number(p3)
        self.assertIsNone(result)

if __name__ == "__main__":
    unittest.main()

If you have correctly modified your code, all tests should pass.

...
----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK

So far, we've only seen a little of what regular expressions can do. It is quite easy to extend the searching facilities to alphanumeric patterns. Suppose we need to find a city, state, and zip code in a paragraph—the text would follow this rough pattern: City Name, State Abbreviation Zip Code. How do you express a pattern to match such strings?

The first thing we want to do is get the capital letter that starts each city name. In regular expressions, we can match a single occurrence from a set of characters by putting the characters in square brackets—to match any upper-case letter, we can use [ABCDEFGHIJKLMNOPQRSTUVWXYZ]. This is rather tedious to type, so we can use a range, [A-Z], instead.

Next we need to match the other letters of the city name (we assume there will be one or more further characters). For that we'll use the brackets and range again, but add a little more: [a-z]+. The plus sign allows for any number of lower-case letters to match. So the pattern to match a capitalized word is [A-Z][a-z]+.

Some cities have multiple words in their name (Falls Church and San Francisco come to mind). Thus, the first word can optionally be followed by one or more further words, each separated from its predecessor by white space. So we need to follow the original pattern with zero or more repeats to the same pattern, with the repeats preceded by a whitespace. The pattern for that is (\s[A-Z][a-z]+)*.

Note that, in order to apply the * character to the whole grouping, parentheses are required.

Now, we need to account for the state abbreviations. The easiest way to do it in regular expressions is via [A-Z]{2}, which only allows two uppercase letters, matching the US postal designation for American states. Add that to our regular expression, include a comma, and allow for a little white space: ,\s[A-Z]{2}.

Finally, we handle zip code handling portion of the pattern. We won't check for nine-digit or foreign postal codes right now, so for our purposes, \d{5} will suffice. This makes the final pattern [A-Z][a-z]+(\s[A-Z][a-z]+)*,\s[A-Z]{2}\s\d{5}.

Now we'll try incorporating that into a function that finds the required addresses. Naturally, we need to write some tests first. Create test_city_search.py as shown:

import unittest

from city_search import city_search

p1 = """While I was at the store I tried to call 555-123-4567 on my mobile
but accidentally called 555-754-4321.  The person on the line redirected me to
999-999-9999 which I don't think is a real number. Neither is 000-000-0000 or 555-555-0000.
Well, I will try (555) 123-4567 again now."""

p2 = "I live in Washington, DC 20002. Where do you live?"
p3 = "I live in Falls Church, VA 20188. And you?"

class TestRegex(unittest.TestCase):

    def test_city_search(self):
        self.assertEqual("Washington, DC 20002", city_search(p2))
        self.assertEqual("Falls Church, VA 20188", city_search(p3))

    def test_city_search_failure(self):
        self.assertIsNone(city_search(p1))

if __name__ == "__main__":
    unittest.main()

Save it. Most of the work has already been done with the design of the regular expression, and the function now simply needs to use it to locate addresses. Create city_search.py as shown:

"""
String regular expressions
"""

import re

def city_search(text):

    regex = r"[A-Z][a-z]+(\s[A-Z][a-z]+)*,\s[A-Z]{2}\s\d{5}"

    search = re.search(regex, text)
    if search:
        return search.group()

Save it, and then run test_city_search.py. The tests should pass.

..
----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

Regular expressions have a power which their apparent simplicity belies, as you can now start to appreciate.

The first programming example in this lesson used a regular expression function named findall(). In that code, it returned a list of non-overlapping matching phone numbers from the paragraph. This is useful for providing a list of strings, but what if you need to know the start and end index of each of those phone numbers, in other words familiar data shown below, but for each found part of the string? While findall returns a list of the matching strings, finditer returns a list of the matching objects, and each match object has these methods:

Suppose you don't want to publish all the phone numbers in this lesson, but you do want to show area codes. Regular expressions let you find patterns, and they also provide tools to allow you to modify them. The regular expression sub() method can make this sort of substitution. Pass in your pattern, what you want it replaced with, and the string to modify: re.sub("\d\d\d-\d\d\d\d", "XXX-XXXX", text).

Let's make a program to show this in action. Create test_phone_hide.py:

import unittest

from phone_hide import phone_hide

text = """While I was at the store I tried to call 555-123-4567 on my mobile
but accidentally called 555-754-4321.  The person on the line redirected me to
999-999-9999 which I don't think is a real number. Neither is 000-000-0000 or 555-555-0000.
Well, I will try (555)-123-4567 again now.
"""

class TestRegex(unittest.TestCase):

    def test_phone(self):
        response = phone_hide(text)
        self.assertFalse("555-123-4567" in response)
        self.assertTrue("555-XXX-XXXX" in response)
        self.assertTrue("(555)-XXX-XXXX" in response)

if __name__ == "__main__":
    unittest.main()

Then, create phone_hide.py in the same folder:

import re

def phone_hide(text):

    # Don't forget to use a raw string constant!
    return re.sub(r"\d{3}-\d{4}", "XXX-XXXX", text)

Save both programs and run the test:

.
----------------------------------------------------------------------
Ran 1 test in 0.022s

OK

What if we want to know how many substitutions occurred? Then we can use the subn() function, which returns a two-element tuple containing the result string and the number of substitutions. Modify test_phone_hide.py as shown:

import unittest

from phone_hide import phone_hide

text = """While I was at the store I tried to call 555-123-4567 on my mobile
but accidentally called 555-754-4321.  The person on the line redirected me to
999-999-9999 which I don't think is a real number. Neither is 000-000-0000 or 555-555-0000.
Well, I will try (555)-123-4567 again now.
"""

class TestRegex(unittest.TestCase):

    def test_phone(self):

        response, count = phone_hide(text)
        self.assertFalse("555-123-4567" in response)
        self.assertTrue("555-XXX-XXXX" in response)
        self.assertTrue("(555)-XXX-XXXX" in response)
        self.assertEqual(6, count)

if __name__ == "__main__":
    unittest.main()

Save and run it:

E
======================================================================
ERROR: test_phone (__main__.TestRegex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\sholden\workspace\Python3_Lesson4\src\test_phone_hide2.py", line 14, in test_phone
    response, count = phone_hide(text)
ValueError: too many values to unpack

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

It fails because the current phone_hide() function still returns a single value. We need to modify it to call subn() instead of sub().

import re

def phone_hide(text):

    # Don't forget the 'r' at the start of the string!
    return re.subn(r"\d{3}-\d{4}", "XXX-XXXX", text)

Inserting that single letter "n" should be enough to restore everything to a fully functional state.

Save it, and run the test again:

.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

Now suppose we want to split up a paragraph into a list of sentences. Python's split() function makes this problem trivial to solve. The regular expression pattern to find a sentence end (assuming some simplifications) is r"[?.!]\s+".

The bracketed set contains the punctuation characters ?, ., and !, which represents the ending of each sentence. Although these characters all have special meanings in regular expressions, remember that within a character set specification, they are treated as literal.

The '\s+' portion of the pattern requires a space or spaces after the ending punctuation of a sentence. The more precise you make a pattern the better your results will be. Without the spaces, a period used as a decimal point inside a number would be treated as ending a sentence.

Let's give it a try! As usual we'll begin by writing the tests. Create test_sentence_split.py as shown:

import unittest

from sentence_split import sentence_split

text = "Hello! My name is Steve. What is yours? I hope you enjoyed this class!"

class TestRegex(unittest.TestCase):

    def test_split_sentence(self):
        numbers = sentence_split(text)
        self.assertEqual(len(numbers), 4)

if __name__ == "__main__":
    unittest.main()

Then, in the same folder, create sentence_split.py:

import re

def sentence_split(text):
    return re.split(r"[?.!]\s+", text)

Save both files, and run the test:

.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

Regular expressions can get extremely complex. For example, let's say you want to pull all of the email addresses from a paragraph. This sounds like a simple enough task, right? Something like r"[a-zA-Z.-]+\@[a-zA-Z.-]+" should work, right?

Unfortunately, if you apply that pattern to "So... um...@oreilly we found his email was steve@oreilly.com." you will get steve@oreilly.com out, but it will also give you um...@oreilly!

So your pattern should be able to handle only proper email prefixes and should not allow repetition of dots. There are other specifications for allowed domain suffixes such as nations, .info, .com, and others. You should really research RFC 2822, which is the official email specification, with all its special cases and rules. And that sort of complexity generates regular expressions that look like this:

"[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*@(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+(?:[A-Z]{2}|com|org|net|edu|gov|mil|biz|info|mobi|name|aero|asia|jobs|museum)\b"

As you can see, regular expressions can get out of hand—and this is just for emails! Regular expression syntax is arguably not very clear compared to the elegance of Python, and it is not uncommon for authors of regular expressions to lose track of what their effort is supposed to do.

There are ways to make regular expressions more legible, but be aware of the code clarity issues that regular expressions can cause.

"When the only tool you own is a hammer, every problem begins to resemble a nail."
-Abraham Maslow, American educator

You've just been introduced to the world of regular expressions, an amazingly powerful toolbox that can do incredible things. You've also been warned about the dangers of regular expressions. There is still much more to learn, and the Python documentation describes regular expressions in rather more detail (a confusing amount of detail for beginners, we suspect).

For over two courses and about thirty lessons, we've been able to rely on string methods. And that is because Python's string methods are fast and powerful, and yet easy to use. By all means, continue to use them when it is easy and faster to do so.

Sometimes regular expressions aren't the right tool for the job. Sometimes it pays to write a dozen lines of Python code instead of a single regular expression. There are no hard and fast rules to follow; it is just something that you learn over time.

For some reason, we find that regexes enthuse people to the point that they become the hammer with which they try to solve all string-processing problems. Don't let this happen to you.