Compiling and Flagging Regular Expressions
In preceding lessons, you learned about regular expressions and their basic use in the Python language. In this lesson, you'll learn about how to compile regular expression patterns, Python's special regular expression flags, additional pattern matching strings, and we'll try more examples. By the end of this lesson, you'll know enough to handle most of your regex needs.
This lesson includes these sections:
So far, we've used the module-level functions in Python's re library in order to do pattern matches. The advantage of this is that it makes for quick-to-write code, but from a performance point of view, it is not the most efficient method. For the small examples we've used so far, it hasn't been a problem, but regular expressions are often called in huge volumes on gigantic strings and the module-level functions have their limits. So when you anticipate a need for greater performance, it is a good practice to compile the regular expressions before use.
Compiled regular expressions are called a pattern object. All of your favorite Python re search functions are methods of the pattern object. Actually, many of these methods have additional features that the basic search functions lack, which allow you to really fine-tune your searches.
When you compile your patterns, since they are no longer strings, your code is more compact, more readable, and more usable.
To compile a regular expression into a pattern object, you pass a pattern string into the re.compile() function. Once you've done that, you can start using the re functions you've learned before, such as match(), search(), findall()—albeit now as methods:
>>> import re
>>> regex = re.compile('Python')
>>> my_str = "I'm glad O'Reilly has Python courses and books!"
>>> result = regex.search(my_str)
>>> result
<re.Match object; span=(22, 28), match='Python'>
>>> result.group()
'Python'
>>> regex.match(my_str) == None # Match fails because 'Python' is not at the start
True
>>> regex.findall(my_str)
['Python']
If you continue to play around with the pattern object, you'll see you can use finditer(), sub(), and subn() as methods. Indeed, the pattern object functionality matches that of the core re library functions.
| Modern Python | In Python 3.7 and later, match objects display as
<re.Match object; span=(...), match='...'> rather than the older
<_sre.SRE_Match object at 0x...> form. The new repr is more
informative, showing the matched span and text directly. |
Actually, the statement 'the pattern object functionality matches that of the core re library functions' is incorrect. The pattern object also includes for many of its methods pos and endpos arguments. These act just like string slicing, but if the endpos argument is less than the pos argument, the method returns a None object instead of an empty string on the match() and search() methods and an empty list/iterator for the findall() and finditer() methods, respectively.
>>> new_str = 'Python is a language; a Python is a snake' >>> regex.findall(new_str) ['Python', 'Python'] >>> regex.findall(new_str, 6) # starts at position 6 ['Python'] >>> regex.findall(new_str, 6, 10) # starts at position 6, ends at position 10 [] >>> regex.findall(new_str, 10, 5) [] >>> type(regex.match(new_str, 10, 5)) <class 'NoneType'>
Not all pattern object methods include position arguments, so here is a reference guide:
| Method | Positional Arguments? |
|---|---|
| search | yes |
| match | yes |
| split | no |
| findall | yes |
| finditer | yes |
| sub | no |
| subn | no |
When you get into writing longer and more complex regular expressions, it becomes hard to read the pattern. Wouldn't it be nice to be able to be able to include comments in your regular expressions? Or spread the regular expression across multiple lines without creating false positives? Or ignore alphabet case by default? Or only Flags give you that and more.
Earlier, we used this regular expression to find cities in a text string:
[A-Z][a-z]+(\s[A-Z][a-z]+)*,\s[A-Z]{2}\s\d{5}
This is not very easy to read. Fortunately, we can break it up and still keep it usable, with the re.VERBOSE flag. Let's make an example. Edit city_search.py as shown:
"""
String regular expressions
"""
import re
def city_search(text):
regex = re.compile(r"""
[A-Z][a-z]+ # the first word of a city
(\s[A-Z][a-z]+)* # possible additional words of a city
,\s[A-Z]{2}\s # The two-letter abbreviation for a US state
\d{5} # five-digit US zip code
""", re.VERBOSE)
search = regex.search(text)
if search:
return search.group()
As you can see, when the pattern object is compiled, you passed in re.VERBOSE as an extra argument. This argument allowed you to include white space and Python-style comments without breaking the regular expression.
The trick with this particular flag is that all the white space is removed, except that which is declared, so you need to remember to include \s, \n, \r, \f, \t, and \v instead of literal spaces, tabs, and return characters.
| Modern Python | re.VERBOSE (also available as re.X) remains the
recommended way to write readable complex patterns. Multiple flags can be combined with
|, for example re.VERBOSE | re.IGNORECASE. Since Python 3.6,
flags can also be embedded inline at the start of a pattern string:
r'(?xi)pattern here'. |
If you need to match a pattern and ignore case, the best way to do it is with the re.IGNORECASE flag:
>>> import re
>>> regex = re.compile(r"""python # the language
... |guido # the bdfl
... """, re.IGNORECASE | re.VERBOSE)
>>> for m in regex.findall("""Python was invented by Guido, and while its mascot is a
... python, it was named after Monty Python"""):
... print(m)
...
Python
Guido
python
Python
Note that when we pass in two flags, we use the pipe (|) symbol, thus: re.IGNORECASE | re.VERBOSE