Engineering Your Programs
In a previous lesson, we learned how to use sys.argv to access command line elements. However, sys.argv is only useful for providing per-run information about what you want a program to do. If there are actions you want a program to always take, you need a mechanism that allows that (while still allowing the use of command-line arguments for the per-run data).
For example, problems quickly arise if you need to accept several arguments. Let's say besides logging when you start a program, you need to point a program to a specific database, let the user set a specific directory to save files, and accept user name/password combinations. Now instead of one, you have five command-line arguments, each of which needs validation and precise help instructions. The logic to handle this would likely involve lots of nested if blocks to handle field determination/validation and print() calls for help instructions, and you'd spend significant effort, not just in writing and testing the command-line code, but also in maintaining it.
Thankfully, Python provides two libraries for handling this exact issue. The first library, optparse, is a more powerful command-line system than simply processing sys.argv "by hand." The second library, configparser, lets you create configuration files, often used to establish default program settings that can become, either for a single user or across a system, the defaults for command-line operation. This can sometimes shorten the "average" command line.
| Modern Python | optparse has been deprecated since Python 3.2. The recommended replacement
is argparse (also in the standard library), which supports sub-commands, type
coercion, and richer help formatting. Everything in this lesson translates directly to
argparse: ArgumentParser replaces OptionParser, and
add_argument() replaces add_option(). The configparser
library is unchanged and remains the standard approach for INI-style configuration files. |
This lesson includes these sections:
optparse is a convenient, flexible, and powerful library for parsing command-line options. It follows the conventional GNU/POSIX syntax, which sounds fancy but really just means that command-line users on Windows, Mac, Unix, Linux, and BSD will find it matches the general operation of their existing command-line tools.
Here we'll see how to capture a loglevel using the optparse library. This behavior may be useful to other programs, so we'll implement it in a new file. Create commands.py as shown:
"""
commands.py: Parse logging level options from sys.argv
"""
from optparse import OptionParser
if __name__ == "__main__":
# instantiate an OptionParser object
parser = OptionParser()
parser.add_option("-l", "--loglevel",
action="store",
dest="level",
default="warning",
help="set level of logger: debug, info, warning (default), error, critical")
(options, args) = parser.parse_args()
print("level: %s" % options.level)
Now, let's try this out by using -l debug as command-line arguments.
level: debug
Try the same thing with -l critical for a different result:
level: critical
Run it again, leaving the arguments field empty. It even provides a default value:
level: warning
Pretty handy, but besides a lot more typing, this isn't doing anything that sys.argv doesn't do, right? Let's go ahead and prove that assumption wrong. Run it with the -h argument:
Usage: commands.py [options]
Options:
-h, --help show this help message and exit
-l LEVEL, --loglevel=LEVEL
set level of logger: debug, info, warning (default),
error, critical
There you have it—instant help! And help that follows the same format that you get any time you do '-h' or '--h' on a command-line tool. Also, note that the print() command did not run. This is because all Python programs, regardless of whether or not they use the optparse library, do not run any code except to produce the help text when the user calls for help. So users can call the -h command without fear that they will inadvertently run your program.
Let's do something familiar and create a very simple email address book program. It will allow you to add, delete, and list all addresses from the command line. The addresses will be stored using the shelve module.
The first thing to do is to get our program to add and delete emails. You'll need two options for this, the first one to let your users pick the add, edit, or delete actions, and the second being the email value in question. So create addressbook.py as shown below.
from optparse import OptionParser
if __name__ == '__main__':
parser = OptionParser()
parser.add_option('-a', '--action', dest="action", action="store", help="requires -e option. Actions: add/delete")
parser.add_option('-e', '--email', dest="email", action="store", help="email used in the -a option")
(options, args) = parser.parse_args()
Save and run it with the -h option:
Usage: addressbook.py [options]
Options:
-h, --help show this help message and exit
-a ACTION, --action=ACTION
requires -e option. Actions: add/delete
-e EMAIL, --email=EMAIL
email used in the -a option
Now, we'll add some more validation. First we'll check that, when a user provides the --action option, they also provide an --email option. Then we'll check that the email provided is valid (for the sake of simplicity, we'll just check that it contains the "@" character).
The first validation, that --action has an --email (and vice versa) is done by checking that if one of those options exists, so should the other. If only one exists, a parser.error() is called. Edit addressbook.py as shown:
from optparse import OptionParser
if __name__ == '__main__':
parser = OptionParser()
parser.add_option('-a', '--action', dest="action", action="store",
help="requires -e option. Actions: add/delete")
parser.add_option('-e', '--email', dest="email", action="store",
help="email used in the -a option")
(options, args) = parser.parse_args()
# validation
if options.action and not options.email:
parser.error("option -a requires option -e")
elif options.email and not options.action:
parser.error("option -e requires option -a")
print(options)
Save and run it with -a add:
Usage: addressbook.py [options] addressbook.py: error: option -a requires option -e
Now, run it with -e steve@oreilly.com:
Usage: addressbook.py [options] addressbook.py: error: option -e requires option -a
The requirement for both options to appear together is working, so it only remains to ensure that when both options are present they are correctly captured in the options dict. You can do this by running the program with both options.
Run it with -a steve -e something:
{'action': 'steve', 'email': 'something'}
Now, can you figure out how to validate that user-provided email includes "@"? Try it before we show you!
...
...
...
...
...
Go ahead; try it!
...
...
...
...
...
We're waiting!
...
...
...
...
...
You should have arrived at something like this:
from optparse import OptionParser
if __name__ == '__main__':
parser = OptionParser()
parser.add_option('-a', '--action', dest="action", action="store",
action="store", help="requires -e option. Actions: add/delete")
parser.add_option('-e', '--email', dest="email", action="store",
action="store", help="email used in the -a option")
(options, args) = parser.parse_args()
# validation
if options.action and not options.email:
parser.error("option -a requires option -e")
elif options.email and not options.action:
parser.error("option -e requires option -a")
elif options.email and '@' not in options.email:
parser.error("option -e requires a valid email address")
print(options)
Run it with -a steve -e something:
Usage: addressbook.py [options] addressbook.py: error: option -e requires a valid email address
Okay, it's time to add the code that handles the emails. But before we do that, let's add the obligatory tests. Create test_addressbook.py as shown:
import unittest
import addressbook
class TestEmailHandlers(unittest.TestCase):
def setUp(self):
self.email = 'test123@t.com'
def test_email_delete(self):
addressbook.email_add(self.email) # ensure the email is active
self.assertEqual(addressbook.email_delete(self.email)[0], True)
self.assertEqual(addressbook.email_delete(self.email)[0], False)
def test_email_add(self):
self.assertEqual(addressbook.email_add(self.email)[0], True)
self.assertEqual(addressbook.email_add(self.email)[0], False)
if __name__ == "__main__":
unittest.main()
Now edit the addressbook program to accommodate the tests:
from optparse import OptionParser
import shelve
import sys
shelf_location = 'email.shelf'
def email_add(email):
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
emails = shelf['emails']
if email in emails:
message = False, 'Email "%s" already in address book' % email
else:
emails.append(email)
message = True, 'Email "%s" added to address book' % email
shelf['emails'] = emails
shelf.close()
return message
def email_delete(email):
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
emails = shelf['emails']
try:
emails.remove(email)
message = True, 'Email "%s" removed from address book' % email
except ValueError:
message = False, 'Email "%s" was not in the address book' % email
shelf['emails'] = emails
shelf.close()
return message
def main(options):
"routes requests"
if options.action == 'add':
return email_add(options.email)
elif options.action == 'delete':
return email_delete(options.email)
if __name__ == '__main__':
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
shelf.close()
parser = OptionParser()
parser.add_option('-a', '--action', dest="action", action="store",
help="requires -e option. Actions: add/delete")
parser.add_option('-e', '--email', dest="email",
action="store", help="email used in the -a option")
(options, args) = parser.parse_args()
# validation
if options.action is None:
sys.exit("You must specify an action (add or delete) with '-a action'")
if options.action and not options.email:
parser.error("option -a requires option -e")
elif options.email and not options.action:
parser.error("option -e requires option -a")
elif options.email and '@' not in options.email:
parser.error("option -e requires a valid email address")
print(main(options)[1])
Of course, you wrote tests for this code before writing it. Better try out the tests before trying to exercise the code. Save both programs and run test_addressbook.py:
.. ---------------------------------------------------------------------- Ran 2 tests in 0.003s OK
Well, that seemed to work out OK, or at least the tests seem to indicate that the add and delete functionality is succeeding and failing where expected. So let's see what we get with various calls from the command line.
Run addrbook.py with -a add -e steve@h.com:
Email "steve@h.com" added to address book
Run addrbook.py with -a add -e steve@h.com again:
Email "steve@h.com" already in address book
Run addrbook.py with -a delete -e steve@h.com:
Email "steve@h.com" removed from address book
Run addrbook.py with -a delete -e steve@h.com again:
Email "steve@h.com" was not in the address book
You've now got a grip on quite a few of the fundamentals of using optparse. Notice how, once the code gets past optparse validation, the action turns to functions. This makes things much easier to extend and test, in turn helping you to reuse this code in other modules. In fact, the email validation ought to take place in its own function called by the email handlers, and would raise an exception that would be caught by the parse handler. Something like this could work:
from optparse import OptionParser
import shelve
import sys
shelf_location = 'email.shelf'
class InvalidEmail(Exception):
pass
def validate_email(email):
if '@' not in email:
raise InvalidEmail("Invalid email: "+email)
def email_add(email):
validate_email(email)
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
emails = shelf['emails']
if email in emails:
message = False, 'Email "%s" already in address book' % email
else:
emails.append(email)
message = True, 'Email "%s" added to address book' % email
shelf['emails'] = emails
shelf.close()
return message
def email_delete(email):
validate_email(email)
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
emails = shelf['emails']
try:
emails.remove(email)
message = True, 'Email "%s" removed from address book' % email
except ValueError:
message = False, 'Email "%s" was not in the address book' % email
shelf['emails'] = emails
shelf.close()
return message
def main(options):
"routes requests"
if options.action == 'add':
return email_add(options.email)
elif options.action == 'delete':
return email_delete(options.email)
if __name__ == '__main__':
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
shelf.close()
parser = OptionParser()
parser.add_option('-a', '--action', dest="action", action="store",
help="requires -e option. Actions: add/delete")
parser.add_option('-e', '--email', dest="email",
action="store", help="email used in the -a option")
(options, args) = parser.parse_args()
# validation
if options.action is None:
sys.exit("You must specify an action (add or delete) with '-a action'")
if options.action and not options.email:
parser.error("option -a requires option -e")
elif options.email and not options.action:
parser.error("option -e requires option -a")
elif options.email and '@' not in options.email:
try:
print(main(options)[1])
except InvalidEmail:
parser.error("option -e requires a valid email address")
print(main(options)[1])
Go ahead and test this code. All tests should continue to pass.
.. ---------------------------------------------------------------------- Ran 2 tests in 0.016s OK
Then run the program itself. It should work just as before, but the refactoring makes the code more easily extended.
Now, suppose we want to list the contents of the shelf file. For this, all we need is an option without a value. Perhaps just -d or --display without a value to show every address in the system. We'll do it with this parser option:
parser.add_option('-d',
'--display', dest="display", action="store_true", help="show all emails")
In this parser option, the action of 'store_true' means that if you call it via the box above, the display attribute of options will be a boolean True. Otherwise it is a None object. With that in your tool-chest, you can add to your existing code base. First, as usual, we'll add a test for the new functionality. Edit test_addressbook.py as shown:
import unittest, shelve
import addressbook
class TestEmailHandlers(unittest.TestCase):
def setUp(self):
self.email = 'test123@t.com'
shelf_location = addressbook.shelf_location
shelf = shelve.open(shelf_location)
if 'emails' in shelf:
if self.email in shelf['emails']:
shelf['emails']=[]
shelf.close()
def test_email_delete(self):
addressbook.email_add(self.email) # ensure the email is active
self.assertEqual(addressbook.email_delete(self.email)[0], True)
self.assertEqual(addressbook.email_delete(self.email)[0], False)
def test_email_add(self):
self.assertEqual(addressbook.email_add(self.email)[0], True)
self.assertEqual(addressbook.email_add(self.email)[0], False)
def test_email_display(self):
addressbook.email_add(self.email)
val, display = addressbook.email_display()
self.assertTrue(self.email in display)
if __name__ == "__main__":
unittest.main()
Now, add the functionality; edit addressbook.py as shown:
from optparse import OptionParser
import shelve
shelf_location = 'email.shelf'
class InvalidEmail(Exception):
pass
def validate_email(email):
if '@' not in email:
raise InvalidEmail("Invalid email: "+email)
def email_add(email):
validate_email(email)
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
emails = shelf['emails']
if email in emails:
message = False, 'Email "%s" already in address book' % email
else:
emails.append(email)
message = True, 'Email "%s" added to address book' % email
shelf['emails'] = emails
shelf.close()
return message
def email_delete(email):
validate_email(email)
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
emails = shelf['emails']
try:
emails.remove(email)
message = True, 'Email "%s" removed from address book' % email
except ValueError:
message = False, 'Email "%s" was not in the address book' % email
shelf['emails'] = emails
shelf.close()
return message
def email_display():
shelf = shelve.open(shelf_location)
emails = shelf['emails']
shelf.close()
text = ''
for email in emails:
text += email + '\n'
return True,text
def main(options):
"routes requests"
if options.action == 'add':
return email_add(options.email)
elif options.action == 'delete':
return email_delete(options.email)
elif options.display == True:
return email_display()
if __name__ == '__main__':
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
shelf.close()
parser = OptionParser()
parser.add_option('-a', '--action', dest="action", action="store",
help="requires -e option. Actions: add/delete")
parser.add_option('-e', '--email', dest="email",
action="store", help="email used in the -a option")
parser.add_option('-d', '--display', dest="display", action="store_true",
help="show all emails")
(options, args) = parser.parse_args()
# validation
if options.action and not options.email:
parser.error("option -a requires option -e")
elif options.email and not options.action:
parser.error("option -e requires option -a")
try:
print(main(options)[1])
except InvalidEmail:
parser.error("option -e requires a valid email address")
Run your tests, and then add some emails and run addressbook.py with the -d/--display flag. You'll get a printed display of all your email entries.
Let's say we want to change the -d/--display flag to provide a number of records based on an integer we pass in. Normally that means we'll have to do type checking via the int() built-in, but with optparse, we get a shortcut.
parser.add_option('-d', '--display', dest="display", type="int",
action="store_true", help="show all emails limited by value")
The optparse module also includes type checking for string, float, and choices. These should all be obvious, except for choices.
The optparse module lets you easily write scripts that handle arguments in a fashion that is consistent with the rest of the world. In other words, it is common to put all of your optparse code under the if __name__ == '__main__' block of code, since that means if another module extends your code it doesn't trigger the optparse code in your program.
Let's say you just bought a brand new computer. The first time you start it up, the computer asks you your name, password, time zone, language, and probably some other questions. It isn't hard to do, but it takes away from your time with your new machine. Wouldn't it be nice if you could simply save this configuration information on one computer and place it on another as needed?
Actually, you can. System Engineers often use tools that set up computers with all the configuration information set exactly how they want it. With some automated scripting, they can start up a new computer this way in minutes and sometimes seconds. This is how companies that provide hosting for individuals or firms that run gigantic server farms can maintain hundreds and thousands of machines.
Python's configparser library provides an easily used API for interacting with one of the popular formats used to save configurations, the INI file format. Frequently associated with Microsoft Windows, INI is in fact also used by other platforms such as Linux and Mac OS X.
In previous courses and earlier in this lesson, we used simple files, pickle, shelve, or SQL databases to save information. The information that handled your settings was coded right into your programs. While this works on small projects under well-defined academic circumstances it can be problematic under professional conditions. For example, because Python is so portable you might save data on Windows at c:\data\emails.shelf, but this simply won't work on Linux or Mac OS X, which might want to see something like /usr/local/data/emails.shelf. Python has tools that make it easy to detect operating systems, but then users might want to save their data in a specific location. This forces them to change your code to store data where they want, which introduces the risk of breaking your code, and only works if they were actually given access to your code (source files).
This is where config files can be priceless. Users not familiar with Python can quickly figure out the format and change things. Furthermore, since the file usually has a .cfg (or less commonly, .ini) extension, most users will be able to quickly identify it as a configuration file.
So, let's make a configuration file. Create addressbook.cfg as shown:
[database] # mac os x or linux # file = /workspace/Python3_lesson12/src/email.shelf # windows file = email.shelf
[database] is a section header. That means any option variables defined under it use "database" as part of the process of displaying them. Under that are a series of comments that use Python '#' syntax so that they are not loaded. Finally, file = email.shelf sets the file variable under the database section. To display this addressbook.cfg file, create a config.py file as shown:
import configparser
# create a config parser object
config = configparser.RawConfigParser()
# open and read the addressbook.cfg file into the config parser
config.read('addressbook.cfg')
# loop through the sections
for section in config.sections():
print(section)
# get all the options for the current section
for option in config.options(section):
# print the option and its value indented for clarity
text = ' %s = %s' % (option, config.get(section, option))
print(text)
Save and run it:
database
file = email.shelf
As you can see, this gives us the ability to provide per-system config files. This is a good thing, because it means you don't have to worry so much about users needing to change settings. A system administrator can establish a central configuration file (and savvy users can provide their own configurations). Let's use the addressbook.cfg file to set the database location in addressbook.py:
import configparser
from optparse import OptionParser
import shelve
config = configparser.RawConfigParser()
config.read('addressbook.cfg')
shelf_location = config.get('database', 'file')
class InvalidEmail(Exception):
pass
def validate_email(email):
if '@' not in email:
raise InvalidEmail("Invalid email: "+email)
def email_add(email):
validate_email(email)
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
emails = shelf['emails']
if email in emails:
message = False, 'Email "%s" already in address book' % email
else:
emails.append(email)
message = True, 'Email "%s" added to address book' % email
shelf['emails'] = emails
shelf.close()
return message
def email_delete(email):
validate_email(email)
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
emails = shelf['emails']
try:
emails.remove(email)
message = True, 'Email "%s" removed from address book' % email
except ValueError:
message = False, 'Email "%s" was not in the address book' % email
shelf['emails'] = emails
shelf.close()
return message
def email_display():
shelf = shelve.open(shelf_location)
emails = shelf['emails']
shelf.close()
text = ''
for email in emails:
text += email + '\n'
return True,text
def main(options):
"routes requests"
if options.action == 'add':
return email_add(options.email)
elif options.action == 'delete':
return email_delete(options.email)
elif options.display == True:
return email_display()
if __name__ == '__main__':
shelf = shelve.open(shelf_location)
if 'emails' not in shelf:
shelf['emails'] = []
shelf.close()
parser = OptionParser()
parser.add_option('-a', '--action', dest="action", action="store",
help="requires -e option. Actions: add/delete")
parser.add_option('-e', '--email', dest="email",
action="store", help="email used in the -a option")
parser.add_option('-d', '--display', dest="display", action="store_true",
help="show all emails")
(options, args) = parser.parse_args()
# validation
if options.action and not options.email:
parser.error("option -a requires option -e")
elif options.email and not options.action:
parser.error("option -e requires option -a")
try:
print(main(options)[1])
except InvalidEmail:
parser.error("option -e requires a valid email address")
Save and run your tests and your code. There should be no difference in the results. Now, let's see what happens when we don't provide a file option. Comment it out in the cfg file as shown:
[database] # mac os x or linux # file = /workspace/Python3_lesson12/src/email.shelf # windows #file = email.shelf
Save it and run your addressbook.py.
Traceback (most recent call last):
File "addressbook.py", line 7, in <module>
shelf_location = config.get('database', 'file')
File "configparser.py", line 327, in get
raise NoOptionError(option, section)
configparser.NoOptionError: No option 'file' in section: 'database'
Your code has just thrown an exception! You can have the config.get() method pass in a default, but usually you want to leave these exceptions as they are. Users who play with config files need as much information as possible to get their configurations working, and passing in defaults means they may not understand what happens when they pass in a setting incorrectly. Remember, good Python programmers like to be as explicit as possible!
Here's a sample config file that might be used to set up a computer. This is just a simple example to show you how system engineers often work:
[personal] first_name = Steve last_name = Holden age = 33 gender = male [professional] occupation = author website = http://holdenweb.com [location] language = English (USA) timezone = EST [authentication] username = sholden password = like I'm going to tell you
The configparser tool is extremely readable and quite machine-friendly. A significant portion of the Python community uses the INI format to describe critical dependency lists and so do users from other programming languages. While there are other competing formats, such as XML, the INI format remains popular because quite simply it is easy for humans to read and for machines it is as fast as, if not faster than, the others to parse and interpret. This ease of interpretation has meant that XML usage for configuration has declined in recent years while use of the older INI format has grown.