Unit Testing
Your first lesson in Python 2 picks up where we left off in the Python 1 course, focused on debugging programs. Here you'll learn about the second, and more widely used, built-in Python testing framework, unittest. Unittest is a more formal testing framework, which can be integrated with existing uses of doctest, if necessary.
An important statement contained within Python that you haven't come across before is the assert statement. The syntax for this statement is:
assert condition[, message]
In the assert statement, the condition is tested, and if it evaluates false, an AssertionError exception is raised. If there's a message, it is printed with the AssertionError. Let's try using the assert statement right now in an interactive console window.
Type this code into the interactive interpreter console:
>>> assert 1 == 1 >>> assert 1 == 2 Traceback (most recent call last): File "<stdin>", line 1, in <module> AssertionError >>> assert 1 == 2, "One isn't two and the universe is still rational" Traceback (most recent call last): File "<stdin>", line 1, in <module> AssertionError: One isn't two and the universe is still rational >>>
We use assert statements in our programs, to assert conditions that we believe must always be true. If we are correct, the program runs as expected. But, if a programming error or mistaken assumption invalidates the condition, Python will let us know, usually early in the life of the program.
AssertionError exceptions are handled using the unittest module. To write tests, we create test cases that are subclasses of the unittest.TestCase class. Our subclasses can use the methods defined by the superclass. Many of those methods' names begin with the prefix "assert." By calling these methods, you have the test case make assertions about your program in a controlled environment. Any AssertionErrors that arise are handled by the framework and reported as a failure of the associated test. Other exceptions are regarded as errors.
For our first example, we'll use the square() method from our testable.py code that we created in the "Introduction to Python" course. Our goal now is to write code that will cube the values passed. This will allow us to compare the two testing modules.
Create a new file called testable.py and type the blue code as shown:
"""Demonstrates the unittest module in action."""
import unittest
def cube(x):
'''Returns the cube of a passed value'''
return x*3
class TestCube(unittest.TestCase):
def test_small_number(self):
self.assertEqual(cube(3), 27, "Cube of 3 is not 27")
def test_large_number(self):
self.assertEqual(cube(1000), 1000000000, "Cube of 1000 should be 1000000000")
def test_bad_input(self):
self.assertRaises(TypeError, cube, 'x')
if __name__ == "__main__":
unittest.main()
This program contains a bug: instead of returning its argument raised to the third power (cubed), the cube() function returns its argument multiplied by three. This is an easy mistake to make—we just omitted a single asterisk (*)—but it renders the function incorrect. When you run the program, you see output that looks something like this:
FFF
======================================================================
FAIL: test_bad_input (__main__.TestCube.test_bad_input)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/testable.py", line 17, in test_bad_input
self.assertRaises(TypeError, cube, 'x')
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: TypeError not raised by cube
======================================================================
FAIL: test_large_number (__main__.TestCube.test_large_number)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/testable.py", line 14, in test_large_number
self.assertEqual(cube(1000), 1000000000, "Cube of 1000 should be 1000000000")
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 3000 != 1000000000 : Cube of 1000 should be 1000000000
======================================================================
FAIL: test_small_number (__main__.TestCube.test_small_number)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/testable.py", line 11, in test_small_number
self.assertEqual(cube(3), 27, "Cube of 3 is not 27")
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 9 != 27 : Cube of 3 is not 27
----------------------------------------------------------------------
Ran 3 tests in 0.001s
FAILED (failures=3)
Failures. Bummer. And not only do we have failures, our program gives us even more data than doctest did. For example, our program gives the number of tests, followed by the length of time it took to run the tests, and the tests themselves can be set up to pass messages to the person running the tests.
When you run the program, it calls the unittest.main() method, which runs the unittest Test Runner. The Test Runner looks in your code for test suites, which are identified as Classes that inherit from the unittest.TestCase class. These test suites contain a number of tests, which are class methods that begin with the word "test."
Because the assertions within your unittest methods raise AssertionErrors, the package reports them as test failures, and the output makes it clear that something is wrong with the program. In fact, because of the message arguments passed to the methods, you get a pretty good idea of what is going wrong. Now, fix the error by changing the operation in the cube() function to an exponentiation. Modify testable.py by adding the blue code as shown:
"""Demonstrates the unittest module in action."""
import unittest
def cube(x):
'''Returns the cube of a passed value'''
return x**3
class TestCube(unittest.TestCase):
def test_small_number(self):
self.assertEqual(cube(3), 27, "Cube of 3 is not 27")
self.assertEqual(cube(3), 27, "Cube of 3 should be 27")
def test_large_number(self):
self.assertEqual(cube(1000), 1000000000, "Cube of 1000 should be 1000000000")
def test_bad_input(self):
self.assertRaises(TypeError, cube, 'x')
if __name__ == "__main__":
unittest.main()
With the error now corrected, your output from testable.py looks like this:
... ---------------------------------------------------------------------- Ran 3 tests in 0.000s OK
The three dots at the top represent the three tests. If they had failed, you would have seen an "F" replacing each failure. If there were significant errors, you would have seen an "E." Such error indications usually mean that something is wrong with your logic. You see the test count and the time for the duration of the tests' run. Chances are that for this basic test, you'll get a value of 0.000, but keep in mind that unittests are not performance tests. You'll cover performance tests in a later course.
| Note | We also corrected the failure on test_bad_input. Why? Because the string x can be "multiplied" to give xxx, so no TypeError is raised. Because exponentiation does not work with strings, the cube function must be fixed before the test passes. |
Now that you have the tests working, consider how they work. Look over this color-coded test code:
"""Demonstrates the unittest module in action.""" import unittest def cube(x): '''Returns the cube of a passed value''' return x**3 class TestCube(unittest.TestCase): def test_small_number(self): self.assertEqual(cube(3), 27, "Cube of 3 should be 27") def test_large_number(self): self.assertEqual(cube(1000), 1000000000, "Cube of 1000 should be 1000000000") def test_bad_input(self): self.assertRaises(TypeError, cube, 'x') if __name__ == "__main__": unittest.main()
The test_small_number() method in the TestCube class has a single statement: a call to the assertEqual() method inherited from unittest.testCase. That statement contains an assertion that its first two arguments are equal—that cube(3) is equal to 27. If the values do not match, then the assertion fails and the message "Cube of 3 should be 27" is returned during the test and reported by the framework.
If you include useful assertion error messages, they will help you remember what your tests are supposed to be doing. They will also help other programmers understand your tests. It's easier to figure out what to fix when error messages are meaningful (fortunately, the default messages produced by unittest have improved recently, as well).
In the third test, test_bad_input() checks to see if the cube() function throws a TypeError exception. The first argument provided is the expected exception; the second argument is the function to test; the remaining arguments will be passed to the function in question— the cube() function (a one-argument function, so you see a single additional argument 'x'); it is possible to use both positional and keyword arguments (but the function you are testing doesn't take any keyword arguments). Using this method lets you verify that certain inputs raise specific exceptions.
Now that you've begun to appreciate the value of testing, follow the basic rule of test-driven development (TDD): only write code to make a failing test pass. This means that you begin your development projects by creating tests, which then act as a specification for the behavior of the program. By developing software this way, the programmer is forced to develop only the necessary functionality, and resists including extraneous elements. As the agile programming community says, "YAGNI"—You Ain't Gonna Need It. If it doesn't help you pass a test, it really isn't necessary.
Kent Beck, the creator of Extreme Programming and Test Driven Development, wrote a testing framework for agile programming in the Smalltalk programming language. Later, along with Erich Gamma, he wrote a Java-based implementation of this test framework called JUnit. This test framework has since been ported to many other languages, including Python, where it is sometimes called "PyUnit."
The advantage of unittest is that the core concepts are tried and tested. This is important in a test framework because that means you can rely on it. As we learned in the previous course, if we refactor our code and it still passes the tests, we can be reasonably sure that we haven't introduced an error.
unittest uses these important concepts:
- Test Fixtures: The setup for your tests. Fixtures include creation of temporary databases, servers, and anything else needed to run the test. The fixtures frequently need to be cleaned up after a test. To use a spelling test analogy, think of a Test Fixture as a combination pencil, eraser, test sheet, and word list.
- Test Cases: Each test case is an individual test. It checks for a specific response to an assertion, and then is distilled to a boolean statement. Using the spelling test analogy again, think of a Test Case as a single question on the test.
- Test Suite: A test suite is a collection of Test Cases (or even other Test Suites). Returning to our spelling test again, think of a Test Suite as the set of all questions on the test sheet.
- Test Runner: The software that actually runs the tests. The runner can be launched from the command line, graphical interface, web interface, or any other input method. It returns special values to indicate the success of the tests, and these values can be evaluated by you or by various automated tools. In the spelling test analogy, the test runner would be you, the reader, going through the list of questions.
| Modern Python | Several older unittest.TestCase method aliases have been removed in
current Python: assertEquals is gone — use assertEqual;
failUnless/assert_ are gone — use assertTrue;
failIf is gone — use assertFalse;
failUnlessEqual/failIfEqual are gone — use assertEqual/assertNotEqual.
The lesson code already uses the current names throughout, so no changes are needed here. |
So, which should you use, doctest or unittest? To a certain extent, this is a matter of individual preference. Let's compare the two:
| doctest | unittest |
|---|---|
| More readily accessible | More challenging to learn |
| Documents your code to some degree | Maintains a clean separation between tests and documentation |
| Harder to maintain as features change | Easier to maintain as features change |
| Assertions are more difficult to incorporate | Assertions are the primary tools for verifying correct performance |
| Verbose | Concise |
The Python community generally agrees that while doctests have their place, unittests are usually more useful. doctests are easier to learn, but in the long run, unittests are the more streamlined choice. It is possible to integrate doctests in a unittest environment, though not quite as straightforward as you might like.
| Modern Python | pytest has become a popular third-party alternative to both doctest
and unittest. It lets you write plain assert statements in test functions (no
subclassing required) and produces rich, readable failure output. It is worth knowing about,
though unittest remains entirely standard and widely used. |
Congratulations! Just like that, you are now equipped with a second Python test framework. In the lessons to come, we'll use both test frameworks to check our work and build good programming habits. According to the tenets of agile programming, test-driven development is the way to go. TDD lets you continue to refactor your code without introducing errors, and it encourages other programmers to love you for your devotion to best practices. In the next lesson, we'll explore test-driven development even further. See you there!