Test-Driven Development

Below is an example of the first step of TDD, writing tests. Suppose that you have been asked to develop an adder(x, y) function that takes two arguments and adds them together using a somewhat unusual definition of "add": integer + integer, string + string and list + list, use regular addition; integer + string converts the integer to a string before concatenation; and adding a string or an integer to a list, appends to the list (regardless of whether it's the first or second argument).

Create a file named testadder.py and type in the code below as shown:

"""
Demonstrates the fundamentals of unittest.
adder() is a function that lets you 'add' integers, strings, and lists.
"""

from adder import adder # keep the tested code separate from the tests

import unittest

class TestAdder(unittest.TestCase):

    def test_numbers(self):
        self.assertEqual(adder(3,4), 7, "3 + 4 should be 7")

    def test_strings(self):
        self.assertEqual(adder('x','y'), 'xy', "x + y should be xy")

    def test_lists(self):
        self.assertEqual(adder([1,2],[3,4]), [1,2,3,4], "[1,2] + [3,4] should be [1,2,3,4]")

    def test_number_and_string(self):
        self.assertEqual(adder(1,'two'), '1two', "1 + two should be 1two")

    def test_numbers_and_list(self):
        self.assertEqual(adder(4,[1,2,3]), [1,2,3,4], "4 + [1,2,3] should be [1,2,3,4]")

if __name__ == "__main__":
    unittest.main()

Don't run the program just yet. Although it imports an adder function (the function it's eventually going to test), that import will fail unless that function is defined. The simplest code we have to allow the test harness (automated test framework) to run, is an adder module that contains an empty adder() function. Create adder.py as shown:

"adder.py: defines an adder function according to a slightly unusual definition."

def adder(x, y):
    pass

Now let's go on to step two of the cycle, run tests.

Save adder.py, then run testadder.py.

FFFFF
======================================================================
FAIL: test_lists (__main__.TestAdder.test_lists)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testadder.py", line 17, in test_lists
    self.assertEqual(adder([1,2],[3,4]), [1,2,3,4], "[1,2] + [3,4] should be [1,2,3,4]")
AssertionError: None != [1, 2, 3, 4] : [1,2] + [3,4] should be [1,2,3,4]

======================================================================
FAIL: test_number_and_string (__main__.TestAdder.test_number_and_string)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testadder.py", line 20, in test_number_and_string
    self.assertEqual(adder(1,'two'), '1two', "1 + two should be 1two")
AssertionError: None != '1two' : 1 + two should be 1two

======================================================================
FAIL: test_numbers (__main__.TestAdder.test_numbers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testadder.py", line 11, in test_numbers
    self.assertEqual(adder(3,4), 7, "3 + 4 should be 7")
AssertionError: None != 7 : 3 + 4 should be 7

======================================================================
FAIL: test_numbers_and_list (__main__.TestAdder.test_numbers_and_list)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testadder.py", line 23, in test_numbers_and_list
    self.assertEqual(adder(4,[1,2,3]), [1,2,3,4], "4 + [1,2,3] should be [1,2,3,4]")
AssertionError: None != [1, 2, 3, 4] : 4 + [1,2,3] should be [1,2,3,4]

======================================================================
FAIL: test_strings (__main__.TestAdder.test_strings)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testadder.py", line 14, in test_strings
    self.assertEqual(adder('x','y'), 'xy', "x + y should be xy")
AssertionError: None != 'xy' : x + y should be xy

----------------------------------------------------------------------
Ran 5 tests in 0.001s

FAILED (failures=5)

All five tests have failed. But we expected them to fail (yes, we did), because our adder() method doesn't actually do anything yet. While failed tests are not the ideal result, at least the tests didn't result in error messages. When you see error messages, they usually indicate the presence of a programming mistake, for instance, a function may have the wrong number of arguments, or a call to a method that an object doesn't have. But since our code didn't return any error messages, we can move on to step three of the TDD cycle: write code to pass the tests. In this first instance, we won't try and pass all of the tests, but instead provide a basic initial implementation that will pass some of them, then build from there. Edit adder.py, adding and removing code as shown:

"adder.py: defines an adder function according to a slightly unusual definition."

def adder(x, y):
    pass
    
    return x + y

Run testadder again.

.E.E.
======================================================================
ERROR: test_number_and_string (__main__.TestAdder.test_number_and_string)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testadder.py", line 20, in test_number_and_string
    self.assertEqual(adder(1,'two'), '1two', "1 + two should be 1two")
  File "adder.py", line 4, in adder
    return x + y
TypeError: unsupported operand type(s) for +: 'int' and 'str'

======================================================================
ERROR: test_numbers_and_list (__main__.TestAdder.test_numbers_and_list)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testadder.py", line 23, in test_numbers_and_list
    self.assertEqual(adder(4,[1,2,3]), [1,2,3,4], "4 + [1,2,3] should be [1,2,3,4]")
  File "adder.py", line 4, in adder
    return x + y
TypeError: unsupported operand type(s) for +: 'int' and 'list'

----------------------------------------------------------------------
Ran 5 tests in 0.001s

FAILED (errors=2)

The first line now contains three dots, each representing a successful test (give yourself a pat on the back for those!), and two "E" characters. Those E's represent errors that we get because our implementation works for only 60% of the test cases. That's not bad for a one-line function though, and the output from the test-run provides lots of information that helps us figure out how to stop the function from throwing exceptions and causing those errors.

The problems in our code seem to pop up when the arguments aren't of the same type. Since the function appears to do what we need it to do most of the time, we'll modify our program explicitly to change its performance just in the failing cases. We'll do that by adding an integer and a string, and adding an integer and a list (this last case should apply when adding anything to a list, not just an integer).

Edit your code as shown below:

"adder.py: defines an adder function according to a slightly unusual definition."
import numbers

def adder(x, y):
    if isinstance(x, list):
        return x + [y]
    elif isinstance(y, list):
        return y + [x]
    elif isinstance(x, numbers.Number) and isinstance(y, str):
        return str(x) + y
    return x+y

We enhanced our code using the built-in isinstance() function. This function lets us check to see if a variable is of a particular type, or a subclass of that type. We have to import the numbers module in order to use numbers.Number, which is a superclass of all numeric types in Python.

Run testadder again. Now both of the original errors are fixed, but unfortunately, one of the test cases that succeeded previously is now broken. Don't worry too much—this a common occurrence. The good news is that the tests work and let us know about the problems!

F....
======================================================================
FAIL: test_lists (__main__.TestAdder.test_lists)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testadder.py", line 17, in test_lists
    self.assertEqual(adder([1,2],[3,4]), [1,2,3,4], "[1,2] + [3,4] should be [1,2,3,4]")
AssertionError: Lists differ: [1, 2, [3, 4]] != [1, 2, 3, 4]

First differing element 2:
[3, 4]
3

Second list contains 1 additional elements.
First extra element 3:
4

- [1, 2, [3, 4]]
?        -     -

+ [1, 2, 3, 4] : [1,2] + [3,4] should be [1,2,3,4]

----------------------------------------------------------------------
Ran 5 tests in 0.001s

FAILED (failures=1)

In the final version of our code, we want to make sure that the new special cases for lists are not applied when both arguments are lists. In those cases we want them to be left to the default elif case at the end of the function. Modify your code as shown:

"adder.py: defines an adder function according to a slightly unusual definition."
import numbers

def adder(x, y):
    if isinstance(x, list) and not isinstance(y, list):
        return x + [y]
    elif isinstance(y, list) and not isinstance(x, list):
        return y + [x]
    elif isinstance(x, numbers.Number) and isinstance(y, str):
        return str(x) + y
    return x+y

Run testadder again. Nice. At last we have the pleasure of seeing all of our tests pass, with five dots on the first line of the output. Programmers who use unittest regularly often refer to themselves as "dot-addicted." It's amazing how gratifying it can be to see a row of dots printed out from a test!

.....
----------------------------------------------------------------------
Ran 5 tests in 0.000s

OK 

The TestCase class is the cornerstone of the unittest module. We've learned to create our own test cases as subclasses of TestCase. Individual tests are written as methods of the subclass and have names that begin with the string "test." If you have only one test to run, you may implement that test as the class's runTest() method. You probably won't do that very much, but you may see it in other people's code, so it's worth knowing.

If you want to define several tests, you could create a separate TestCase subclass for each one, but it's much simpler to create a single subclass with several test methods instead. So, why might you need more than one TestCase subclass? Well, one possibility is so that you can include setUp() and tearDown() methods, which would be run before and after each test method. In this case (as well as in others), grouping tests that require the same set-up and tear-down processing, is a good way to go.

Suppose you want to run some tests of code you have written that creates files. Each test needs to create files. And since the tests create random files (or at least since each test creates different files), if you run the tests in any old directory, clean-up could be difficult. To avoid creating such problems for ourselves, we'll write our code so that each test method creates the directory itself and cleans up the files it creates. To make our code even more efficient, we'll have it call a function to create the directory which was called by each test method. We could take it even further and create the directory within the setUp() method. This is called automatically before the framework calls each test method, just as the tearDown()method is called after each one. So we could use tearDown() to empty and delete the directory.

If the setUp() method raises an exception, the test framework will declare this test to have errors, and the test method will not be run. If it succeeds, the test is run, followed by the tearDown() method.

Let's check this out. Create a new program named setupDemo.py and type in the following code:

"""
Demonstration of setUp and tearDown.
The tests do not actually test anything - this is a demo.
"""
import unittest
import tempfile
import shutil
import glob
import os

class FileTest(unittest.TestCase):

    def setUp(self):
        self.origdir = os.getcwd()
        self.dirname = tempfile.mkdtemp("testdir")
        print("Created", self.dirname)
        os.chdir(self.dirname)

    def test_1(self):
        "Verify creation of files is possible"
        for filename in ("this.txt", "that.txt", "the_other.txt"):
            f = open(filename, "w")
            f.write("Some text\n")
            f.close()
            self.assertTrue(f.closed)

    def test_2(self):
        "Verify that the current directory is empty"
        self.assertEqual(glob.glob("*"), [], "Directory not empty")

    def tearDown(self):
        os.chdir(self.origdir)
        shutil.rmtree(self.dirname)
        print("Deleted", self.dirname)

if __name__ == "__main__":
    unittest.main()

Here, you have defined a test case with two test methods. In order to make the test runnable anywhere, first the setUp() method saves the process's current directory (obtained with os.getcwd() in an instance variable). Then it uses tempfile.mkdtemp() to create a new temporary directory—the location it chooses will depend on your platform, so the method prints the directory path out for your inspection. Having created the new directory, setUp() then makes it the current directory.

The tearDown() method is called after each test. It makes the saved directory the current directory again (thereby ensuring that the temporary directory is no longer in use), and removes it (along with any content it may have) using shutil.rmtree().

When you run the program, you might see something like this:

..
----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK
Created /tmp/tmpabc123testdir
Deleted /tmp/tmpabc123testdir
Created /tmp/tmpdef456testdir
Deleted /tmp/tmpdef456testdir

Here, the output from the test code itself is mixed with the .. output from the testing framework, making it difficult to see exactly what's happening (though the absence of error messages is reassuring). It isn't usually a good idea to produce output from test cases for a couple of reasons. First, when the test succeeds there should be no output—this makes it much easier to determine whether tests have passed or failed. Second, it's quite possible that nobody will read that output anyway.

So instead, we'll remove the print statements when we modify our code. Let's do that now. Edit setupDemo.py as shown:

"""
Demonstration of setUp and /tearDown.
The tests do not actually test anything much - this is a demo.
"""
import unittest
import tempfile
import shutil
import glob
import os

class FileTest(unittest.TestCase):

    def setUp(self):
        self.origdir = os.getcwd()
        self.dirname = tempfile.mkdtemp("testdir")
        print("Created", self.dirname)
        os.chdir(self.dirname)

    def test_1(self):
        "Verify creation of files is possible"
        for filename in ("this.txt", "that.txt", "the_other.txt"):
            f = open(filename, "w")
            f.write("Some text\n")
            f.close()
            self.assertTrue(f.closed)

    def test_2(self):
        "Verify that the current directory is empty"
        self.assertEqual(glob.glob("*"), [], "Directory not empty")

    def tearDown(self):
        os.chdir(self.origdir)
        shutil.rmtree(self.dirname)
        print("Deleted", self.dirname)

if __name__ == "__main__":
    unittest.main()

Run this module; your output looks like this:

..
----------------------------------------------------------------------
Ran 2 tests in 0.003s

OK

When you run unittest.main(), all subclasses of unittest.TestCase are taken from the module. An instance of each subclass is created, and each method of the class with a name that begins with "test" is called. (These calls are preceded by a call to the setUp() method if it exists, and followed by a call to the tearDown() method if it exists).

All of the above actions are taken when we call the TestCase's run() method. The TestCase class records the results of the call in a special object, and they are summarized in the output of the test framework, after all tests have been run.

There are a number of methods you can call to make assertions about your program's state. The most commonly used TestCase Methods are:

The methods above do have alternative names (assertTrue(), for example, is also known as assert_()), but the names above are preferred. Most of these methods take an optional message argument. If you don't provide a message, unittest will try to formulate one that gives you as much information as possible. To test this, create a new program named messagetest.py as shown:

"""
Demonstrate a message formulated by the unittest system.
"""

import unittest

class DemoCase(unittest.TestCase):

    def testMessage1(self):
        self.assertEqual([1,2,3,4], [1, 2, [3, 4]])

if __name__ == "__main__":
    unittest.main()

Run it; the output looks something like this:

F
======================================================================
FAIL: testMessage1 (__main__.DemoCase.testMessage1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "messagetest.py", line 9, in testMessage1
    self.assertEqual([1,2,3,4], [1, 2, [3, 4]])
AssertionError: Lists differ: [1, 2, 3, 4] != [1, 2, [3, 4]]

First differing element 2:
3
[3, 4]

First list contains 1 additional elements.
First extra element 3:
4

- [1, 2, 3, 4]
+ [1, 2, [3, 4]]
?        +     +


----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

The system has performed a fairly detailed analysis of the differences between the two lists, and points out, in the lengthy message, that the lists differ at element 2, and that the first list has an extra element. This informative message is the result of some recent clean-up work that was done to Python's unittest module. With this tool available, now if you can't come up with a particularly good error message yourself, you can try letting the system generate one for you.

In this lesson, you've learned about some basic functions of the unittest module. This will serve you well during the course, but we've only scratched the surface of unittest!

You have also learned to engage test-driven development practices. For the rest of this course, and all following courses, you'll be expected to use this methodology. By the end of this course, you should be really comfortable with TDD and unittest, and writing tests will become second nature to you!