Building and Debugging Whole Programs

The title of this subsection is also the title of a book series being written by Donald Knuth, a computer science professor at Stanford University. Though not yet finished, it already fills four massive volumes. Don't be discouraged if you feel like you don't have a handle on all there is to know about Python yet! If you're going to become good at this, your programming education will be ongoing.

As your grasp of the language increases, you'll inevitably find that when you review code you wrote some time ago, you'll have discovered better ways to express the same algorithms. It's good practice to reevaluate your code occasionally—even if it works, it can probably be improved. On the flipside, as they say, "if it ain't broke, don't fix it." Unless there's a benefit to changing the code (like increased speed or reduced complexity), then leave it alone.

It's all too easy to introduce subtle errors into your programs by changing code to other code you think is equivalent. Until you're a bit more experienced and confident in your decisions, just be satisfied with programs that work.

Two common terms flung about by programmers are top-down and bottom-up design. In top-down design, you defer thinking about the detail of a problem until you have mapped out the overall structure it will have. Working bottom-up, you begin by building a set of primitive operations that you can then fold together with glue logic to solve your problem.

The top-down approach lets you avoid having too much confusing detail to deal with early in the design cycle. Good top-down design focuses first on the program's large-scale architectural features.

The bottom-up approach is useful when you already understand your data and the ways you need to manipulate it. Using a test-driven development approach to programming, you write tests first, and then write your program to pass the tests. Each function and method is written to pass its tests, so you know that your lower-level components do indeed behave as expected.

The top-down and the bottom-up approaches can also be used together on the same project. It's a little like two teams boring a tunnel from opposite sides of a mountain: if the two do not meet, they have not been working harmoniously together.

By taking a top-down approach initially, you can operate a divide-and-conquer scheme, and avoid being overwhelmed by detail early in the design. If your coding problem isn't too complex, you might find that you have already solved it before you ever start working bottom-up.

Agile programming techniques focus on delivering the simplest code that meets the requirements, or as agile practitioners often say, "the simplest thing that could possibly work." Agile methods place great emphasis on refactoring your code when it becomes too complex. Refactoring means changing the way your program is organized without changing its behavior. Refactoring is generally used when handling large programs, but it can be helpful whenever complexity starts to overwhelm you. Refactoring can help you to:

• Remove duplicate code: When two different functions provide the same result, or one function is a special case of another, we refactor the two functions into one, and we'll have less code to maintain.
• Isolate existing logic from a needed change: If you have to change certain cases currently handled by a single class, you might find it advantageous to refactor the class by turning it into two subclasses of a common base class. The changed behavior can then be implemented in just one of the subclasses.
• Make the program run faster: When performance becomes sluggish, it may be that your original choice of algorithm or data structure was inappropriate, so you refactor to streamline your process.

Some aspects of agile development are meant to be used by teams of software developers rather than individuals. Let's go over a few key principles that apply to most agile technologies:

• Design and code are test-driven: Whenever you add functionality to your program, you first write a test, for automatic execution, that checks to make sure that the functionality is present and performs properly. Your work should proceed in small increments—never add two features at the same time.
• Integrate continuously: Each time you change or fix a module, after running its tests, integrate the module back into the system and run the system tests to make sure that your change has not had any unintended consequences.
• Refactor mercilessly: If tasks are performed similarly in two places, move them around so they're done in one place instead, and then called or inherited by the two original places. If you have coding standards and they are violated, fix them. If you notice structural defects, fix them. After each change, rerun all of your tests to verify that the refactoring process has not broken your code.
• Release early and often: Release your program to the users before adding too many features. You can use their feedback to guide further development, and deliver the most important functions of your program faster.
• Keep it simple: Don't include complexity that you think might be handy later. Simplicity has many benefits, and often "later" never arrives.
• Code is not owned: Agile programming is a team effort, so it is never "Joe's code" or "Jim's code;" it's "our code." Never fear changing code created by someone else—it's yours to use and testing should help you make sure you don't break it.

Python comes with two testing frameworks built-in. If you have been using the JUnit testing framework, consider using the unittest module, which is based on JUnit. You'll probably find the doctest module easier to use, because it works by embedding executable Python statements and their expected outcomes into the docstrings that are embedded into all Python code.

Because the docstrings are available to the program, testing framework can use information embedded in them to verify that code is functioning correctly.

To see how doctest works, create a new program as shown:

"""Demonstrates the doctest module in action."""

def square(x):
    '''Returns the square of a numeric argument.

    >>> square(3)
    9
    >>> square(1000)
    1000000
    >>> square("x")
    Traceback (most recent call last):
    ...
    TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int'
    '''
    return x*2

def _test():
    import doctest, testable
    return doctest.testmod(testable)

if __name__ == "__main__":
    _test()

Save it as testable.py and run it. This program contains a bug: instead of returning its argument raised to the second power (squared), the squared() function returns its argument multiplied by two. This is an easy mistake to make—we only left out a single asterisk—but it renders the function incorrect. Our output looks like this:

**********************************************************************
File "testable.py", line 6, in testable.square
Failed example:
    square(3)
Expected:
    9
Got:
    6

**********************************************************************
File "testable.py", line 8, in testable.square
Failed example:
    square(1000)
Expected:
    1000000
Got:
    2000

**********************************************************************
File "testable.py", line 10, in testable.square
Failed example:
    square("x")
Expected:
    Traceback (most recent call last):
    ...
    TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int'
Got:
    'xx'

**********************************************************************
1 item had failures:
   3 of   3 in testable.square
***Test Failed*** 3 failures.

When you run the program, it calls the _test() function, which in turn imports the doctest module. It also imports the program itself, and then finally calls the doctest.testmod() function with the module as an argument. This causes the examples in the square() function's docstring to be run, and compared with the output listed under each expression.

Because the results don't agree with the predictions in the docstring, the differences are reported as errors, and the output makes it clear that something is wrong with the program.

Let's fix the error by changing the operation in the square() function to an exponentiation (feel free to toss the word exponentiation into conversation as well, to impress your friends), as shown:

"""Demonstrates the doctest module in action."""

def square(x):
    '''Returns the square of a numeric argument.
    '''Returns the effective length of a string
    allowing for tabs of a given length tlen.

    >>> square(3)
    9
    >>> square(1000)
    1000000
    >>> square("x")
    Traceback (most recent call last):
    ...
    TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int'
    '''
    return x**2

def _test():
    import doctest, testable
    return doctest.testmod(testable)

if __name__ == "__main__":
    _test()

Save and run it. You get no output. That's good. The doctest system is designed to help you to detect when your code is working incorrectly and to hone in on the tests that are failing. You'll learn more about testing in other courses, but for now, doctest is a great place to start. Your doctests can be integrated into other schemes as you move forward.

The KISS principle (albeit a tad harsh) is one that programmers find helpful. Of course, when you're first learning a language, sometimes nothing seems simple. Breaking up our operations into smaller pieces helps us understand the big picture. We can see, then, that every program is made up of a sequence of operations. Each operation is either a basic statement, or a choice between several alternatives, or a loop. When the user makes a choice, the action to be taken is a sequence of operations—and each operation can be a basic statement, or a choice between several alternatives, or a loop.

The concept of refactoring code can be compared to the editions of a textbook over time. The first edition provides the main body of text, while in following editions, editors clean up mistakes, make style changes, or add more information. The core content of the book doesn't really change, but the details get better.

When you refactor, you aren't adding new functionality, you are making the code better. You exchange duplicate code for calls and inheritance where possible, fix structural defects, change code to match coding standards (if you have them), and most importantly, make sure that it passes all of your tests.

If you are going to refactor your code mercilessly, you must have tests. Without sufficient testing, you cannot be certain that your changes have not broken your program.

Let's take some code and refactor it. It isn't often we can strive to be merciless, so let's enjoy this rare opportunity! In our sample program, we have some code that is truly miserable to look at, but it works. Create the file shown below:

"""Demonstrates an opportunity for refactoring."""

def list_multiply(LIST_A, LIST_B):
    """ Sums two lists of integers and multiplies them together

    >>> list_multiply([3,4],[3,4])
    49
    >>> list_multiply([1,2,3,4],[10,20])
    300
    """

    TOTAL_A = 0
    for i in LIST_A:
        TOTAL_A += i
    TOTAL_B = 0
    counter = 0
    while True:
        if counter > len(LIST_B) - 1:
            break
        TOTAL_B = TOTAL_B + LIST_B[counter]
        counter += 1
    return TOTAL_A * TOTAL_B

def _test():
    import doctest, refactor
    return doctest.testmod(refactor)

if __name__ == "__main__":
    _test()

Save it as refactor.py and run it. You should get no errors, but if this code makes you wince, then you are developing sound Python instincts! While the code is technically correct, it just plain smells. Some variables are upper-case and some are lower-case. Two different loops are used to do the same action of summing up the integers in two lists, when a simple built-in sum() function would suffice. Can you imagine making the necessary alterations if you had to add the capability to handle a third or fourth list to your code? Ouch.

Fortunately the code comes with doctests, so you can do some (merciless?) refactoring. Edit the program as shown:

"""Demonstrates an opportunity for rRefactored versiong of previous example."""

def list_multiply(LIST_Aa, LIST_Bb):
    """ Sums two lists of integers and multiplies them together

    >>> list_multiply([3,4],[3,4])
    49
    >>> list_multiply([1,2,3,4],[10,20])
    300
    """

    TOTAL_A = 0
    for i in LIST_A:
        TOTAL_A += i
    TOTAL_B = 0
    counter = 0
    while True:
        if counter > len(LIST_B) - 1:
            break
        TOTAL_B = TOTAL_B + LIST_B[counter]
        counter += 1
    return TOTAL_A * TOTAL_B
    return sum(a) * sum(b)

def _test():
    import doctest, refactor
    return doctest.testmod(refactor)

if __name__ == "__main__":
    _test()

Huge difference! Save and run it again; the doctest should still work. Refactoring like this allows you to make changes to improve your code without the fear of breaking it.

And if you need to add functionality, refactored code makes it that much easier. Because the code is generally simpler (always remember KISS), it will be less difficult to extend it to work with any number of lists of integers. Try this version of the code, and make sure it passes the tests:

"""RAdding functionality, much easier with refactored vcodersion of previous example.!"""

def list_multiply(a, b*lists):
    """ Sums twany number of lists of integers and multiplies them together

    >>> list_multiply([3,4],[3,4])
    49
    >>> list_multiply([1,2,3,4],[10,20])
    300
    >>> list_multiply([4,3,2,1],[50,50],[5,5,5])
    15000
    """

    return sum(a) * sum(b)
    total = 1
    for l in lists:
        total *= sum(l)

    return total

def _test():
    import doctest, refactor
    return doctest.testmod(refactor)

if __name__ == "__main__":
    _test()

Save and run it again; the doctest should still work. Wow. Remember when you were a total Python newbie? You've come a long way since Lesson 1! Now you know almost all of Python's syntax, and you're familiar with the statements that make up the language. You know how to structure programs as sets of functions, and how to deliver functions in modules that can be re-used by several different programs.

You still don't know all there is to know about Python (who does?), but now you're in position to understand much of the Python code you encounter. Read lots of Python code; it's a great way to learn more about the language and to increase your understanding of the library and third-party modules it uses. You can practice doing just that and applying the Python tools you've acquired here, in your final project. Good luck!