Reading and Writing Files

To create a file, you use the open() function. This function returns a file object. If you try to open a file that does not exist, the interpreter creates a file with that name for you. To see how the open() function works to create a file, start an interactive session and enter these commands:

>>> f = open('funnies.txt', 'w')
>>> f
<_io.TextIOWrapper name='funnies.txt' mode='w' encoding='UTF-8'>

The statement you executed assigns the result of the call you made to the open() function, to the variable f. The open() function opens a file to read, write, or append. In this case, the file is funnies.txt, which names our working file. The second argument 'w' tells open() that we want to write to the file.

When you ask the interpreter to display that variable, it displays the name of the open file associated with a particular object.

Writing to a file is a good way to save information. Python provides two different ways to write to a text file. Let's return to the interactive session and take a look:

>>> f = open('funnies.txt', 'w')
>>> f.write('Larry\n')
6
>>> f.write('Curly\n')
6
>>> f.write('Moe\n')
4
>>> names = ['Groucho\n', 'Chico\n', 'Harpo\n']
>>> f.writelines(names)
>>> f.close()

The write() method adds the string (for example, 'Moe\n') to the current content of your funnies.txt file, and returns the length of the string added (for example, 4—including the \n as one character). The writelines() method takes a list of strings and adds each element to the funnies.txt file. Unlike the print() function, neither method adds newlines to the content it writes, so we include '\n' with each string so it will be on a separate line in the file. The close() method makes sure that all data is written out to the file and that the connection between it and the program is dropped.

with open('funnies.txt', 'w') as f:
    f.write('Larry\n')
    f.writelines(['Groucho\n', 'Chico\n', 'Harpo\n'])

Now that you have some sample data in the funnies.txt file, let's see what Python provides to read its contents. Reopen the file in a readable mode. When you open a file for reading, the file object returned by the open() function is iterable, which means that if you use it in a for loop, each iteration gets the next line from the file until there are no more lines left. Go back to the interactive session and type in the commands as shown:

>>> f = open('funnies.txt', 'r')
>>> f.read()
'Larry\nCurly\nMoe\nGroucho\nChico\nHarpo\n'
>>> f = open('funnies.txt', 'r')
>>> f.readline()
'Larry\n'
>>> f.readlines()
['Curly\n', 'Moe\n', 'Groucho\n', 'Chico\n', 'Harpo\n']
>>> f = open('funnies.txt', 'r')
>>> for line in f:
...    print(line)
...
Larry

Curly

Moe

Groucho

Chico

Harpo

>>> f.read()
''
>>> f.close()

You may have noticed that we opened the funnies.txt file three times. That's because the file content is "used up" by using the read(), readline(), and readlines() methods, and we have to reopen the file to return to the top. read() returns all of the content of the file as a single string. readlines() returns the file content as a list of lines. readline() returns the next line from the file, so when you called it once and then called readlines(), the second call returned a list that didn't include the first line of the file. The last method, f.read() returns nothing because the "pointer" is at the end of the file.

When we're done with the file, we close it with the close() method. This releases resources that Python was using to look at or write to the file. Some programmers assume files will close automatically at the end of a program, but it's better programming "hygiene" to close files when you finish using them.

There are six lines of text in your funnies.txt file. Let's add some more. If you open an existing file with the write option w, you truncate the file's contents and produce an empty file—any new input will replace the file's original contents. So we don't use the write (w) option, using the append option (a) instead, as the second argument to open(). The next example shows the append functionality at work. Enter these commands in your interactive Python console:

>>> f = open('funnies.txt','a')
>>> f.write('A child of five could understand this. Fetch me a child of five.\n')
65
>>> f.write('Room service? Send up a larger room.\n')
37
>>> f.close()
>>> f = open('funnies.txt', 'r')
>>> for line in f:
...     print(line[:-1])
...
Larry
Curly
Moe
Groucho
Chico
Harpo
A child of five could understand this. Fetch me a child of five.
Room service? Send up a larger room.
>>>
>>> f.close()

So, you opened an existing file to append content, and wrote in two new lines before closing it. When you opened it again, all of the old content was followed by the new content you had just written. In this example, you used [:-1] to slice each line to exclude the last character (the newline) from the end. This prevents your code from producing the blank lines that were printed out in the previous example.

As we've seen, when a file is being read or written, it has a "current position." You can change this position using the seek() method. The first argument should be the position you want to move to within the file (an integer—the beginning of the file is always position 0).

If you give a second integer argument, it must be 0, 1, or 2:

Continue your interactive session:

>>> f = open('funnies.txt', 'r')
>>> f.seek(115)
115
>>> f.read()
'Send up a larger room.\n'

You can't seek() past the end of a file. To find the current position in a file, call the tell() method.

Python file objects give you many handy attributes and methods that you can use to analyze a file, including tools to figure out the name of the file associated with the file object, whether a file is opened or closed, readable, writable, how it handles errors, and if it is seekable. Type the commands as shown below:

>>> f = open('funnies.txt','a')
>>> f.name
'funnies.txt'
>>> f.readable()
False
>>> f.writable()
True
>>> f.seekable()
True
>>> f.encoding
'UTF-8'
>>> f.errors
'strict'
>>> f.closed
False
>>> f.close()
>>> f.closed
True

The built-in open() function provides one method to add persistence to your applications. In this case, persistence means that when you turn off or quit the application, the data remains available. So when you use an application to store information, you can end your Python session, turn off your computer, and still come back and find that data later. Persistence can take many forms, from the types of files we're using in this lesson, to database records, to audio/video files, to various document formats.

Sophisticated persistence engines are called databases. Most of the world's data is stored in databases. These are integrated sets of logically organized files or records. Databases can store text, integers, dates, images, and much more. Usually, databases use the relational model, but there are also hierarchical, object, network, and flat-file models.

To demonstrate the power of persistence, and ways to take advantage of it, we'll create a to-do list application as our next example. Create a new file as shown:

"""File-based to-do list maintainer."""

otasks = open('open_tasks.txt','a')
otasks.close()
dtasks = open('done_tasks.txt','a')
dtasks.close()
options = ('add','done','quit')
string_input = 'Pick an option from the list (%s): ' % ', '.join(options)

while True:
    open_tasks = open('open_tasks.txt','r').readlines()
    if open_tasks:
        print('-' * 10)
        print('Open Tasks')
        print('-' * 10)
        for i, task in enumerate(open_tasks):
            print(i, task.strip())

    done_tasks = open('done_tasks.txt','r').readlines()
    print('-' * 12)
    print('Done Tasks')
    print('-' * 12)
    for i, task in enumerate(done_tasks):
        print(i, task.strip())

    inp = input(string_input)
    if inp not in options:
        print('Please pick a valid option')
        continue

    if inp == 'add':
        new_task = input('Enter new task: ')
        tasks = open('open_tasks.txt','a')
        tasks.write(new_task + '\n')
        tasks.close()

    if inp == 'done':
        while True:
            done_task = input('Please enter the number of your completed task: ').strip()
            if done_task.isdigit():
                done_task = int(done_task)
                break
            print('Please enter a task number')

        open_tasks = open('open_tasks.txt','r').readlines()
        for i, task in enumerate(open_tasks):
            if i == done_task:
                print('Task removed: {0}'.format(task))
                open_tasks.remove(task)
                f = open('open_tasks.txt','w')
                f.writelines(open_tasks)
                f.close()
                f = open('done_tasks.txt','a')
                f.write(task)
                f.close()
                break

    if inp == 'quit':
        break

Save it as todo.py, and run it. Without adding any tasks, type quit and press Enter. This program starts out by opening each of two data files to append, and then closes them immediately. This forces the computer to create the files, in case this is the very first run. You'll notice two new files, open_tasks.txt and done_tasks.txt.

Now, run the program again and create a few tasks. Quit the program and start it again, and you will see the tasks you added are still there! Congratulations, you've implemented a persistence engine that uses the flat file model! Open the files and you'll see your tasks have been added to open_tasks.txt.

So far you've stored simple text data, which is easy to read with any text editor. However, simple text data is no substitute for the audio files that store our favorite music! But if we open audio files with a text editor, it displays what appears to be a lot of gibberish. Actually, these files are storing the audio data encoded in binary form. This data is impossible for us to read without a special tool called a hex editor. Fortunately, your computer doesn't have the same limitations. And because the binary file doesn't have to be comprehensible to humans, in many cases it can represent data more efficiently than a simple text file.

All the audio, image, and video files on your computer are binary files. So are any files compressed into zip or tar format. In fact, the majority of programs on your computer—such as your favorite browser—are comprised of binary files, the notable exception being the Python programs you are writing as part of this course. And even those Python programs are compiled into binary format before the computer actually runs them!

Now that you have a background in binary data and files, let's check out how Python can handle a binary file. We'll use the Python logo image below as our example:

Open a Python interactive session and try reading it in binary mode:

>>> i = open('python-logo.gif', 'rb')
>>> i
<_io.BufferedReader name='python-logo.gif'>
>>> print(i.read(1))
b'G'
>>> print(i.read(1))
b'I'
>>> print(i.read(1))
b'F'
>>> i.read(10)
b'89a\xd3\x00G\x00\xf7\x00\x00'
>>> i.tell()
13
>>> i.seek(0)
0
>>> i.read(3)
b'GIF'
>>> i.close()

When you first look at binary data, it can be pretty daunting. Even so, at a glance we can see a useful method and a handy bit of information. The read() method fetches the byte(s) you request. Subsequent read() requests reading from your current location in the file, where the last read() left off. The file is a GIF image, and all GIF files begin with the three bytes 'GIF'

Wat about the b'89a\xd3\x00G\x00\xf7\x00\x00'? Well, that's part of the image content used to generate the Python logo. Our example above also shows the tell() and seek() methods. seek(0) "rewinds" the file to the beginning.

Adding an integer argument to the read() method will read the given number of bytes. If there aren't enough bytes remaining in the file, read(n) returns as many as there are. This means that if you get an empty sequence of bytes back, you are at the end of the file.

Finally, the "strings" that you get when you read a file in binary mode are what we call byte strings—each byte is eight bits, so the ordinal value of the elements is in the range 0 to 255. Regular Python strings should always be used to represent text. Bytestrings are better used for data that is arranged for computational convenience rather than human readability–such as GIF files!

So now you know a little more about files, the basic way to provide persistent storage of information. Files are the basis of most computer-based information storage, so there is a huge amount of literature that covers how to store various types on information in files. For now, the basics will suffice. You can write data out from one program run, and read it back in to make use of it in another program (or a different run of the same program). This is what gives computers the power to run systems with long-term memory.

You should now be reasonably confident about working with files.