Memory-Mapped Files
Files can be so large that it is impractical to load all of their content into memory at once. The mmap.mmap() function creates a virtual file object. Not only can you perform all the regular file operations on a memory-mapped file, you can also treat it as a vast object (far larger than any real object could be) that you can address just like any other sequence.
This technique deals with files by mapping them into your process's address space. The mmap module allows you to treat files as similar to bytearray objects—you can index them, slice them, search them with regular expressions and the like. Many of these operations can make it much easier to handle the data in a file: without memory mapping, you have to read the file in chunks and process the chunks (assuming the files are too large to read into memory as a single chunk). This makes it very difficult to process strings that overlap the inter-chunk boundaries. Memory mapping allows you to pretend that all the data is in memory at the same time even when that is not actually the case. The necessary manipulations to allow this are performed automatically.
In this lesson, we primarily cover only the details of mmap that apply across both Windows and Unix platforms, and a few Windows-specific features. You should be aware that there are different additional feature sets available for Windows and Unix platforms. The documentation on the module is fairly specific about the implementation differences.
In standard file operations, there is no difference between a memory-mapped file and one that is opened in the regular way—all regular file access methods continue to work, and you can also treat the file content pretty much like a bytearray.
Here's a simple example from the module's documentation to get you started.
>>> with open("/tmp/hello.txt", "wb") as f:
... f.write(b"Hello Python!\n")
...
14
>>> import mmap
>>> with open("/tmp/hello.txt", "r+b") as f:
... mapf = mmap.mmap(f.fileno(), 0)
... print(mapf.readline()) # prints b"Hello Python!\n"
... print(mapf[:5]) # prints b"Hello"
... mapf.tell()
... mapf[6:] = b" world!\n"
... mapf.seek(0)
... print(mapf.readline()) # prints b"Hello world!\n"
... mapf.close()
...
b'Hello Python!\n'
b'Hello'
14
b'Hello world!\n'
>>>
The code above opens a file, then memory maps it. It exercises the readline() method of the mapped file, demonstrating that it works just as with a standard file. It then reads and writes slices of the mapped file (an equally valid way to access the mapped file's content, which does not alter the file pointer). Finally the file pointer is repositioned at the start and the (updated) contents are read in. (The "14" is the return value of the write() function, which always returns the number of bytes written.)
>>> with open("/tmp/hello.txt", "wb") as f:
... f.write(b"Hello Python!\n")
...
14
>>> with open("/tmp/hello.txt", "r+b") as f:
... mapf = mmap.mmap(f.fileno(), 0)
... print(mapf.readline()) # prints b"Hello Python!\n"
... print(mapf[:5]) # prints b"Hello"
... mapf.tell()
... mapf[6:] = b" world!\n"
... mapf.seek(0)
... print(mapf.readline()) # prints b"Hello world!\n"
... # close the map
... mapf.close()
...
b'Hello Python!\n'
b'Hello'
14
b'Hello world!\n'
>>>
As we observed in an earlier lesson, file objects are context managers, albeit of a slightly degenerate kind (because they return themselves as the result of their __enter__() method). The first argument to mmap.mmap is a file number (an internal number used to identify the file to the operating system), which is obtained by calling the file's fileno() method. The call to readline() demonstrates normal file handling, but then you see indexed access to the content, which nevertheless demonstrates that the file pointer is unchanged by such access.
Next you see that the content of the file can also be changed by subscripting, though in this case it is essential that the new content is the same length as the slice being assigned. Finally you observed that the file had been changed by restarting at the beginning.
The difference between using a memory-mapped file and a standard one is that standard files are independently buffered in each process that uses them, meaning that a write to a file from one program is not necessarily immediately written to disk, and will not necessarily be seen immediately by a separate program reading the file using its own buffers.
For calls to mmap.mmap() to be cross-platform compatible they should stick to the following signature:
mmap(fileno, length, access=ACCESS_WRITE, offset=0)
The file number is used simply because this mirrors the interface of the underlying C library (not always the best design decision, but fortunately the file number is easily obtained from an open file's fileno() method). Using a file number of -1 creates an anonymous share (one that cannot be accessed from the filestore).
The call above maps length bytes from the beginning of the file, and returns an mmap object that gives both file- and index-based access to that portion of the file's contents. If length exceeds the current length of the file, the file is extended to the new length before operations continue. If length is zero, the mmap object will map the current length of the file, which in turn sets the maximum valid index that can be used.
The optional access argument can take one of three values, all defined in the mmap module:
| Access Value | Meaning |
|---|---|
| ACCESS_READ | Any attempt to assign to the memory map raises a TypeError exception. |
| ACCESS_WRITE | Assignments to the map affect both the map's content and the underlying file. |
| ACCESS_COPY | Assignments to the memory map change the map's contents but do not update the file on which the map was based (a copy-on-write mapping). |
The offset argument, when present, establishes an offset within the file for the starting position of the memory map. The offset must be a multiple of the constant mmap.ALLOCATIONGRANULARITY (which is typically the size of a virtual memory block, 4096 bytes on many systems).
The real benefit of mmap over other techniques is twofold: first, the file is mapped directly into memory (hence the name). When only one process is using the mapped file, this is a pedestrian application, but remember that modern computers use virtual memory systems. Each process's memory consists of a list of "memory pages." The actual address of the memory page does not matter to the process: the process accesses "virtual memory," and the hardware uses a "memory map" to determine whereabouts in a process's memory a particular page appears.
When a file is memory-mapped, the operating system effectively reserves enough memory to hold the whole file's contents (or that portion of the file that is being mapped) in memory, and then maps that memory into the process's address space. If another process comes along and maps the same file, then exactly the same block of memory is mapped into the second process's address space. This allows the two processes to exchange information extremely rapidly by writing into the shared memory. Since each is writing into the same memory, each can see the other's changes immediately.
| Note | Be careful with large files. Remember that if you memory map a file it gets mapped into your process's virtual address space. If you are using 32-bit Python (either because you are running on a 32-bit system or because your system administrators chose to install a 32-bit Python interpreter on a system built using 64-bit technology), each process has a 4GB upper limit on the size of its address space. Since there are many other claims on a process's memory, it is unlikely you will be able to map all of a file much above 1GB in size in a 32-bit Python environment. |
| Modern Python | mmap objects support the buffer protocol, so you can wrap one in a
memoryview for zero-copy slicing without creating intermediate bytes
objects. In Python 3, all mmap read operations (indexing, slicing, readline(),
etc.) return bytes, not str; encode/decode explicitly when crossing
the bytes/str boundary. mmap objects are also context managers themselves, so you can write
with mmap.mmap(f.fileno(), 0) as mm: and the map will be closed automatically
on exit from the with block, without a separate call to mm.close(). |
The following example code gives you some idea how memory-mapped files might be used for interprocess communication. The program creates a file that will hold data (encoded by the struct module) to be passed between the main program and its subprocesses. The file is split up into "slots," each large enough to hold a byte used to indicate the status of the slot, a 7-character string, and three double-length floating-point numbers. The status starts as EMPTY, and is set to the slot number every time new data becomes available. When there is no more data, the status is set to TERM, which indicates to the subprocess that there is no more work available.
The whole program is given in the listing below. This is a rather larger program than we normally ask you to enter in one go, but by now you should be able to understand what a lot of the code does as you type it in (explanations follow the listing).
"""
mpmmap.py: use memory-mapped file as an interprocess communication area
to support multi-processed applications.
"""
import struct
import mmap
import multiprocessing as mp
import os
import time
import sys
FILENAME = "mappedfile"
SLOTFMT = b"B7s3d"
SLOTSIZE = struct.calcsize(SLOTFMT)
SLOTS = 6 # Number of subprocesses
EMPTY = 255
TERM = 254
def unpackslot(bytes):
"""Return slot data as (slot#, string, float, float, float)."""
return struct.unpack(SLOTFMT, bytes)
def packslot(slot, s, f1, f2, f3):
"""Generate slot string from individual data elements."""
return struct.pack(SLOTFMT, slot, s, f1, f2, f3)
def run(slot):
"""Implements the independent processes that will consume the data."""
offset = SLOTSIZE*slot
print("Process", slot, "running")
sys.stdout.flush()
f = open(FILENAME, "r+b")
mapf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE)
while True:
while mapf[offset] == EMPTY:
time.sleep(0.01)
if mapf[offset] == TERM:
print("Process", slot, "done")
sys.stdout.flush()
mapf.close()
return
x, s, f1, f2, f3 = unpackslot(mapf[offset:offset+SLOTSIZE])
print(slot, ":", s, f1*f2*f3)
sys.stdout.flush()
mapf[offset] = EMPTY
def numbers():
"""Generator: 0.01, 0.02, 0.03, 0.04, 0.05, ..."""
i = 1
while True:
yield i/100.0
i += 1
if __name__ == "__main__":
f = open(FILENAME, "wb")
f.write(SLOTSIZE*SLOTS*b'\0')
f.close()
f = open(FILENAME, "r+b")
mapf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE)
ptbl = []
for slot in range(SLOTS):
offset = slot*SLOTSIZE
mapf[offset] = EMPTY
p = mp.Process(target=run, args=(slot, ))
ptbl.append(p)
print("Starting", p)
p.start()
numseq = numbers()
b = next(numseq)
c = next(numseq)
for i in range(4):
for slot in range(SLOTS):
a, b, c = b, c, next(numseq)
offset = slot*SLOTSIZE
while mapf[offset] != EMPTY:
time.sleep(0.01)
mapf[offset+1:offset+SLOTSIZE] = packslot(slot, b"*******", a, b, c)[1:]
mapf[offset] = slot
for slot in range(SLOTS):
offset = SLOTSIZE*slot
while mapf[offset] != EMPTY:
time.sleep(0.01)
mapf[offset] = TERM
for p in ptbl:
p.join()
mapf.close()
print(f.read())
sys.stdout.flush()
f.close()
os.unlink(FILENAME)
| Modern Python | The original source used the string literal "*******" as the second argument
to packslot(). In Python 3, struct.pack's s format requires a
bytes object, so this would raise a struct.error at runtime. The listing above
corrects it to b"*******". |
There are a couple of utility functions for packing and unpacking the slot data; these are simple calls to standard struct functions that you may remember. Next comes the run() function that will be the meat of the subprocesses. When it is called it is passed the process's slot number, and it uses the computed size of the slot to work out where its particular portion of the data file begins. It then establishes a mapping onto the standard data file and goes into an infinite loop (which will be terminated by the logic it contains). It repeatedly looks at the first byte of its slot, until the EMPTY value it starts with is changed (by the main program). The process sleeps between different looks at the first byte, to avoid using too much CPU. The sleep should be long enough that the computations in the loop take a relatively insignificant time. If the value has changed to TERM, the process closes everything down and terminates. Otherwise it extracts the data from the slot, performs a calculation and prints out the results, and then sets the slot indicator back to EMPTY so the main program will refill the slot.
The run() function is followed by a simple numbers() generator function that separates the task of generating numbers from their use inside the main program. It is an infinite generator that yields numbers starting at 0.01 and increasing by 0.01 each call.
Now, we see the logic of the main program. The program first creates a data file large enough to contain the mapped data for all slots, then maps the file into memory. It then iterates over the slots, setting their status to EMPTY, creates a new process with the current slot number, saves it in a list and starts it. The newly-started process will wait until its slot is switched from EMPTY status before taking any action.
Next the program loops four times over all the slots, filling them with data and only then setting the slot indicator to the slot number. This avoids a potential hazard which might occur if the slot status was set at the same time as the rest of the data: it is just possible that a subprocess might see its status change and start trying to act before the rest of the data is copied in. Yes, this would be a low-probability occurrence, but that does not mean you are at liberty to ignore it.
Once the main loop is over, the program waits for each slot to become EMPTY and sets it to TERM to indicate that the associated process should terminate. Finally, the program waits for all the processes it started to terminate, deletes the file it created at the start of the run, and itself terminates. When you run the program, you should see the following output.
Starting <Process(Process-1, initial)>
Starting <Process(Process-2, initial)>
Starting <Process(Process-3, initial)>
Process 0 running
Process 1 running
Starting <Process(Process-4, initial)>
Process 2 running
Starting <Process(Process-5, initial)>
Starting <Process(Process-6, initial)>
Process 3 running
Process 5 running
5 : b'*******' 0.000336
1 : b'*******' 2.4e-05
2 : b'*******' 6e-05
Process 4 running
4 : b'*******' 0.00021
3 : b'*******' 0.00012
0 : b'*******' 6e-06
5 : b'*******' 0.002184
1 : b'*******' 0.00072
2 : b'*******' 0.00099
4 : b'*******' 0.001716
3 : b'*******' 0.00132
0 : b'*******' 0.000504
5 : b'*******' 0.00684
1 : b'*******' 0.00336
2 : b'*******' 0.00408
4 : b'*******' 0.005814
0 : b'*******' 0.00273
3 : b'*******' 0.004896
5 : b'*******' 0.0156
1 : b'*******' 0.00924
2 : b'*******' 0.010626
4 : b'*******' 0.0138
0 : b'*******' 0.00798
3 : b'*******' 0.012144
Process 5 done
Process 1 done
Process 2 done
Process 4 done
Process 3 done
Process 0 done
b'\xfe*******R\xb8\x1e\x85\xebQ\xc8?\x9a\x99\x99\x99\x99\x99\xc9?\xe1z\x14\xaeG\xe1\xca?\xfe*******\x9a\x99\x99\x99\x99\x99\xc9?\xe1z\x14\xaeG\xe1\xca?)\\\x8f\xc2\xf5(\xcc?\xfe*******\xe1z\x14\xaeG\xe1\xca?)\\\x8f\xc2\xf5(\xcc?q=\n\xd7\xa3p\xcd?\xfe*******)\\\x8f\xc2\xf5(\xcc?q=\n\xd7\xa3p\xcd?\xb8\x1e\x85\xebQ\xb8\xce?\xfe*******q=\n\xd7\xa3p\xcd?\xb8\x1e\x85\xebQ\xb8\xce?\x00\x00\x00\x00\x00\x00\xd0?\xfe*******\xb8\x1e\x85\xebQ\xb8\xce?\x00\x00\x00\x00\x00\x00\xd0?\xa4p=\n\xd7\xa3\xd0?'
| Note | The program above is for demonstration purposes only so you can start to understand the advantages of shared memory. The multiprocessing module actually has other ways to keep processes synchronized, and you should investigate those for production purposes. But if you understand the logic of the code above, you know what mapped files do and how they work, which is a significant piece of learning. |
Memory-mapped files allow you to treat huge tracts of data as though they were large strings, and also allow you to share those large chunks of data between independent processes. They allow you to use inter-process communication.
In the final lesson, we consider some of the differences between small projects and large ones.