Context Managers

The following interactive console session shows how to use files as context managers.

>>> with open("localtextfile") as f:
...     print("f:", f)
...     print("closed:", f.closed)
...     for line in f:
...         print(line, end='')
...
f: <_io.TextIOWrapper name='localtextfile' mode='r' encoding='utf-8'>
closed: False
The open function returns a file object.
This has an __enter__() method that simply
returns self. Its __exit__() method calls
its __close__() method.

>>> f
<_io.TextIOWrapper name='localtextfile' mode='r' encoding='utf-8'>
>>> f.closed
True

>>> f = open("localtextfile", 'r')
>>> 3/0
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ZeroDivisionError: division by zero
>>> f
<_io.TextIOWrapper name='localtextfile' mode='r' encoding='utf-8'>
>>> f.closed
False
>>> with open("localtextfile") as f:
...     3/0
...
Traceback (most recent call last):
  File "<console>", line 2, in <module>
ZeroDivisionError: division by zero
>>> f.closed
True
>>>

You can see that the with statement is a way of controlling the context of execution for the controlled suite. You might wonder why we didn't simply bind the Python file object (the result of opening the file) using an assignment statement. The major purpose of using with in this case is to ensure that, if anything goes wrong inside the context-controlled indented suite, the file will be correctly closed (similarly to the way it might be in the finally clause of a try ... finally statement.

>>> with open("localtextfile") as f:
...     print("f:", f)
...     print("closed:", f.closed)
...     for line in f:
...         print(line, end='')
...
f: <_io.TextIOWrapper name='localtextfile' mode='r' encoding='utf-8'>
closed: False
The open function returns a file object.
This has an __enter__() method that simply
returns self. Its __exit__() method calls
its __close__() method.
>>> f
<_io.TextIOWrapper name='localtextfile' mode='r' encoding='utf-8'>
>>> f.closed
True
>>> f = open("localtextfile", 'r')
>>> 3/0
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ZeroDivisionError: division by zero
>>> f
<_io.TextIOWrapper name='localtextfile' mode='r' encoding='utf-8'>
>>> f.closed
False
>>> f.close()
>>> with open("localtextfile") as f:
...     3/0
...
Traceback (most recent call last):
  File "<console>", line 2, in <module>
ZeroDivisionError: division by zero
>>> f.closed
True
>>>

In the first with example, we saw that f was a standard IO Wrapper object (in point of fact, exactly the same object returned by the open() call, though as you will learn this is not typical of context managers). When the indented suite is run, the file is initially open. Next we see that the file object (still available after the with) is closed when the with statement terminates, even though no explicit action was taken to close it. You will understand this after the next interactive session.

Next you reminded yourself that when an exception occurs during regular file processing the file remains open unless explicit action is taken to close it. When the exception occurs inside the suite of the with statement, however, once again we see that the file is magically closed without any explicit action being taken. The magic is quite easily explained (as usual in Python, where a simple, easy-to-understand style is preferred) by two file magic methods we have not previously discussed.

The with statement has rules for interacting with the object it is given as a context manager. It processes with expr by evaluating the expression and saving the resulting context manager object. The context manager's __enter__() method is then called, and if the as name clause is included, the result of the method call is bound to the given name. Without the as name clause, the result of the __enter__() method is not available. The indented suite is then executed.

As the execution of the suite progresses, an exception may be raised. If so, the execution of the suite ends and the context manager's __exit__() method is called with three arguments together referencing detailed information about the causes and location of the exception.

If no exception is raised and the suite terminates normally (that is, by "dropping off the end"), the context manager's __exit__() method is called with three None arguments.

There are other ways that the with suite can be exited, all fairly normal—how many ways can you think of? In those circumstances, the context manager's __exit__() method is called with three None arguments, and then the normal exit is taken.

The reason for the name "context manager" is that the indented suite in a with statement is surrounded by calls to the manager's __enter__() and __exit__() methods, which can therefore provide some context to the execution of the suite. Note carefully that the __exit__() method is always called—even when the suite raises an exception.

As is so often the case in Python, it is quite easy to write a class that demonstrates exactly how the context manager objects work with the interpreter as it executes the with statement. Since there are two alternative strategies for handling the raising of an exception in the indented suite, an __init__() method can record in an instance variable which strategy the creator (the code calling the class) chooses. If no exception is raised, this will make no difference.

Besides the very simple __init__() outlined (which is not itself a part of the context manager protocol), you just need the __enter__() and __exit__() methods. If you are only interested in finding out how the with statement works, these methods don't have to do a lot except print out useful information. Try this out in an interactive interpreter session:

>>> class ctx_mgr:
...     def __init__(self, raising=True):
...         print("Created new context manager object", id(self))
...         self.raising = raising
...     def __enter__(self):
...         print("__enter__ called")
...         cm = object()
...         print("__enter__ returning object id:", id(cm))
...         return cm
...     def __exit__(self, exc_type, exc_val, exc_tb):
...         print("__exit__ called")
...         if exc_type:
...             print("An exception occurred")
...             if self.raising:
...                 print("Re-raising exception")
...             return not self.raising
...
>>> with ctx_mgr(raising=True) as cm:
...     print("cm ID:", id(cm))
...
Created new context manager object 4300642640
__enter__ called
__enter__ returning object id: 4300469808
cm ID: 4300469808
__exit__ called
>>> with ctx_mgr(raising=False):
...     3/0
...
Created new context manager object 4300642768
__enter__ called
__enter__ returning object id: 4300469904
__exit__ called
An exception occurred
>>> with ctx_mgr(raising=True) as cm:
...     3/0
...
Created new context manager object 4300642640
__enter__ called
__enter__ returning object id: 4300469744
__exit__ called
An exception occurred
Re-raising exception
Traceback (most recent call last):
  File "<console>", line 2, in <module>
ZeroDivisionError: division by zero
>>>

Your context manager object does not get too much of a workout in the above session, but as always you should feel free to try other things out in the session. You are unlikely to cause a fire or bring the server to a halt by being a little adventurous: you are now a seasoned Python programmer, and can (we hope) be trusted to flex your muscles a little. Let's just review the output from that session:

>>> with ctx_mgr(raising=True) as cm:
...     print("cm ID:", id(cm))
...
Created new context manager object 4300642640
__enter__ called
__enter__ returning object id: 4300469808
cm ID: 4300469808
__exit__ called
>>> with ctx_mgr(raising=False):
...     3/0
...
Created new context manager object 4300642768
__enter__ called
__enter__ returning object id: 4300469904
__exit__ called
An exception occurred
>>> with ctx_mgr(raising=True) as cm:
...     3/0
...
Created new context manager object 4300642640
__enter__ called
__enter__ returning object id: 4300469744
__exit__ called
An exception occurred
Re-raising exception
Traceback (most recent call last):
  File "<console>", line 2, in <module>
ZeroDivisionError: division by zero
>>> 

In the first example, you can see that this context manager returns an entirely different object as the result of its __enter__() method. The print statement which forms the indented suite demonstrates that the name cm is bound in the with statement to the result of the context manager's __enter__() method and not the context manager itself. (The file open() example earlier is atypical, as a file object's __enter__() method returns self). No exception is raised by the indented suite, and so the __exit__() method simply reports it has been called.

The second example raises an exception in the context of a context manager that was created not to re-raise the exception. So it does report the fact that an exception was raised, but then it again terminates normally (because its self.raising attribute has the value False, and so the method returns True).

The third example is exactly the same as the second except that the instance is created with its raising attribute True. This means that once the instance has reported the exception it announces its intention to re-raise it, and does so by returning False.

Although you have just seen it is very easy to write a simple context manager class, it can be even easier to use context managers if you use the contextlib module. This contains a decorator called contextmanager that you can use to create context managers really simply. There is no need to declare a class with __enter__() and __exit__() methods.

You must apply the contextlib.contextmanager decorator to a generator function that contains precisely one yield expression. When the decorated function is used in a with statement, the (decorated) generator's next method is called for the first time, so the function body runs right up to the yield. The yielded value is returned as the result of the context manager's __enter__() method, and the indented suite of the with statement then runs.

If the indented suite raises an exception, it appears inside the context manager as an exception raised by the yield. Your context manager can choose to handle the exception (by processing the yield as part of the indented suite of a try statement) or not (in which case the exception must be re-raised after logging or other actions if the surrounding logic is to see it). So your context manager can trap exceptions raised by the indented suite and suppress them simply by choosing not to re-raise them.

>>> from contextlib import contextmanager
>>> @contextmanager
... def ctx_man(raising=False):
...     try:
...         cm = object()
...         print("Context manager returns:", id(cm))
...         yield cm
...         print("With concluded normally")
...     except Exception as e:
...         print("Exception", e, "raised")
...         if raising:
...             print("Re-raising exception")
...             raise
...
>>> with ctx_man() as cm:
...     print("cm from __enter__():", id(cm))
...
Context manager returns: 4300470512
cm from __enter__(): 4300470512
With concluded normally
>>> with ctx_man(False) as cm:
...     3/0
...
Context manager returns: 4300801264
Exception division by zero raised
>>> with ctx_man(True) as cm:
...     3/0
...
Context manager returns: 4300801280
Exception division by zero raised
Re-raising exception
Traceback (most recent call last):
  File "<console>", line 2, in <module>
ZeroDivisionError: division by zero
>>>

This interactive session shows that it is possible to create equivalent context managers using this approach. The same parameterization of the functionality is provided (so you can say when creating the context manager whether or not it should re-raise exceptions). contextlib.contextmanager provides a nice compromise between writing a full context manager and using older, less well-controlled methods (such as try ... except ... finally) of controlling the execution context. You will find that the other members of the contextlib library can also be useful in creating and supporting context managers.

The statement:

with expr1 as name1, expr2 as name2:
    [indented suite]

is equivalent to:

with expr1 as name1:
    with expr2 as name2:
        [indented suite] 

This shows that the expr1 context wraps the name2 context. If an exception occurs in the indented suite, it will present as a call to expr2.__exit__() with the necessary exception-related arguments. As always, the __exit__() method has the choice of returning True (which suppresses the exception, resulting in a call to expr1.__exit__() with three None arguments) or False, in which case the exception is automatically re-raised and expr1.__exit__() is called with the traceback arguments. It also has the choice of returning True to suppress the exception or False to re-raise it a second time.

The multi-context form of the with statement is a simple syntactic convenience; no new functionality is introduced, but it does reduce the indentation level required for the indented suite. This enhances readability without compromising simplicity.

This section will briefly introduce the decimal module, to whose documentation you are referred for further information. The context in which decimal arithmetic is performed has several elements:

The decimal module has been carefully written to ensure that each thread can have an independent decimal context (because it would be disastrous if one thread could affect another by making changes to a shared context).

Most of the attributes of the context are fairly esoteric stuff that you really don't need to alter. For many applications, you can just use the default context. While prec and rounding are fairly frequently adjusted, capitals and clamp are rarely touched.

Certain things can happen during arithmetic operations that cause the results to be imprecise or otherwise misleading, and the operations raise signals to indicate this. The decimal code responds to these signals by setting flags in the arithmetic context. If the trap corresponding to a signal is set, an exception is raised after the flag is set. The following flags are defined:

You can access the default decimal context using the getcontext() function from the decimal module. Contexts know how to present themselves in a fairly readable form, and you can modify the context just by assigning to its various attributes. You can also create copies of contexts and switch between them. Finally, of course, you can create instances of the decimal.Context class, providing the non-default required attributes as keyword arguments. Note that if you modify decimal.DefaultContext, it will change the default values used to create future contexts. This is useful for setting up defaults before creating multiple threads, but should not be used casually in non-threaded programs.

>>> from decimal import *
>>> myothercontext = Context(prec=60, rounding=ROUND_HALF_DOWN)
>>> setcontext(myothercontext)
>>> getcontext()
Context(prec=60, rounding=ROUND_HALF_DOWN, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[], traps=[InvalidOperation, DivisionByZero, Overflow])
>>> Decimal(1) / Decimal(7)
Decimal('0.142857142857142857142857142857142857142857142857142857142857')
>>> ExtendedContext
Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[], traps=[])
>>> setcontext(ExtendedContext)
>>> getcontext()
Context(prec=9, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[], traps=[])
>>> Decimal(1) / Decimal(7)
Decimal('0.142857143')
>>> Decimal(42) / Decimal(0)
Decimal('Infinity')
>>> setcontext(BasicContext)
>>> getcontext()
Context(prec=9, rounding=ROUND_HALF_UP, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[], traps=[Clamped, InvalidOperation, DivisionByZero, Overflow, Underflow])
>>> Decimal(42) / Decimal(0)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
decimal.DivisionByZero: [<class 'decimal.DivisionByZero'>]
>>> with localcontext() as ctx:
...     ctx.prec = 42
...     s = Decimal(1) / Decimal(7)
...     print(s)
...
0.142857142857142857142857142857142857142857
>>> s = +s
>>> print(s)
0.142857143
>>>

You can see that the decimal module provides a number of "ready-made" contexts, which can easily be modified by attribute assignment. It is easy to make changes to the current context's attributes, but these changes are permanent. The decimal.localcontext() function returns a context manager that sets the active thread's current context to the context provided as an argument or (in the case above where no argument is provided) the current context. The with statement provides a natural way to perform such localised changes. Note that the unary plus sign in "+s" does actually perform a conversion, because it is an arithmetic operation whose result must be conditioned by the (now restored) original context.