Functions and Other Objects

You already know that the arguments passed to a function call must match the parameter specifications in the function's definition. Any mismatch can be taken up in the definition, where a parameter of the form *name associates unmatched positional arguments with a tuple and one of the form **name associates the names and values of unmatched keyword arguments with the keys and values of a dict.

You have also seen that a positional argument may be associated with a keyword parameter and vice versa. You currently have no way, however, of requiring that specific arguments be presented as keyword arguments. You can specify such a requirement by inserting an asterisk on its own as a parameter specification: any parameters that follow the star (other than the *args and **kwargs arguments, if present) must be provided as keyword arguments on the call.

Investigating this phenomenon is quite easy in the interactive console:

>>> def f(a, *, b, c=2):
...     print("A", a, "B", b, "C", c)
...
>>> f(1, 2)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: f() takes 1 positional argument but 2 were given
>>> f(1, c=3)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: f() missing 1 required keyword-only argument: 'b'
>>> f(1, b=2, c=3)
A 1 B 2 C 3
>>> f(1, b=2)
A 1 B 2 C 2
>>>

Attempting to provide a positional argument for b raises an exception because of the wrong number of positional arguments. The second test is the most telling one, as that explains the requirement for a keyword argument b.

We mention this feature because you may come across some code that uses it, and wonder what on Earth is going on. In Python 3, functions and their parameters can be annotated. A parameter is annotated by following its name with a colon and an expression, and a function is annotated by following its parameter list with "->" and an expression.

The language definition specifically avoids associating any kind of meaning to annotations. The stated intention is that if people find ways of using annotations that find general acceptance, specific semantics may be added to the interpreter at a later date; for now you can access them through the __annotations__ attribute of the function object. This is a dict in which each of the function's annotated parameters is stored against the parameter name as key. The function's return-value annotation, if present, is stored against key "return" which, being a Python keyword, cannot be the name of any parameter.

Just to show you how annotations appear in practice, we'll create an annotated function in an interactive interpreter session:

>>> def f(i: int, x:float=1.2) -> str:
...     return str(i*x)
...
>>> f.__annotations__
{'i': <class 'int'>, 'x': <class 'float'>, 'return': <class 'str'>}
>>>

Although there is no restriction on the expressions used as annotations, in practice most people see them as being useful for making assertions about the types of arguments and the function's return value. At present, nothing in the interpreter uses the annotation information at all. You would need to specifically action such uses with additional code if you don't want your annotation data to be ignored. It is likely that, as the feature becomes better known, frameworks will emerge to make use of different types of annotation data.

Although you have seen functions with function definitions inside them, we have not yet formalized the rules for looking up names within those functions. You already know the general rule for (unqualified) name resolution in Python: first look in the local namespace, then look in the (module) global namespace, and finally look in the built-in namespace.

The only additional complexity that nested functions introduce is that the local namespace is actually enhanced by names from surrounding functions (unless they are redefined in the contained function). Remember that a name is only considered local to a function if the name is bound in that function. So when a function is defined inside a function, a name can be a reference from the function call's namespace, or a reference to the namespace of the function call during which the inner function was defined, and this regress can go on until the outermost function call is encountered.

Understanding Python as you do now, you will see that it requires some trickery to allow a function to return another function defined inside the first function. That is because the returned function may contain references to values defined in the local namespace of the (now completed) function call that returned it! We do not need to examine the mechanism the interpreter uses to resolve this issue, but since it is a genuine feature of the language, it is one that every implementation has to solve in its own way.

Python 3 also introduces a second declaration statement, the nonlocal statement. This can be used to force an apparently local variable to instead be treated as though it came from the containing scope where it is already defined. This is slightly different from the global statement, in that the interpreter searches the containing scopes (function namespaces) to locate the one that already contains a definition of the name(s) listed after the nonlocal keyword. (The global statement always and unambiguously places the name in the module global namespace, whether it has been defined there or not).

a, b, c = "Module a", "Module b", "Module c"

def outer():

    def inner():
        nonlocal b
        global c
        a = "Inner a"
        b = "Inner b"
        c = "Inner c"
        print("inner", a, b, c)
    a = "Outer a"
    b = "Outer b"
    c = "Outer c"
    print("outer", a, b, c)
    inner()
    print("outer", a, b, c)

print("module", a, b, c)
outer()
print("module", a, b, c)

When you run this program, you should see the following output:

module Module a Module b Module c
outer Outer a Outer b Outer c
inner Inner a Inner b Inner c
outer Outer a Inner b Outer c
module Module a Module b Inner c

Just as the global statement allows the inner() function to refer to the module-global "c" name, so the nonlocal statement allows it to use the name "b" to refer to the outer function's "b." After the call to outer(), only the module-global "c" has changed, because only "c" was declared as global in the inner() function.

You learned about the functools module when we were discussing decorators earlier in this course. The module contains another useful function that allows you to take a function and define another function that is the same as the first function, but with fixed values for some arguments. The signature of the function is:

functools.partial(f[, *args[, **kw]]) returns a function g which is the same as f with the positional arguments args giving values for the initial positional arguments and the keyword arguments kw setting default values for the given named arguments. The intention is to allow you to fix some arguments of a function, leaving you with a function-like object to which the remaining arguments can be applied at your convenience. The resulting partial function objects cannot be called with quite the same abandon as real functions, however, since certain counterintuitive behaviors can occur.

>>> import functools
>>> def fp(a, b, c="summat", d="nowt"):
...     print("a b c d", a, b, c, d)
...
>>> fp("ayeup", "geddaht")
a b c d ayeup geddaht summat nowt
>>> fp1 = functools.partial(fp, 1, b=2)
>>> fp1()
a b c d 1 2 summat nowt
>>> fp1("ayeup", "geddaht")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: fp() got multiple values for argument 'b'
>>> fp1(c="ayeup", d="geddaht")
a b c d 1 2 ayeup geddaht
>>> fp2 = functools.partial(fp, 1, c="two")
>>> fp2("ayeup", "geddaht")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: fp() got multiple values for argument 'c'
>>> fp2
<functools.partial object at 0x...>
>>> fp2("ayeup", c="geddaht")
a b c d 1 ayeup geddaht nowt
>>> 

fp1 is ostensibly a function taking two keyword arguments (its two positionals having been applied in the creation of the partial). The expression fp1("ayeup", "geddaht"), however, makes it plain that the first positional argument is being provided to match up with fp()'s b argument, and that when the same keyword argument is later applied a duplication is detected.

The simplest solution to this dilemma is to always replace positional parameters with positional arguments and replace keyword parameters with keyword arguments when using partial(). This rule also has to be extended to the calls of the partial functions. The first call to fp2() shows that although the partial function has one positional and one keyword parameter, it is not possible to match a positional argument to the keyword parameter d as would be possible with a real function. So remember to treat partials carefully when you encounter them.

One very nice little example from the documentation shows how a default can be applied to a required argument. The int() built-in type can be called with a number or a string as an argument. When called with a string, a second argument base can be provided which determines the number system used to interpret the string. Providing that argument creates a partial object that will convert base-2 strings to integers.

>>> from functools import partial
>>> basetwo = partial(int, base=2)
>>> basetwo.__doc__ = "Convert base-2 string to int."

>>> basetwo("1111")
15
>>> basetwo("1001010")
74
>>>

Beware of the differences between partial objects and true functions, and respect them. While partials can be very helpful, they are only a shorthand and not a complete replacement.

This simplified treatment expresses the way that the interpreter works to a first approximation. As always, we try to be as precise as possible without necessarily providing exact detail of what goes on in the more complex corner cases.

When you see the expression s = x + y in a program, the interpreter has to decide how to evaluate it. It does so by looking for specific methods on the x and y objects. For addition, the relevant methods are __add__() and __radd__(). First the interpreter looks for an x.__add__() method (special/magic methods are always looked up on the class and its parents, never on the instance). If such a method exists, x.__add__(y) is called. If this call returns a result, that becomes the value of the expression.

The method may, however, choose to indicate that it is unable to compute a response (for example because y is incompatible) by returning a special built-in value NotImplemented. In that case, the interpreter next looks for a y.__radd__() method ("radd" is intended to be a mnemonic for "reflected add"). If such a method exists, y.__radd__(x) is called and, unless it returns NotImplemented, the return value becomes the value of the expression. There is one exception to this rule: if the two values are of the same type, the __radd__() method is not called. The assumption is that if a and b are of the same type and you can't (say) add a to b, then you shouldn't be able to add b to a either, and there is no point trying.

Try it out in an interactive session:

>>> class mine:
...     def __add__(self, other):
...         print("__add__({}, {})".format(self, other))
...         return NotImplemented
...     def __radd__(self, other):
...         print("__radd__({}, {})".format(self, other))
...         return 42
...     def __repr__(self):
...         return "[Mine {}]".format(id(self))
...
>>> class yours:
...     def __add__(self, other):
...         print("__add__({}, {})".format(self, other))
...         return NotImplemented
...     def __radd__(self, other):
...         print("__radd__({}, {})".format(self, other))
...         return NotImplemented
...     def __repr__(self):
...         return "[Yours {}]".format(id(self))
...
>>> m1 = mine()
>>> m2 = mine()
>>> m1, m2
([Mine 0x...], [Mine 0x...])
>>> y1 = yours()
>>> y2 = yours()
>>> y1, y2
([Yours 0x...], [Yours 0x...])
>>>
>>> m1+m2
__add__([Mine 0x...], [Mine 0x...])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'mine' and 'mine'
>>> y1+y2
__add__([Yours 0x...], [Yours 0x...])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'yours' and 'yours'
>>> m1+y2
__add__([Mine 0x...], [Yours 0x...])
__radd__([Yours 0x...], [Mine 0x...])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'mine' and 'yours'
>>> y1+m2
__add__([Yours 0x...], [Mine 0x...])
__radd__([Mine 0x...], [Yours 0x...])
42
>>>

As you can see, since both classes' __add__() methods return NotImplemented, attempting to add a mine to a mine or a yours to a yours will fail, raising an exception. The third case also raises an exception because the __radd__() method of the yours right-hand operand also returns the value NotImplemented. The final test works, however, because mine.__radd__() actually returns a value (albeit one that does not depend on its operands at all).

There is another series of special methods associated with the augmented arithmetic operations (that is, "+=", "-=" and so on). When you see a statement such as x += y (that is to say, any statement using augmented assignment operations) in a program, the interpreter evaluates it by looking for a specific method on the x object. For addition, the relevant method is __iadd__(). If this method does not exist, the statement is treated as though it read x = x+y. If the x.__iadd__() method is found, however, it is called with y as an argument, and the result (which may be a modified version of the existing object or a completely new object, entirely at the option of the implementor of the object in question) is bound to x. Following are the methods corresponding to the basic Python arithmetic operations.