Advanced Objects: Special Methods

At first glance, the __new__() method seems similar to the __init__() method, but it is actually quite different. You will remember that you instantiate a class (that is, create a new instance of that class) by calling the class. The __init__() method returns nothing—it merely initializes what __new__() has created. The __new__() method, on the other hand, returns the object that will become the return value of the instantiation call.

Like __init__(), __new__() receives the arguments that the caller passes when calling the class. Unlike __init__(), __new__() receives a first argument that is the class to be created rather than the newly-created instance.

This is important: the default __new__() method (inherited from the object type) can be used to create immutable object instances. Most of the time you use this when extending immutable built-in types like numbers and strings, since it would not be possible to change them in the __init__() method. In our example, we'll create ustr, an extension of the basic str type that returns a string object that always has upper-case versions of any letters it may contain. Create newmagic.py as shown:

"""
Python classes with magic methods
"""

class ustr(str):
    "An upper case string object."

    def __new__(cls, arg):
        arg = str(arg)
        return str.__new__(cls, arg.upper())

Before we continue, let's look closely at this class.

class ustr(str):
    "An upper case string object."
    def __new__(cls, arg):
        arg = str(arg)
        return str.__new__(cls, arg.upper())

This example defines the class ustr as a subclass of the built-in str type used to represent Unicode strings. Because Python's type names are lower-case, we break from the tradition of naming a class in MixedCase, and use a lower-case class name. The class defines a __new__() method that accepts cls and arg arguments.

The cls parameter is the actual class that was called—this is different from the self argument passed to other methods, which represents the instantiated object. Like self, the cls argument is provided automatically by the interpreter. The whole purpose of the __new__() method is to create and return the new object: this method is directly responsible for instantiation!

The arg parameter is the argument provided to the class when it is called. The value of arg is converted into a string, and the last statement returns a new string. It does so, however, by calling the built-in string type's __new__() method explicitly, asking it (with the first argument) to return an object of the correct type. The second argument to str.__new__() provides the value for the string, and upper case is guaranteed by calling its .upper() method.

The call to str.__new__() is analogous to an explicit call on a class method giving an instance as the first argument. The call to str.__new__() returns a ustr object, because the first argument to __new__() specifies the return type required. Test the class at the interactive console:

>>> from newmagic import *
>>> s = ustr("Steve Holden")
>>> s
'STEVE HOLDEN'
>>> type(s)
<class 'newmagic.ustr'>
>>> s.lower()
'steve holden'
>>> len(s)
12
>>> s.size = 12
>>> s.size
12
>>> ss = "A regular string"
>>> ss.size = 16
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'size' and no __dict__ for setting new attributes
>>>

You can see that instances of the ustr class behave almost the same as instances of str, except that they have to be created with an explicit call to the class. There is also the difference that you can set new attributes on ustr instances, but you cannot do that with the built-in type.

The built-in str() seems to be a function, but strictly speaking, it is actually a built-in type that can be used like a function. When you call it with a single argument, it tries to return a string representation of the argument by calling the argument's __str__() method. The print() built-in function does the same thing.

The return value of __str__() must be a string object. In our example below, the Person class is a normal class and the NamedPerson class provides a more attractive print statement. Create strmagic.py as shown:

"""
Demonstrate string representations using inheritance
"""

class Person:
    "Represents a person"

    def __init__(self, name):
        self.name = name

class NamedPerson(Person):
    "Represents a person using their name"

    def __str__(self):
        return self.name

The difference between the two classes is that NamedPerson has an __str__() method, which Person does not. You can see the difference quite easily in an interactive interpreter session.

>>> from strmagic import *
>>> p1 = Person("Danny Greenfeld")
>>> p1
<strmagic.Person object at 0x...>
>>> print(p1)
<strmagic.Person object at 0x...>
>>> p2 = NamedPerson("Danny Greenfeld")
>>> p2
<strmagic.NamedPerson object at 0x...>
>>> print(p2)
Danny Greenfeld
>>>

The string returned by __str__() is supposed to be an "informal" representation of the object, which can be used to convey its principal characteristics without necessarily allowing you to reproduce the object exactly. For the latter purpose, Python expects your objects to provide another magic method, __repr__().

The __repr__() method of object o is called by repr(o). The built-in function repr() is supposed to represent the "official" string representation of an object. Ideally, this representation should look like a valid Python expression which, when evaluated, produces the object being represented. Its primary use is in debugging or logging, and is best not revealed to users. The information should be as rich as possible.

The str() representation of a container object such as a list or a tuple also represents the contained objects using their repr() representation. Containers themselves generally use the same representation for both str() and repr(), and this is the easiest way to ensure that their representations are meaningful.

To determine a little more about the relationship between the two representational methods, we'll create four different classes that have different combinations of those methods. You can then see how they interact with the interactive interpreter and the print() function. (Don't forget, if you have other questions about this, the interactive interpreter is the best way to answer those questions). Create reprmagic.py as shown:

"""
Demonstrate differences between __str__() and __repr__().
"""

class neither:
    pass

class stronly:

    def __str__(self):
        return "STR"

class repronly:

    def __repr__(self):
        return "REPR"

class both(stronly, repronly):
    pass

class Person:

    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __str__(self):
        return self.name

    def __repr__(self):
        return "Person({0.name!r}, {0.age!r})".format(self)

The neither class simply inherits all its behavior from Python's fundamental object. The stronly class, as its name implies, only implements __str__(), while the repronly class only implements __repr__(). The both class uses multiple inheritance to implement both methods. Finally, the Person class represents its instances as a string by using the instance's name and provides a full representation that could actually be pasted into a Python program as code. Note that it uses the !r format effector to include the formal representations of the instance's name and age. This avoids any tricky problems of representing strings with characters inside them that require escapes and so on.

>>> from reprmagic import *
>>> o1 = neither()
>>> print(str(o1), repr(o1))
<reprmagic.neither object at 0x...> <reprmagic.neither object at 0x...>
>>> o2 = stronly()
>>> print(str(o2), repr(o2))
STR <reprmagic.stronly object at 0x...>
>>> o3 = repronly()
>>> print(str(o3), repr(o3))
REPR REPR
>>> o4 = both()
>>> print(str(o4), repr(o4))
STR REPR
>>> o1
<reprmagic.neither object at 0x...>
>>> o2
<reprmagic.stronly object at 0x...>
>>> o3
REPR
>>> o4
REPR
>>> steve = Person("Steve Holden", 21)
>>> print(str(steve), repr(steve))
Steve Holden Person('Steve Holden', 21)
>>> tim = Person('Tim O\'Reilly', 55)
>>> tim
Person("Tim O'Reilly", 55)
>>>

In the lines where we asked the interactive interpreter directly for the objects o1 through o4, it presented the repr() of the objects. Remember that this behavior is specific to the interpreter's interactive mode: if you write an expression on its own in a Python module that is run as a main program, the interpreter simply calculates the value of the expression. Also note, from the example of the repronly() object bound to o3, that if an object has a __repr__() method but no __str__() method, the __repr__() method is used for both purposes.

Normally, when you set an attribute on an object, the name is used as a key and the value is stored in the object's __dict__, a special attribute used specifically to store instance variables. This method is called each and every time an attribute is set. In the code sample below, we use this feature to print a message each time an attribute is set. Create attrmagic.py as shown:

"""
Demonstrate magic methods for attribute access.
"""

class AttrMixin:
    "Displays a message when an instance's attributes are set."

    def __setattr__(self, key, value):
        print("ATTR: setting attribute {0!r} to {1!r}".format(key, value))
        self.__dict__[key] = value

class Person(AttrMixin):
    "Represents a person"

    def __init__(self, name):
        self.name = name

Here the Person class inherits its attribute setting behavior from the AttrMixin class. The AttrMixin.__setattr__() method does here make the definite assumption that the classes it will be mixed in with are storing all attributes using the standard instance __dict__ mechanism. When you start to see the layers of behavior that Python allows you to add to the process of attribute assignment, you will realize that this may be a dangerous assumption, but certainly it holds for the Person class.

As usual, you can verify the actions of this code in the interactive interpreter. Note that the setting of an attribute is reported whether it is set inside an object method or in external code.

Attribute retrieval works a little differently from attribute setting. When you try to access an attribute of some instance o, the interpreter looks in o.__dict__; if the attribute is not found there, it looks in the instance's class, then in that class's superclass, and so on. Only if the attribute is not found does the interpreter then call the instance's __getattr__() method with the name of the attribute. It is conventional for __getattr__() to raise the AttributeError exception when the attribute name provided is unacceptable for some reason.

"""
Demonstrate magic methods for attribute access.
"""

class AttrMixin:
    "Displays a message when an insobjectance's attributes are retrieved or set."

    def __setattr__(self, key, value):
        print("ATTR: setting attribute {0!r} to {1!r}".format(key, value))
        self.__dict__[key] = value

    def __getattr__(self, key):
        print("ATTR: getting attribute {0!r}".format(key))
        self.__setattr__(key, "No value")
        return "No value"

class Person(AttrMixin):
    "Represents a person"

    def __init__(self, name):
        self.name = name

Start an entirely new interactive console session in which to test the updated module—remember, the code of a module is executed only on the first import. Trying to import the updated module will therefore fail, and you will not see the expected behaviors.

>>> from attrmagic import *
>>> steve = Person("Steve Holden")
ATTR: setting attribute 'name' to 'Steve Holden'
>>> steve.newattr
ATTR: getting attribute 'newattr'
ATTR: setting attribute 'newattr' to 'No value'
'No value'
>>> steve.newattr
'No value'
>>> steve.name
'Steve Holden'
>>>

Observe that while the first access to the newattr attribute results in a call to __getattr__(), the second one does not. This is because the first call actually sets a value in the object's __dict__ and so the second attempt finds the attribute using the standard methods.

This method is called whenever an attribute is deleted from an object. Again, we'll publish a message to show what can be done with this method.

"""
Demonstrate magic methods for attribute access.
"""

class AttrMixin:
    "Displays a message when an object's attributes are retrieved, deleted or set."

    def __setattr__(self, key, value):
        print("ATTR: setting attribute {0!r} to {1!r}".format(key, value))
        self.__dict__[key] = value

    def __getattr__(self, key):
        print("ATTR: getting attribute {0!r}".format(key))
        self.__setattr__(key, "No value")
        return "No value"

    def __delattr__(self, key):
        print("ATTR: Deleting key {0!r}".format(key))
        object.__delattr__(self, key)

class Person(AttrMixin):
    "Represents a person"

    def __init__(self, name):
        self.name = name

Note that your version of __delattr__ simply delegates the deletion to Python's standard object behavior. This allows you to ignore whatever complexities may be required in deletion. Again, test your modifications:

>>> from attrmagic import *
>>> student = Person("your name")
ATTR: setting attribute 'name' to 'your name'
>>> student.age
ATTR: getting attribute 'age'
ATTR: setting attribute 'age' to 'No value'
'No value'
>>> student.age = 21
ATTR: setting attribute 'age' to 21
>>> student.age
21
>>> del student.age
ATTR: Deleting key 'age'
>>> student.age
ATTR: getting attribute 'age'
ATTR: setting attribute 'age' to 'No value'
'No value'
>>> del student.__delattr__
ATTR: Deleting key '__delattr__'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    del student.__delattr__
        ^^^^^^^^^^^^^^^^^^^
  File "attrmagic.py", line 18, in __delattr__
    object.__delattr__(self, key)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
AttributeError: 'Person' object has no attribute '__delattr__'
>>>

You can see that the attempt to delete __delattr__ fails, because the special attributes are not discovered in the same way as the regular ones. This avoids the deletion of standard behaviors that are required to be true of all objects.