Uses of Introspection

The word "introspection" means "looking inside." Introspective people are ones who think about themselves, usually to increase self-understanding. In Python, introspection is a way that your programs can learn about the environment in which they operate and the properties of the modules they import.

You have already learned about several of Python's introspection mechanisms. The built-in dir() function, for example, attempts to return (to quote from the documentation) "an interesting set of names"—meaning the names of attributes accessible from the object passed as an argument. If no argument is passed, it returns the attributes found in the current local namespace.

dir() in Python 3.x has a hook that looks for a __dir__() method on its argument. If such a method is present, it is called and dir() returns what the method returns. This allows you to determine what users see about your object, and this can be useful if you are using "virtual" attributes (that is, if your objects handle access to methods that do not appear in the class's __dict__). If no __dir__() method is found, dir() uses a standard mechanism to compose its result after examining its argument.

>>> class Something:
...     pass
...
>>> s = Something()
>>> s
<__main__.Something object at 0x...>
>>> repr(s)[1:-1].split()[0].split(".")[1] # WRONG!
'Something'
>>> s.__class__.__name__ # RIGHT (AND SO MUCH EASIER)
'Something'
>>> repr(4)[1:-1].split()[0].split(".")[1] # Fail
Traceback (most recent call last):
  File "<console>", line 1, in <module>
IndexError: list index out of range
>>> 4.__class__.__name__
  File "<string>", line 1
SyntaxError: invalid decimal literal
>>> (4).__class__.__name__ # SUCCEED
'int'
>>> str(type(4))[1:-1].split()[1][1:-1] # Way too complex
'int'
>>> str(type(s))[1:-1].split()[1][1:-1] # And only give same result for built-ins (see s above)
'__main__.Something'

If you took earlier courses in this Certificate Series (or otherwise possibly from private study) you've encountered the getattr(obj), setattr(obj), and delattr(obj) functions, and learned that they result in a call to their argument obj's __getattr__(), __setattr__(), and __delattr__() methods. There is also the hasattr() predicate, which can be used to determine whether or not a given attribute is present in an object. There is, however, no corresponding __hasattr__() method. You might wonder what hasattr() does to find out what value to return, and the answer to that question is complex enough to have received the attention of some of the best minds in Python.

Without going too deeply into the internals, it is fairly easy for you to determine whether or not __getattr__() gets called by hasattr() under at least some circumstances. You simply write a class whose instances report calls of their __getattr__() method, and then call hasattr() on an instance:

>>> class X:
...     def __getattr__(self, name):
...         print("getattr", name)
...         return 0
...
>>> x = X()
>>> hasattr(x, "thing")
getattr thing
True
>>>

hasattr(obj, "__call__") can be used to tell you whether or not obj can be called like a function. Older versions of Python provide a callable() built-in function, which should have been removed in Python 3.0 because the given test is now all that is required— everything callable has a __call__ attribute. Its deletion was omitted in error for the 3.0 release, with the result that callable is available for that release. It was then removed from 3.1 (the version in use when this course was being written), but has returned in 3.2 because the above test turns out not to be quite as specific as the version that can be written in C with full access to the object structures. Being able to determine the presence or absence of a particular attribute is occasionally useful in other contexts.

Of course you can implement whole "virtual namespaces" within your own objects by using getattr() and setattr(), but remember that these functions can also be used (assuming you can gain access to the required namespaces) to modify your current environment. Understand that doing so in this way is not recommended except in rather extreme cases, because it results in "magical" changes—changes whose origin is difficult or impossible to discern by reading the program code:

>>> import sys
>>> __name__
'__main__'
>>> module = sys.modules[__name__]
>>> a
Traceback (most recent call last):
  File "<console>", line 1, in <module>
NameError: name 'a' is not defined
>>> setattr(module, "a", 42)
>>> a
42
>>>

Before the setattr() call, there was no "a" defined in the module's namespace. Since all imported modules are available under their natural names from sys.modules, you can access the current module's namespace by looking it up.

If it were possible to subclass the module object to change its attribute access methods, we could be faced with some extremely hard-to-understand code! Fortunately this is not something you need to worry about in practice. Most of the code you will encounter does not use such tricks (indeed, the Django framework mentioned earlier had a period in its development devoted to "magic removal" to make the code easier for Python programmers and beginners to understand, and provide a framework that was less brittle).

Frameworks use introspection frequently, to discover the capabilities of objects the user has passed; for example, "does this object's class have a something() method? If so, call the object's do_something(); otherwise call the do_something_similar() framework function with the object as an argument." Some built-in functions also do this kind of introspection. The dir() built-in mentioned above returns the result of the argument object's __dir__() method if it has one; otherwise it uses built-in functionality to provide an "interesting" set of names (the result is not defined more clearly than that anywhere in the code).

This module allows you to dig as deep as you ever need to in terms of introspection. It provides many functions by which you can determine the properties of objects, including sixteen predicates that allow you to easily determine whether an object is of a particular type.

>>> import inspect
>>> from smtplib import SMTP
>>> from pprint import pprint
>>> pprint(inspect.getmembers(SMTP))
[('__class__', <class 'type'>),
 ('__delattr__', <slot wrapper '__delattr__' of 'object' objects>),
 ('__dict__', mappingproxy({...})),
 ('__doc__',
  "This class manages a connection to an SMTP or ESMTP server.\n
    SMTP Objects:\n
        SMTP objects have the following attributes:\n
            helo_resp\n
                This is the message given by the server in response to the\n
                most recent HELO command.\n\n
            ehlo_resp\n
                This is the message given by the server in response to the\n
                most recent EHLO command. This is usually multiline.\n\n
            does_esmtp\n
                This is a True value _after you do an EHLO command_, if the\n
                server supports ESMTP.\n\n
            esmtp_features\n
                This is a dictionary, which, if the server supports ESMTP,\n
                will _after you do an EHLO command_, contain the names of the\n
                SMTP service extensions this server supports, and their\n
                parameters (if any).\n\n
                Note, all extension names are mapped to lower case in the\n
                dictionary.\n\n
        See each method's docstrings for details.  In general, there is a\n
        method of the same name to perform each SMTP command.  There is also a\n
        method called 'sendmail' that will do an entire mail transaction.\n        "),
 ('__eq__', <slot wrapper '__eq__' of 'object' objects>),
    ...
 ('__weakref__', <attribute '__weakref__' of 'SMTP' objects>),
 ('_get_socket', <function SMTP._get_socket at 0x...>),
 ('close', <function SMTP.close at 0x...>),
    ...
 ('verify', <function SMTP.verify at 0x...>),
 ('vrfy', <function SMTP.verify at 0x...>)]
>>>
>>> pprint(inspect.getmembers(SMTP, inspect.ismethod))
[]
>>> pprint(inspect.getmembers(SMTP, inspect.isfunction))
[('__enter__', <function SMTP.__enter__ at 0x...>),
 ('__exit__', <function SMTP.__exit__ at 0x...>),
 ('__init__', <function SMTP.__init__ at 0x...>),
    ...
 ('verify', <function SMTP.verify at 0x...>),
 ('vrfy', <function SMTP.verify at 0x...>)]
>>> smtp = SMTP()
>>> pprint(inspect.getmembers(smtp, inspect.ismethod))
[('__enter__',
  <bound method SMTP.__enter__ of <smtplib.SMTP object at 0x...>>),
 ('__exit__',
  <bound method SMTP.__exit__ of <smtplib.SMTP object at 0x...>>),
 ('__init__',
  <bound method SMTP.__init__ of <smtplib.SMTP object at 0x...>>),
    ...
 ('verify',
  <bound method SMTP.verify of <smtplib.SMTP object at 0x...>>),
 ('vrfy', <bound method SMTP.verify of <smtplib.SMTP object at 0x...>>)]
>>>

>>> import inspect
>>> def f(a, b, c=1, d="one", *args, **kw):
...     print('a', a, 'b', b, 'c', c, 'd', d, 'args', args, 'kw', kw)
...
>>> inspect.getfullargspec(f)
FullArgSpec(args=['a', 'b', 'c', 'd'], varargs='args', varkw='kw', defaults=(1, 'one'), kwonlyargs=[], kwonlydefaults=None, annotations={})
>>> inspect.formatargspec(*inspect.getfullargspec(f))
"(a, b, c=1, d='one', *args, **kw)"
>>>

>>> str(inspect.signature(f))
"(a, b, c=1, d='one', *args, **kw)"