10. Further object-oriented features

In this chapter, we’ll tie up a few loose ends by examining in detail some programming concepts and Python features which we have encountered but not really studied in the preceding chapters.

10.1. Decorators

Video: decorators.

Imperial students can also watch this video on Panopto.

In Chapter 9 we encountered the functools.singledispatch() decorator, which turns a function into a single dispatch function. More generally, a decorator is a function which takes in a function and returns another function. In other words, the following:

@dec
def func(...):
    ...

is equivalent to:

def func(...):
    ...
func = dec(func)

Decorators are therefore merely syntactic sugar, but can be very useful in removing the need for boiler-plate code at the top of functions. For example, your code for Exercise %s probably contains a lot of repeated code a similar to the following:

def __add__(self, other):
    """Return the Expr for the sum of this Expr and another."""
    if isinstance(other, numbers.Number):
        other = Number(other)
    return Add(self, other)

We could define a decorator to clean up this code as follows:

Listing 10.1 A decorator which casts the second argument of a method to an expressions.Number if that argument is a number.

from functools import wraps

def make_other_expr(meth):
    """Cast the second argument of a method to Number when needed."""
    @wraps(meth)
    def fn(self, other):
        if isinstance(other, numbers.Number):
            other = Number(other)
        return meth(self, other)
    return fn

Now, each time we write one of the special methods of Expr, we can instead write something like the following:

@make_other_expr
def __add__(self, other):
    """Return the Expr for the sum of this Expr and another."""
    return Add(self, other)

Let’s look closely at what the decorator in Listing 10.1 does. The decorator takes in one function, meth() an returns another one, fn(). Notice that we let fn() take the same arguments as meth(). If you wanted to write a more generic decorator that worked on functions with different signatures, then you could define function as fn(*args, **kwargs) and pass these through to meth().

The contents of fn() are what will be executed every time meth() is called. We use this to check the type of other and cast it to Number, and then call the original meth() on the modified arguments. We could also execute code that acts on the value that meth() returns. To do this we would assign the result of meth() to a variable and then include more code after line 9.

Finally, notice that we have wrapped fn in another decorator, functools.wraps(). The purpose of this decorator is to copy the name and docstring from meth() to fn(). The effect of this is that if the user calls help() on a decorated function then they will see the name and docstring for the original function, and not that of the decorator.

10.1.1. Decorators which take arguments

Our make_other_expr decorator doesn’t have brackets after its name, and doesn’t take any arguments. However functools.wraps() does have brackets, and takes a function name as an argument. How does this work? The answer is yet another wrapper function. A decorator is a function which takes a function and returns a function. functools.wraps() takes an argument (it happens to be a function but other decorators take other types) and returns a decorator function. That is, it is a function which takes in arguments and returns a function which takes a function and returns a function. It’s functions all the way down!

10.1.2. The property decorator

Back in Chapter 3, we gave the Polynomial class a degree() method:

def degree(self):
    return len(self.coefficients) - 1

This enables the following code to work:

In [1]: from example_code.polynomial import Polynomial

In [2]: p = Polynomial((1, 2, 4))

In [3]: p.degree()
Out[3]: 2

However, the empty brackets at the end of degree() are a bit clunky: why should we have to provide empty brackets if there are no arguments to pass? This represents a failure of encapsulation, because we shouldn’t know or care from the outside whether degree() is a method or a data attribute. Indeed, the developer of the polynomial module should be able to change that implementation without changing the interface. This is where the built-in property decorator comes in. property transforms methods that take no arguments other than the object itself into attributes. So, if we had instead defined:

@property
def degree(self):
    return len(self.coefficients) - 1

Then degree would be accessible as an attribute:

In [1]: from example_code.polynomial import Polynomial

In [2]: p = Polynomial((1, 2, 4))

In [3]: p.degree
Out[3]: 2

10.1.3. The functools module

The functools module is part of the Python Standard Library. It provides a collection of core higher order functions, some of which we have already met earlier in the course. Since decorators are an important class of higher order function, it is unsurprising that functools provides several very useful ones. We will survey just a few here:

functools.cache

Some functions can be very expensive to compute, and may be called repeatedly. A cache stores the results of previous function calls. If the function is called again with a combination of argument values that have previously been used, the function result is returned from the cache instead of the function being called again. This is a trade-off of execution time against memory usage, so one has to be careful how much memory will be consumed by the cache.

functools.lru_cache

A least recently used cache is a limited size cache where the least recently accessed items will be discarded if the cache is full. This has the advantage that the memory usage is bounded, but the drawback that cache eviction may take time, and that more recomputation may occur than in an unbounded cache.

functools.singledispatch

We met this in Section 9.4.3. This decorator transforms a function into a single dispatch function.

10.2. Abstract base classes

Video: Abstract base classes.

Imperial students can also watch this video on Panopto.

We have now on several occasions encountered classes which are not designed to be instantiated themselves, but merely serve as parent classes to concrete classes which are intended to be instantiated. Examples of these classes include numbers.Number, example_code.groups.Group, and the Expr, Operator, and Terminal classes from Chapter 9. These classes that are only ever parents are called abstract base classes. They are abstract in the sense that they define (some of the) properties of their children, but without providing full implementations of them. They are base classes in the sense that they are intended to be inherited from.

Abstract base classes typically fulfil two related roles: they provide the definition of an interface that child classes can be expected to follow, and they provide a useful way of checking that an object of a concrete class has particular properties.

10.2.1. The abc module

The concept of an abstract base class is itself an abstraction: an abstract base class is simply a class which is designed not to be instantiated. This requires no support from particular language features. Nonetheless, there are features that a language can provide which makes the creation of useful abstract base classes easy. In Python, these features are provided by the abc module in the Standard Library.

The abc module provides the ABC class. This is itself an abstract base class: there is no need to ever make an object of type ABC. Instead, classes inherit from ABC in order to access features that it provides.

10.2.2. Abstract methods

Let’s look back at the groups example from Chapter 7. We defined the base Group class and specified that child classes had to implement the _validate() and operator() methods as well as the symbol class attribute. But how should we actually know that these methods and attribute are required? This might be documented, but that is somewhat hit and miss: it is often less than completely obvious where to look for the documentation. Abstract methods provide a much more satisfactory solution to this problem. The example_code.groups_abc module is an update of the example_code.groups module which uses the ABC class.

Listing 10.2 An abstract base class version of the Group class. Note that the class itself inherits from ABC, and the methods and attribute to be implemented by the child classes have the abstractmethod decorator.

from abc import ABC, abstractmethod

class Group(ABC):
    """A base class containing methods common to many groups.

    Each subclass represents a family of parametrised groups.

    Parameters
    ----------
    n: int
        The primary group parameter, such as order or degree. The
        precise meaning of n changes from subclass to subclass.
    """

    def __init__(self, n):
        self.n = n

    @property
    @abstractmethod
    def symbol(self):
        """Represent the group name as a character."""
        pass

    @abstractmethod
    def _validate(self, value):
        """Ensure that value is an allowed element value in this group."""
        pass

    @abstractmethod
    def operation(self, a, b):
        """Return a ∘ b using the group operation ∘."""
        pass

    def __call__(self, value):
        """Create an element of this group."""
        return Element(self, value)

    def __str__(self):
        """Return a string in the form symbol then group parameter."""
        return f"{self.symbol}{self.n}"

    def __repr__(self):
        """Return the canonical string representation of the element."""
        return f"{type(self).__name__}({repr(self.n)})"

There are a few features of Listing 10.2 which are noteworthy. First, observe that Group now inherits from ABC. This simply enables the features that we will use next. The new Group class has _validate() and operator() methods, but these don’t actually do anything (their contents are merely pass). They are, however, decorated with abstractmethod. The effect of this decorator can be observed if we try to instantiate Group:

In [1]: from example_code.groups_abc import Group

In [2]: Group(1)
--------------------------------------------------------------------------
TypeError                                Traceback (most recent call last)
Cell In [2], line 1
----> 1 Group(1)

TypeError: Can't instantiate abstract class Group with abstract methods _validate, operation, symbol

The combination of inheriting from ABC and the abstractmethod decorator has the effect that instantiating this class is an error, and we are told why. We have skipped over the symbol attribute. There is no abstractattribute decorator, but the same effect can be achieved by creating an abstractmethod and converting it into a data attribute using property. In this case, the order of decorators is important: abstractmethod always needs to be the last, innermost, decorator.

The subclasses of Group that we defined, define all three of these attributes, so they can still be instantiated. For example:

In [1]: from example_code.groups_abc import CyclicGroup

In [2]: C = CyclicGroup(3)

In [3]: print(C(1) * C(2))
0_C3

This illustrates the utility of this use of abstract base classes: the base class can specify what subclasses need to implement. If a subclass does not implement all the right attributes then a helpful error is generated, and subclasses that do implement the class fully work as expected.

10.2.3. Duck typing

Before we turn to the second use of abstract base classes, it is useful to divert our attention to what might be thought of as the type philosophy of Python. Many programming languages are strongly typed. This means that in situations such as passing arguments to functions, the type of each variable is specified, and it is an error to pass a variable of the wrong type. This is not the Python approach. Instead, Python functions typically do not check the types of their arguments beyond ensuring that they have the basic properties required of the operation in question. It doesn’t really matter what the type of an object is, so long as it has the operations, methods, and attributes required.

The term that is used for this approach to data types is duck typing: if a data type walks like a duck and quacks like a duck, then it might as well be a duck. This does, however, beg the question of how a program should know if an object has the right properties in a given circumstance. It is here that the second use of abstract base classes comes into play.

10.2.4. Virtual subclasses

Video: virtual subclasses.

Imperial students can also watch this video on Panopto.

We learned in Chapter 3 that we can determine if a type is a number by checking if it is an instance of numbers.Number. This is a slightly different usage of abstract base classes. Rather than providing part of the implementation of types such as float, numbers.Number provides a categorisation of objects into numbers and non-numbers. This aids duck typing, by enabling much more general type checking.

The Standard Library contains many abstract base classes whose role is to support duck typing by identifying objects with particular properties. For example, the collections.abc module provides abstract base classes for container objects with particular properties. The collections.abc.Iterable abstract base class groups all iterable containers. For example:

In [1]: from collections.abc import Iterable

In [2]: from example_code.linked_list import Link

In [3]: issubclass(Link, Iterable)
Out[3]: True

Hang on, though, what magic is this? We didn’t declare Link as inheriting from Iterable.

What is going on here is a form of reverse inheritance process. Rather than Link declaring that it inherits from Iterable, Iterable determines that Link is its subclass. It’s a sort of adoption mechanism for classes. Of course the authors of the Standard Library don’t know that we will declare Link, so there is no code explicitly claiming Link as a subclass of Iterable. Instead, any class which implements the __iter__() special method is a subclass of Iterable. How does this work? Well, isinstance() and issubclass() are implemented with the help of, you guessed it, yet another special method. This time the special method is __subclasshook__().

Listing 10.3 The source code for collections.abc.Iterable extracted from the Git repository for the standard Python language implementation.

from abc import ABCMeta, abstractmethod

...

def _check_methods(C, *methods):
    mro = C.__mro__
    for method in methods:
        for B in mro:
            if method in B.__dict__:
                if B.__dict__[method] is None:
                    return NotImplemented
                break
        else:
            return NotImplemented
    return True

...

class Iterable(metaclass=ABCMeta):

    __slots__ = ()

    @abstractmethod
    def __iter__(self):
        while False:
            yield None

    @classmethod
    def __subclasshook__(cls, C):
        if cls is Iterable:
            return _check_methods(C, "__iter__")
        return NotImplemented

    __class_getitem__ = classmethod(GenericAlias)

Listing 10.3 shows the actual source code for Iterable [1]. Let’s walk through this. The inheritance in line 19 is essentially equivalent to inheriting from abc.ABC. Similarly, lines 21 and 34 are unrelated technical code. At line 24 we see the object.__iter__() special method, decorated with abstractmethod. This ensures that classes that do explicitly inherit from Iterable have to implement object.__iter__().

The part that currently concerns us, though, is the declaration of __subclasshook__() at line 29. __subclasshook__() is declared as class method. This means that it will be passed the class itself as its first argument, in place of the object. It is conventional to signal this difference by calling the first parameter cls instead of self. The second parameter, C is the class to be tested.

In common with the special methods for arithmetic, __subclasshook__() returns NotImplemented to indicate cases that it cannot deal with. In this case, if the current class is not Iterable (this would happen if the method were called on a subclass of Iterable) then NotImplemented is returned. If we really are checking C against Iterable then the _check_methods helper function is called. The fine details of how this works are a little technical, but in essence the function loops over C and its superclasses in order (C.__mro__ is the method resolution order) and checks if the relevant methods are defined. If they are all found then True is returned, otherwise the result is NotImplemented. An implementation of __subclasshook__() could also return False to indicate that C is definitely not a subclass.

10.3. Glossary

abstract base class

A class designed only to be the parent of other classes, and never to be instantiated itself. Abstract classes often define the interfaces of methods but leave their implementations to the concrete child classes.

abstract method

A method whose presence is required by an abstract base class but for which concrete subclasses are required to provide the implementation.

class method

A method which belongs to the class itself, rather than to the instances of the class. Class methods are declared using the classmethod decorator and take the class (cls) as their first argument, instead of the instance (self). See also: class attribute.

decorator

A syntax for applying higher order functions when defining functions. A decorator is applied by writing @ followed by the decorator name immediately before the declaration of the function or method to which the decorator applies.

duck typing

The idea that the precise type of an object is not important, it is only important that the object has the correct methods or attributes for the current operation. If an object walks like a duck, and quacks like a duck then it can be taken to be a duck.

higher order function

A function which acts on other functions, and which possibly returns another function as its result.

method resolution order

MRO

A sequence of the superclasses of the current class ordered by increasing ancestry. The superclasses of a class are searched in method resolution order to find implementations of methods and attributes.

syntactic sugar

A feature of the programming language which adds no new functionality, but which enables a clearer or more concise syntax. Python special methods are a form of syntactic sugar as they enable, for example, the syntax a + b instead of something like a.add(b).

virtual subclass

A class which does not declare its descent from the superclass through its definition, but which is instead claimed as a subclass by the superclass.

10.4. Exercises

Using the information on the book website obtain the skeleton code for these exercises.

Exercise

The objective of this exercise is to write a decorator which logs whenever the decorated function is called. This sort of decorator could be very useful in debugging code. Create the decorator in the log_decorator.log_decorator module and ensure it is importable as log_decorator.log_call. The decorator should be applicable to functions taking any combination of arguments.

The logging itself should be accomplished using the built-in logging module by calling logging.info() and passing the log message.

The log message should comprise the string "Calling: " followed by the function name (accessible using the __name__ attribute), followed by round brackets containing first the repr() of the positional arguments, followed by the key=value pairs the keyword arguments.

Exercise

The groups.groups module in the skeleton code is the new version introduced above, using an abstract base class. The log_decorator.log_call decorator has been applied to the Group._validate() abstract method. However, even once you have implemented this decorator, it never gets called. Your challenge is to modify groups.groups so that the decorator is called every time a subclass _validate() method is called, but without moving or duplicating @log_call.

Footnotes

[1]

Most of the Python Standard Library is written in Python, so diving in and reading the source code is often an option if you really want to know how some part of the language works.

[2]

https://object-oriented-python.github.io/edition3/exercises.html