6. Errors and exceptions

Video: errors and exceptions.

Imperial students can also watch this video on Panopto.

It is a sight familiar to every programmer: instead of producing the desired result, the screen is filled with seemingly unintelligible garbage because an error has occurred. Producing errors is an unavoidable part of programming, so learning to understand and correct them is an essential part of learning to program.

6.1. What is an error?

In mathematics, we are used to the idea that an expression might not be defined. For example, in the absence of further information, \(0/0\) does not have a well-defined value. Similarly, the string of symbols \(3 \times \%\) does not have a mathematical meaning. It is likewise very easy to create statements or expressions in a programming language which either don’t have a well-defined meaning, or which just don’t amount to a meaningful statement within the rules of the language. A mathematician confronting an undefined mathematical expression can do little else than throw up their hands and ask the author what they meant. The Python interpreter, upon encountering code which has no defined meaning, responds similarly; though rather than raising its non-existent hands, it raises an exception. It is then up to the programmer to divine what to do next.

Let’s take a look at what Python does in response to a simple error:

In [3]: 0./0.
--------------------------------------------------------------------------
ZeroDivisionError                        Traceback (most recent call last)
Cell In [3], line 1
----> 1 0./0.

ZeroDivisionError: float division by zero

An important rule in interpreting Python errors, the reasons for which we will return to, is to always read the error message from the bottom up. In this case, the last line contains the name of the exception which has been raised, ZeroDivisionError, followed by a colon, followed by a descriptive string providing more information about what has gone wrong. In this case, that more or less says the same as the exception name, but that won’t be the case for all exceptions. The four lines above the exception are called a traceback. We’ll return to interpreting tracebacks presently. In this case the error is easy to interpret and understand: the code divided the float value 0. by another zero, and this does not have a well-defined result in Python’s arithmetic system.

6.1.1. Syntax errors

Now consider the case of an expression that doesn’t make mathematical sense:

In [5]: 3 * %
Cell In [5], line 1
    3 * %
        ^
SyntaxError: invalid syntax

This creates a syntax error, signified by a SyntaxError exception. In programming languages, as with human languages, the syntax is the set of rules which defines which expressions are well-formed. Notice that the earlier lines of a syntax error appear somewhat different to those of the previous exception. Almost all exceptions occur because the Python interpreter attempts to evaluate a statement or expression and encounters a problem. Syntax errors are a special case: when a syntax error occurs, the interpreter can’t even get as far as attempting to evaluate because the sequence of characters it has been asked to execute do not make sense in Python. This time, the error message shows the precise point in the line at which the Python interpreter found a problem. This is indicated by the caret symbol (^). In this case, the reason that the expression doesn’t make any sense is that the modulo operator (%) is not a permissible second operand to multiplication (*), so the Python interpreter places the caret under the modulo operator.

Even though the Python interpreter will highlight the point at which the syntax doesn’t make sense, this might not quite actually be the point at which you made the mistake. In particular, failing to finish a line of code will often result in the interpreter assuming that the expression continues on the next line of program text, resulting in the syntax error appearing to be one line later than it really occurs. Consider the following code:

a = (1, 2
print(a)

The error here is a missing closing bracket on the first line, however the error message which the Python interpreter prints when this code is run is:

  File "syntax_error.py", line 2
    print(a)
        ^
SyntaxError: invalid syntax

To understand why Python reports the error on the line following the actual problem, we need to understand that the missing closing bracket was not by itself an error. The user could, after all, validly continue the tuple constructor on the next line. For example, the following code would be completely valid:

a = (1, 2
     )
print(a)

This means that the Python interpreter can only know that something is wrong when it sees print, because print cannot follow 2 in a tuple constructor. The interpreter, therefore, reports that the print is a syntax error.

Hint

If the Python interpreter reports a syntax error at the start of a line, always check to see if the actual error is on the previous line.

6.2. Exceptions

Aside from syntax errors, which are handled directly by the interpreter, errors occur when Python code is executed and something goes wrong. In these cases the Python code in which the problem is encountered must signal this to the interpreter. It does this using a special kind of object called an exception. When an exception occurs, the interpreter stops executing the usual sequence of Python commands. Unless the programmer has taken special measures, to which we will return in Section 6.5, the execution will cease and an error message will result.

Because there are many things that can go wrong, Python has many types of exception built in. For example, if we attempt to access the number 2 position in a tuple with only two entries, then an IndexError exception occurs:

In [1]: (0, 1)[2]
--------------------------------------------------------------------------
IndexError                               Traceback (most recent call last)
Cell In [1], line 1
----> 1 (0, 1)[2]

IndexError: tuple index out of range

The exception type provides some indication as to what has gone wrong, and there is usually also an error message and sometimes more data to help diagnose the problem. The full list of built-in exceptions is available in the Python documentation. Python developers can define their own exceptions so there are many more defined in third-party packages. We will turn to the subject of defining new exception classes in Section 7.4.

6.3. Tracebacks: finding errors

Video: tracebacks.

Imperial students can also watch this video on Panopto.

The errors we have looked at so far have all been located in the top level of code either typed directly into iPython or executed in a script. However, what happens if an error occurs in a function call or even several functions down? Consider the following code, which uses the Polynomial class from Chapter 3:

In [1]: from example_code.polynomial import Polynomial

In [2]: p = Polynomial(("a", "b"))

In [3]: print(p)
bx + a

Perhaps surprisingly, it turns out that we are able to define a polynomial whose coefficients are letters, and we can even print the resulting object. However, if we attempt to add this polynomial to the number 1, we are in trouble:

In [4]: print(1 + p)
--------------------------------------------------------------------------
TypeError                                Traceback (most recent call last)
Cell In [4], line 1
----> 1 print(1 + p)

File ~/docs/principles_of_programming/object-oriented-programming/example_code/polynomial.py:59, in Polynomial.__radd__(self, other)
    58 def __radd__(self, other):
---> 59     return self + other

File ~/docs/principles_of_programming/object-oriented-programming/example_code/polynomial.py:38, in Polynomial.__add__(self, other)
    36 def __add__(self, other):
    37     if isinstance(other, Number):
---> 38         return Polynomial((self.coefficients[0] + other,)
    39                           + self.coefficients[1:])
    41     elif isinstance(other, Polynomial):
    42         # Work out how many coefficient places the two polynomials have in
    43         # common.
    44         common = min(self.degree(), other.degree()) + 1

TypeError: can only concatenate str (not "int") to str

This is a much larger error message than those we have previously encountered, however, the same principles apply. We start by reading the last line. This tells us that the error was a TypeError caused by attempting to concatenate (add) an integer to a string. Where did this error occur? This is a more involved question than it may first appear, and the rest of the error message above is designed to help us answer this question. This type of error message is called a traceback, as the second line of the error message suggests. In order to understand this message, we need to understand a little about how a Python program is executed, and in particular about the call stack.

6.3.1. The call stack

Video: the call stack.

Imperial students can also watch this video on Panopto.

A Python program is a sequence of Python statements, which are executed in a sequence determined by the flow control logic of the program itself. Each statement contains zero or more function calls [2], which are executed in the course of evaluating that statement.

One of the most basic features of a function call is that the contents of the function execute, and then the code which called the function continues on from the point of the function call, using the return value of the function in place of the call. Let’s think about what happens when this occurs. Before calling the function, there is a large amount of information which describes the context of the current program execution. For example, there are all of the module, function, and variable names which are in scope, and there is the record of which instruction is next to be executed. This collection of information about the current execution context is called a stack frame. We learned about stacks in Section 5.1, and the term “stack frame” is not a coincidence. The Python interpreter maintains a stack of stack frames called the call stack. It is also sometimes called the execution stack or interpreter stack.

The first frame on the stack contains the execution context for the Python script that the user ran or, in the case where the user worked interactively, for the iPython shell or Jupyter notebook into which the user was typing. When a function is called, the Python interpreter creates a new stack frame containing the local execution context of that function and pushes it onto the call stack. When that function returns, its stack frame is popped from the call stack, leaving the interpreter to continue at the next instruction in the stack frame from which the function was called. Because functions can call functions which call functions and so on in a nearly limitless sequence, there can be a number of stack frames in existence at any time.

6.3.2. Interpreting tracebacks

Let’s return to the traceback for our erroneous polynomial addition:

In [4]: print(1 + p)
--------------------------------------------------------------------------
TypeError                                Traceback (most recent call last)
Cell In [4], line 1
----> 1 print(1 + p)

File ~/docs/principles_of_programming/object-oriented-programming/example_code/polynomial.py:59, in Polynomial.__radd__(self, other)
    58 def __radd__(self, other):
---> 59     return self + other

File ~/docs/principles_of_programming/object-oriented-programming/example_code/polynomial.py:38, in Polynomial.__add__(self, other)
    36 def __add__(self, other):
    37     if isinstance(other, Number):
---> 38         return Polynomial((self.coefficients[0] + other,)
    39                           + self.coefficients[1:])
    41     elif isinstance(other, Polynomial):
    42         # Work out how many coefficient places the two polynomials have in
    43         # common.
    44         common = min(self.degree(), other.degree()) + 1

TypeError: can only concatenate str (not "int") to str

This shows information about a call stack comprising three stack frames. Look first at the bottom-most frame, which corresponds to the function in which the exception occurred. The traceback for this frame starts:

File ~/docs/principles_of_programming/object-oriented-programming/example_code/polynomial.py:38, in Polynomial.__add__(self, other)

This indicates that the frame describes code in the file polynomial.py (which, on the author’s computer, is located in the folder ~/principles_of_programming/object-oriented-programming/example_code/). Specifically, the stack frame describes the execution of the __add__() method, which is the special method responsible for polynomial addition. The lines below this show the line on which execution stopped (line 38, in this case) and a couple of lines on either side, for context.

The stack frame above this shows the function from which the __add__() method was called. In this case, this is the reverse addition special method, __radd__(). On line 59 __radd__() calls __add__() through the addition of self and other.

Finally, the top stack frame corresponds to the command that the user typed in iPython. This stack frame looks a little different from the others. Instead of a file name there and a function name there is Cell In [4], line 1. This indicates that the exception was raised on line 1 of the IPython cell In [4].

Hint

Older versions of Python display less helpful location information for the top stack frame, so in that case you might see something like <ipython-input-2-c3aeb16193d4> in rather than Cell In [4], line 1.

Hint

The proximate cause of the error will be in the last stack frame printed, so always read the traceback from the bottom up. However, the ultimate cause of the problem may be further up the call stack, so don’t stop reading at the bottom frame!

6.4. Raising exceptions

Video: raising an exception.

Imperial students can also watch this video on Panopto.

Thus far we’ve noticed that an exception occurs when something goes wrong in a program, and that the Python interpreter will stop at that point and print out a traceback. We’ll now examine the process by which an exception occurs.

An exception is triggered using the raise keyword. For example, suppose we want to ensure that the input to our Fibonacci function is an integer. All Python integers are instances of numbers.Integral, so we can check this. If we find a non-integer type then the consequence should be a TypeError. This is achieved by raising the appropriate exception, using the raise statement. The keyword raise is followed by the exception. Almost all exceptions take a string argument, which is the error message to be printed. In Listing 6.1, we inform the user that we were expecting an integer rather than the type actually provided.

Listing 6.1 A version of the Fibonacci function which raises an exception if a non-integer type is passed as the argument.

from numbers import Integral


def typesafe_fib(n):
    """Return the n-th Fibonacci number, raising an exception if a
    non-integer is passed as n."""
    if not isinstance(n, Integral):
            raise TypeError(
                f"fib expects an integer, not a {type(n).__name__}"
            )
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-2) + fib(n-1)

If we now pass a non-integer value to this function, we observe the following:

In [1]: from fibonacci.typesafe_fibonacci import typesafe_fib
In [2]: typesafe_fib(1.5)
--------------------------------------------------------------------------
TypeError                                Traceback (most recent call last)
Cell In [2], line 1
----> 1 typesafe_fib(1.5)

File ~/docs/principles_of_programming/object-oriented-programming/fibonacci/typesafe_fibonacci.py:8, in typesafe_fib(n)
    5 """Return the n-th Fibonacci number, raising an exception if a
    6 non-integer is passed as n."""
    7 if not isinstance(n, Integral):
----> 8     raise TypeError(
    9         f"fib expects an integer, not a {type(n).__name__}"
    10     )
    11 if n == 0:
    12     return 0

TypeError: fib expects an integer, not a float

This is exactly what we have come to expect: execution has stopped and we see a traceback. Notice that the final line is the error message that we passed to TypeError. The only difference between this and the previous errors we have seen is that the bottom stack frame explicitly shows the exception being raised, while previously the stack showed a piece of code where an error had occurred. This minor difference has to do with whether the particular piece of code where the exception occurred is written in Python, or is written in a language such as C and called from Python. This distinction is of negligible importance for our current purposes.

Note

An exceptionally common mistake that programmers make when first trying to work with exceptions is to write:

return Exception

instead of:

raise Exception

This mistake is the result of a confusion about what return and raise do. return means “the function is finished, here is the result”. raise means “something exceptional happened, execution is stopping without a result”.

6.5. Handling exceptions

Video: handling exceptions.

Imperial students can also watch this video on Panopto.

So far we have seen several different sorts of exception, how to raise them, and how to understand the resulting traceback. The traceback is very helpful if the exception was caused by a bug in our code, as it is a rich source of the information needed to understand and correct the error. However, sometimes an exception is a valid result of a valid input, and we just need the program to do something out of the ordinary to deal with the situation. For example, Euclid’s algorithm for finding the greatest common divisor of \(a\) and \(b\) can very nearly be written recursively as:

def gcd(a, b):
    return gcd(b, a % b)

This works right up to the point where b becomes zero, at which point we should stop the recursion and return a. What actually happens if we run this code? Let’s try:

In [2]: gcd(10, 12)
--------------------------------------------------------------------------
ZeroDivisionError                        Traceback (most recent call last)
Cell In[2], line 1
----> 1 gcd(10, 12)

Cell In[1], line 2, in gcd(a, b)
    1 def gcd(a, b):
----> 2     return gcd(b, a % b)

Cell In[1], line 2, in gcd(a, b)
    1 def gcd(a, b):
----> 2     return gcd(b, a % b)

    [... skipping similar frames: gcd at line 2 (1 times)]

Cell In[1], line 2, in gcd(a, b)
    1 def gcd(a, b):
----> 2     return gcd(b, a % b)

ZeroDivisionError: integer modulo by zero

Notice how the recursive call to gcd() causes several stack frames that look the same. Indeed, the Python interpreter even notices the similarity and skips over one. That makes sense: gcd() calls itself until b is zero, and then we get a ZeroDivisionError because modulo zero is undefined. To complete this function, what we need to do is to tell Python to stop at the ZeroDivisionError and return a instead. Listing 6.2 illustrates how this can be achieved.

Listing 6.2 A recursive implementation of Euclid’s algorithm which catches the ZeroDivisionError to implement the base case.

def gcd(a, b):
    try:
        return gcd(b, a % b)
    except ZeroDivisionError:
        return a

The new structure here is the try… except block. The try keyword defines a block of code, in this case just containing return gcd(b, a % b). The except is optionally followed by an exception class, or a tuple of exception classes. This case, the except is only followed by the ZeroDivisionError class. What this means is that if a ZeroDivisionError is raised by any of the code inside the try block then, instead of execution halting and a traceback being printed, the code inside the except block is run.

In the example here, this means that once b is zero, instead of gcd being called a further time, a is returned. If we run this version of gcd() then we have, as we might expect:

In [2]: gcd(10, 12)
Out[2]: 2

6.5.1. Except clauses

Video: further exception handling.

Imperial students can also watch this video on Panopto.

Let’s look in a little more detail at how except works. The full version of the except statement takes a tuple of exception classes. If an exception is raised matching any of the exceptions in that tuple then the code in the except block is executed.

It’s also possible to have more than one except block following a single try statement. In this case, the first except block for which the exception matches the list of exceptions is executed. For example:

In [1]: try:
    ...:     0./0
    ...: except TypeError, KeyError:
    ...:     print("Type or key error")
    ...: except ZeroDivisionError:
    ...:     print("Zero division error")
    ...:
Zero division error

Note

It is also possible to omit the list of exceptions after except. In this case, the except block will match any exception which is raised in the corresponding try block. Using unconstrained except blocks like this is a somewhat dangerous strategy. Usually, the except block will be designed to deal with a particular type of exceptional circumstance. However, an except block that catches any exception may well be triggered by a completely different exception, in which case it will just make the error more confusing by obscuring where the issue actually occurred.

6.5.2. Else and finally

It can also be useful to execute some code only if an exception is not raised. This can be achieved using an else clause. An else clause after a try block is caused only if no exception was raised.

It is also sometimes useful to be able to execute some code no matter what happened in the try block. If there is a finally clause then the code it contains will be executed whether or not an exception is raised and whether or not any exception is handled by an except clause. The contents of the finally clause will always execute. This may be useful, for example, if it is necessary to close an external file or network connection at the end of an operation, even if an exception is raised. The full details of the finally clause are covered in the section of the official Python tutorial on handling exceptions.

This plethora of variants on the try block can get a little confusing, so a practical example may help. Listing 6.3 prints out a different message for each type of clause.

Listing 6.3 A demonstration of all the clauses of the try block.

def except_demo(n):
    """Demonstrate all the clauses of a `try` block."""

    print(f"Attempting division by {n}")
    try:
        print(0./n)
    except ZeroDivisionError:
        print("Zero division")
    else:
        print("Division successful.")
    finally:
        print("Finishing up.")

If we execute except_demo() for a variety of arguments, we can observe this complete try block in action. First, we provide an input which is a valid divisor:

In [1]: from example_code.try_except import except_demo
In [2]: except_demo(1)
Attempting division by 1
0.0
Division successful.
Finishing up.

Here we can see the output of the division, the else block, and the finally block. Next we divide by zero:

In [3]: except_demo(0)
Attempting division by 0
Zero division
Finishing up.

This caused a ZeroDivisionError, which was caught by the first except clause. Since an exception was raised, the the else block is not executed, but the finally block still executes. Finally, if we attempt to divide by a string, the exception is not handled, but the finally block executes before the exception causes a traceback:

In [4]: except_demo("frog")
Attempting division by frog
Finishing up.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[4], line 1
----> 1 except_demo("frog")

Cell In[3], line 6, in except_demo(n)
      4 print(f"Attempting division by {n}")
      5 try:
----> 6     print(0./n)
      7 except ZeroDivisionError:
      8     print("Zero division")

TypeError: unsupported operand type(s) for /: 'float' and 'str'

6.5.3. Exception handling and the call stack

An except block will handle any matching exception raised in the preceding try block. The try block can, of course, contain any code at all. In particular it might contain function calls which themselves may well call further functions. This means that an exception might occur several stack frames down the call stack from the try clause. Indeed, some of the functions called might themselves contain try blocks with the result that an exception is raised at a point which is ultimately inside several try blocks.

The Python interpreter deals with this situation by starting from the current stack frame and working upwards, a process known as unwinding the stack. Listing 6.4 shows pseudocode for this process.

Listing 6.4 Pseudocode for the process of unwinding the stack, in which the interpreter successively looks through higher stack frames to search for an except clause matching the exception that has just been raised.

while call stack not empty:
    if current execution point is in a try block \
            with an except matching the current exception:
        execution continues in the except block
    else:
        pop the current stack frame off the call stack

# Call stack is now empty
print traceback and exit

6.6. Exceptions are not always errors

This chapter is called “Errors and exceptions”, so it is appropriate to finish by drawing attention to the distinction between these two concepts. While user errors and bugs in programs typically result in an exception being raised, it is not the case that all exceptions result from errors. The name “exception” means what it says, it is an event whose occurrence requires an exception to the normal sequence of execution.

The StopIteration exception which we encountered in Section 5.6 is a good example of an exception which does not indicate an error. The end of the set of things to be iterated over does not indicate that something has gone wrong, but it is an exception to the usual behaviour of __next__(), which Python needs to handle in a different way from simply returning the next item.

6.7. Glossary

call stack

execution stack

interpreter stack

The stack of stack frames in existence. The current item on the stack is the currently executing function, while the deepest item is the stack frame corresponding to the user script or interpreter.

exception

An object representing an out of the ordinary event which has occurred during the execution of some Python code. When an exception is raised the Python interpreter doesn’t continue to execute the following line of code. Instead, the exception is either handled or execution stops and a traceback is printed.

stack frame

An object encapsulating the set of variables which define the execution of a Python script or function. This information includes the code being executed, all the local and global names which are visible, the last instruction that was executed, and a reference to the stack frame which called this function.

syntax

The set of rules which define what is a well-formed Python statement. For example the rule that statements which start blocks must end with a colon (:) is a syntax rule.

syntax error

The exception which occurs when a statement violates the syntax rules of Python. Mismatched brackets, missing commas, and incorrect indentation are all examples of syntax errors.

traceback

stack trace

back trace

A text representation of the call stack. A traceback shows a few lines of code around the current execution point in each stack frame, with the current frame at the bottom and the outermost frame at the top.

6.8. Exercises

Using the information on the book website obtain the skeleton code for these exercises.

Exercise 6.1

The Newton-Raphson method is an iterative method for approximately solving equations of the form \(f(x)=0\). Starting from an initial guess, a series of (hopefully convergent) approximations to the solution is computed:

\[x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}\]

The iteration concludes successfully if \(|f(x_{n+1})| < \epsilon\) for some user-specified tolerance \(\epsilon>0\). The sequence is not guaranteed to converge for all combinations of function and starting point, so the iteration should fail if \(n\) exceeds a user-specified number of iterations.

The skeleton code for this chapter contains a function nonlinear_solvers.solvers.newton_raphson() which takes as arguments a function, its derivative and a starting point for the iteration. It can also optionally be passed a value for \(\epsilon\) and a maximum number of iterations to execute. Implement this function. If the iteration succeeds then the last iterate, \(x_{n+1}\), should be returned.

nonlinear_solvers.solvers also defines an exception, ConvergenceError. If the Newton-Raphson iteration exceeds the number of iterations allowed then this exception should be raised, with an appropriate error message.

Exercise 6.2

The bisection method is a slower but more robust iterative solver. It requires a function \(f\) and two starting points \(x_0\) and \(x_1\) such that \(f(x_0)\) and \(f(x_1)\) differ in sign. At each stage of the iteration, the function is evaluated at the midpoint of the current points \(x^* = (x_0 + x_1)/2\). If \(|\,f(x^*)|<\epsilon\) then the iteration terminates successfully. Otherwise, \(x^*\) replaces \(x_0\) if \(f(x_0)\) and \(f(x^*)\) have the same sign, and replaces \(x_1\) otherwise.

Implement nonlinear_solvers.solvers.bisection(). As before, if the iteration succeeds then return the last value of \(x\). If the maximum number of iterations is exceeded, raise ConvergenceError with a suitable error message. The bisection method has a further failure mode. If \(f(x_0)\) and \(f(x_1)\) do not differ in sign then your code should raise ValueError with a suitable message.

Exercise 6.3

Implement the function nonlinear_solvers.solvers.solve(). This code should first attempt to solve \(f(x)=0\) using your Newton-Raphson function. If that fails it should catch the exception and instead try using your bisection function.

Footnotes

[1]

https://docs.python.org/3/tutorial/errors.html#tut-handling

[2]

“Function call” here includes method calls and operations implemented using a special method.

[3]

https://object-oriented-python.github.io/edition3/exercises.html