Testing

Overview

Testing is the action of checking that a program behaves as expected. In general, when we talk about testing, we talk about automated testing. Automated testing consists in writing a program that verifies that another program (the main program) returns the expected output for a set of given inputs.

The usual location for the tests is in <your-program-module-dir>/tests. For example, if we have a program named pacman, and we have its source code in pacman/*, we will usually find its tests in pacman/tests/.

Python testing frameworks

As mentioned earlier, a test is just a program that verifies the returns of another program. For example, we could have a function add and a test for it:

def add(num1, num2):
    return num1 + num2

def test_add():
   res = add(2, 2)
   if res != 4:
      raise ValueError(f'add(2, 2) was expected to return 4, but it returned {res} instead.')

While this is a correct and meaningful test, by convention we will have the tests in separate files (and directories) than the main program. And we will use a testing framework that will standardise the structure, and will provide helpers for our tests.

pytest

Until few years ago, the de-facto standard for unit testing was the unittest framework, based on Java's Junit. For this reason it was included in the Python standard library, and does not require to be installed separately.

But in the last years, pytest has been growing in popularity. Mainly because a more Pythonic interface, and being able to express the tests in a more simple and correct way. pytest needs to be installed as a dependency, and once installed, tests can be autodetected and run with python -m pytest.

Tests can be anywhere in the subdirectories where the command is executed. But in general they will be in <module>/tests. To help pytest identify which tests to run, the files containing tests need to start with test_*.py. And inside them, tests will be groupped in functions, also starting with test_*. For example, if we want to test an hypothetic library named algebra:

# file: algebra/tests/test_add.py
import algebra

def test_add_positive():
    res = algebra.add(2, 2)
    assert isintance(res, int)
    assert res == 4

def test_add_negative():
    res = algebra.add(2, -3)
    assert isinstance(res, int)
    assert res == -1

Exceptions

While for new users, a Python exception is an error, in practice, they are a powerful way to give information about the execution of a piece of code. For example, when we execute 1 / 0, we can consider that the program failed. But we can also consider that the return of the program was a ZeroDivisionError exception, letting us know that we tried dividing something by zero, and that the result is unexpected. We then decide what to do by adding a try/except block.

Because exceptions are an important part of how a program interacts with programs calling it, we may want to validate that the program is generating the right exceptions for the cases where we except them. If we use pytest, we can use its pytest.raises to test that exceptions are raised in the expected way. More information can be found in this documentation.

Fixtures

In some cases, we want to perform the same exact tests for a set of different values. pytest comes with fixtures, a way to repeat tests, that among other things is able to run a tests for different parameters. Check the fixtures documentation for more information.

Tests output

There are many ways to customise the output of the test suite execution. The results can be displayed in the console in different formats, and also generated in a XML and upload to some internet services to analyse them. We can also generate a coverage report, indicating which percentage of the main program code is tested.

Use pytest --help to see the available options.