Automate Your Python Tests With Continuous Integration

Photo by Birmingham Museums Trust on Unsplash

This article is part of a series How to Organically Grow your Python Project in which I cover the different steps you will go through when starting a new Python hobby project. The focus of this series lies on not knowing in advance what you want to make and thus not planning the whole project from the start.

In this article I’ll talk about how you can automate running the tests of your Python hobby project. Automated testing can be achieved in several ways but I will limit this article to explicitly showing one and conceptually talk about the others. The one I will focus on is the easiest to implement. As with the other articles in this series, I will elaborate on the steps I have chronologically taken to achieve this for my own hobby project. This should convince you that developing an arbitrary complex application can, and should, be done incrementally. One step at a time.

At this stage, my project has grown to as stage with several source code files in my package directory factpy and a number of unit tests in a few files in the tests directory. I am at a point in the development process where I continuously add functionality, write corresponding tests, run those tests using pytest, and push my code to my remote repository on Bitbucket. This is a well-oiled process. But what if I forget to run my tests and push straight to my remote repository? There might be no problem. With the emphasis on might. If a test would have failed, I will not have caught this problem. Sure, I will notice it the next time I run all tests but at that time I’ll probably be working on an unrelated part of the code, meaning that I’ll have to shift my attention. You might think: “Pfff, that’s not really a problem!” But I guarantee that after a few times you’ll find yourself getting frustrated fairly quickly.

But what if I forget to run my tests and push straight to my remote repository?

So, let’s get started! The easiest way to automate your tests, is to make them run each time you push to your remote repository. Luckily, all well-known version control repository hosting services have built-in functionality for this. Bitbucket and Gitlab have Pipelines, Github has Github Actions, and AWS has CodeBuild. These tools are part of the continuous integration and continuous delivery/deployment process, commonly abbreviated to CI/CD. These CI/CD pipelines frequently consist of several phases which will be executed such as building code, testing code, and deploying code. In practice, they will most likely be a combination of Docker containers. But just think of a pipeline as a separate machine, like your own, on which you can execute a list of commands. Executing this list, or running the pipeline, can be done manually or via a trigger such as pushing to the repository. As my repository is hosted on Bitbucket, I’ll just refer to their pipeline option.

Bitbucket makes it easy to create your first pipeline by offering numerous pre-made templates. It even suggests to use the Build and test Python template for my project. They also have a guide and a documentation page for Python pipelines.

Bitbucket Pipeline templates

Starting from such a template, modifying it slightly, and removing the parts I don’t yet need, the pipeline specifications for my project boil down to this bitbucket-pipelines.yml file in the project’s root directory:

# This is a sample build configuration for Python.
# Check our guides at https://confluence.atlassian.com/x/x4UWN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: python:3.8
pipelines:
  default:
    - step:
        script: 
          - pip install -r pip-requirements.txt
          - python -m pytest

It uses the Official Python 3.8 Docker image as a base image which has Python with common libraries pre-installed. Only a default pipeline is defined which applies to all branches that don’t match another pipeline definition in the file. The default pipeline runs on every push to the repository unless a branch-specific pipeline is defined. Next, a step is defined, where each step loads a new Docker container that includes a clone of the current repository. Within this step , a script contains a list of commands that are run, in order. This particular pipeline will execute two commands, the pip install one and the pytest one. I’ll come back to why I’m using pip instead on conda, like on my development machine. More details on how create such a bitbucket-pipelines.yml file can be found in the documentation.

Using conda in a CI server or Docker image is not straightforward because you cannot emulate the activation of conda environments but you need Conda’s own activation infrastructure. Nevertheless, you can use conda but you will need a more complex setup and this is beyond the scope of this article. Because pip is pre-installed in the Python image, it is more convenient to use. In order to install packages using pip, I need to list the packages from within my conda environment in a pip-friendly format. Even though, you should avoid mixing pip and conda, using pip freeze > pip-requirements.txt (inside your environment) will correctly return the installed packages, irrespective of being installed with conda or pip. Note that I do not 100 percent guarantee that this will always work. My pip-requirements.txt file looks like this:

attrs==19.3.0
autopep8==1.4.4
certifi==2020.4.5.1
more-itertools==8.2.0
mypy==0.770
mypy-extensions==0.4.3
packaging==20.3
pluggy==0.13.1
psutil==5.7.0
py==1.8.1
pycodestyle==2.5.0
pyparsing==2.4.7
pytest==5.4.2
six==1.14.0
typed-ast==1.4.1
typing-extensions==3.7.4.1
wcwidth==0.1.9

To run the unit tests, the pipeline executes the python -m pytest command. Because I do not have to type this command frequently, it is not needed to tweak the CI Python environment’s path such that it can run the tests with the shorter pytest command (like I did in the previous article).

That’s it! From now on, every push to the remote repository will trigger the pipeline which will run all the unit tests. The results of this run can be inspected in Bitbucket. The output will either be a failed or successful run.

Results of pipeline execution

Automatically running test after pushing to the remote repository is good but this still allows tests to fail after pushing if you didn’t test everything locally. To catch failing tests before pushing, you can create something call git hooks which are scripts that run automatically every time a particular event occurs in a Git repository. You can find samples of such scripts in your own repository’s .git/hooks directory. In particular, the pre-commit hook can be used to automatically run units test before actually committing to the local repository. If the tests fail, the commit will be aborted. This allows you to more quickly catch mistakes and better enforce the shift-left movement.

Shift-left testing is an approach to software testing and system testing in which testing is performed earlier in the lifecycle.

In the next article I’ll talk about how to publish your well-tested application as a Python package to PyPi. This way other people can use your package or even incorporate it in their own.