Introduction

Over the years of coding, nothing has given me more piece-of-mind (in the beginning frustration) than having proper CI/CD for my projects. This includes properly tested, formatted, linted, and type-hinted code with automated deployments. If any of this sounds scary, dont worry, we'll go over all of it plus how to do it in an easy to maintain fashion.

What is CI/CD?

They stand for continuous integration and continuous deployment. While these are two separate principles for maintaining quality production code, they mesh very nicely together so they are almost always mentioned with one another. Lets jump in!

Continuous Integration

I like to think of Continuous integration as

The practice of merging new code into your project in a way that reduces the chance of breaking old functionality, creating bugs, or introducing code smell.

Now you might be wondering what that means lets look through examples with and without continuous integration.

Without CI

In my early days of development, I would either commit directly to master (now main) or open an unchecked PR and hope for the best. While this allowed for quick development in the short term, it led to a world of headaches and even slower development speed as the code and/or organization grew in complexity.

Lets say I initially create a simple exponential function which I use that later in the code without passing in power as I'm okay with the default value of 2 like so,

def exp(base: float, power: float = 2) -> float:
    return base**power

exp(4) # 16

Now let's say I go back and say "Hm I don't like that power has a default value so I'm going to make it required" BUT I don't update all the calls of exp we'll get a TypeError.

exp(4) # TypeError: exp() missing 1 required positional argument: 'power'

If this function was properly tested, once the function is changed we should have the test fail which would indicate we're making something backwards-incompatible. While this is an extremely simple example, you can see how it'd be much easier to cause these issues if you have 10,000+ lines of code in a project. Additionally, some code doesn't have the benefit of being fully incapsulated so you have to be very careful when introducing backwards-incompatibility. Imagine if pandas started working differently on the next release without notifying the community, that wouldn't be good!

With CI

Now let's add a unit test and then break it! How fun! Before we start, a unit test is code that tests (duh) a unit (duh) of the code which is subjective but typically a function, class method, etc. So for this we'll make a test of exp,

def test_exp():
    assert exp(2,2) = 4
    assert exp(2) = 4

Initially no issue! But once we remove the default argument, we see how the second argument would throw a TypeError and we'd be alerted by our CI system.

Continuous Deployment

I like to think of Continuous deployment as

Delivering new versions of a software thought a fully automated process.

While this is more generic it is just as useful when wanting to efficiently ship new versions of your program without all the hassle. It's less "defensive programming" (although there is some) in the sense of stopping bugs like CI but saves the headaches and monotony of manual deployments. A good example of this is deploying new versions of your python package to PyPI.

Although typically not done for this reason, CD can have the benefit of lessening the chance of deploying the wrong version / branch of your code. With manual deployment, you may pull the wrong code or be on the wrong branch while an automated process will always grab from the same production source.

Tools

This all sounds nice but I know you're itching to see how it's applied and specifically applied in Python so let's go over the processes / tools. For Python CI, the current standards to have are unit testing (ideally 100% coverage), linting, formatting, and type-hinting. I'll explain the standards as well as what tools to use.

Dependency Management

To start, I'd like to point out the dependency manager and packaging tool I use. Although relatively new, Poetry has been getting quite popular for it's single file configuration (using pyproject.toml compared to the traditional setup.py system) and is the only one I use at this point.

The reason I mention it early on is it also allows you to run commands with the poetry run ... interface.

Unit/Coverage Testing

As mentioned before, unit testing is testing your code in chunks to make sure all the logic is sound and you're not introducing unintended effects. An additional piece is the idea of coverage testing which is making sure you're covering x% of your production code by unit tests. Let's say I have another 3 line function after exp but didn't test it. All my tests would pass but I'd only have 50% coverage which isn't ideal. To be extra secure, all your unit tests should pass and you should have 100% coverage.

My go to here is pytest. While unittest is what comes with Python, I like pytest's interface, it's configuration of coverage testing, and advanced features like fixtures.

Pytest uses the pyproject.toml file for configuration and can be executed with a simple poetry run pytest

Linting

The process of checking code for programmatic and stylistic errors.

While linting and code-smell is subjective, there's some widely agreed upon practices that lead to easy-to-read and easy-to-maintain code. These can range from max characters in a line to function complexity. While typically disregarded as "annoying" and a "waste of time" they can help immensely with readability. Not only does this help with onboarding new developers to your project but also helps when you revisit a file months later to be confused by what's going on.

My go to tool right now for linting is flake8. There's also Pylint which is more "restrictive". For less crucial codebases, I typically go towards flake8.

flake8 uses their own .flake8 file for configuration and can be executed with a simple poetry run flake8 [directories]

Formatting

Formatting sometimes overlaps with linting for stylistic things like line-length. The main things formatted are proper spacing for comments, maximum line-length, etc. to have cleaner code. Additionally, formatting allows to have the smallest diffs needed when reading a pull-request as there aren't any diffs for differences in your developers think the code should look.

In Python, black is rapidly becoming the standard for formatting while technically in beta, it's used in production all around the world so it's considered quite stable.

Black uses the pyproject.toml file for configuration and can be executed with a simple poetry run --check black [directories]

Type-hinting

If you haven't noticed yet, Python doesn't require types like other lower-level languages do (x = 1 vs. int x = 1). To bring some of the benefits of types to Python, the core team added type-hinting in Python 3.5. It looks like x: int = 1. While this isn't enforced at run-time, it allows for better documentation and static type analysis.

Currently, the go-to tool is mypy but I could see more tools pop up in the future as type-checking gains wider adoption.

Mypy uses the mypy.ini file for configuration and can be executed with a simple poetry run mypy

GitHub Actions

Now we bring it all together! To do so we'll be using Github Actions which is incredibly FREE. Yes you read that right, $0.00.

To have all these run on Github Actions, you need a YAML file stored in your .github/workflows directory.

Cookiecutter

For a full implementation of all these features plus docs publishing and publishing to PyPI checkout my cookiecutter,

Summary

Here we learned all the necessary tools and implementations for an easy-to-use CI/CD pipeline. I hope this helps! Feel free to leave a comment or reach out for any further questions.