Introduction to Build Systems

What are few of the things that happen pretty frequently in a project? Continuous Integration checks? Deployment after every pull request merge? Onboarding new members and starting their local setup? Running a code scan? All of these can be simplified with a build System.

Comment: Do you use Build Systems in your projects? And what tooling you found interesting.

Photo by Ryan Quintal on Unsplash — Projects without Build Systems

What is a Build System?

A build system is a program which helps convert your source code into any form of meaningful output that you want to deploy. To make this more generic, it could even work for something that’s not even code. For example, this output could be a PDF for a book, a group of files as a zip or a tarball, an npm package, a docker image, a helm chart to deploy said docker image; anything under the Sun really, as long as a bunch of inputs are properly transformed into a meaningful output.

Apart from this, a Build System also helps automate many other tasks for your project – running your app locally, linting or testing your code, generating a documentation site from a set of static markdown files, periodically cleaning up left-out branches from your git repository, etc.,.

Why do you need a Build System?

As the process of converting your source code to these outputs could be wildly different, a build system hides away these complex processes with a unified interface at the front, so the developers and the Continuous Integration system (Github Actions, Gitlab, Azure DevOps or what have you) can benefit from this common interface.

A build system would also help with, at least the following (not an exhaustive list) —

• Dev workflow tasks – linting, testing and building the (various parts of the) project
• Caching previous results for all tasks to prevent recalculation and save time.
• Running each task in a sandboxed environment to prevent accidental dependencies on the host machine where it is run. These are called hermetic builds.
• Being CI/CD tooling neutral (this is debatable as there might be some integrations that your team uses)
• Having language specific integrations – the build system should provide good integrations with the programming language the project is written in. This is to help configure the tasks easily.
• Have a general purpose specification – if the tasks to be written are not available as part of the repository of pre defined tasks, we should be able to setup these custom tasks with the language of the build system.
• Reuse of any these custom tasks, across different packages in a project or across many projects in an organization.

The ‘Paper’ Project – Life without a Build System 🥲

Let’s illustrate how we evolve from a project without a Build System and see how having one can help us.

Note: The source code can be found at the repo for this blog – blog-build-system. Feel free to clone it and give it a spin, and a star too. 😉

The simplest build system would be a shell script. The reason for this being every Operating System already has one. The shell can be chosen based on the team – bash works pretty well with Linux and macOS and now WSL on Windows (even though some commands have slightly different switches) or PowerShell — a cross OS shell.

Here’s a simplified structure to demonstrate how a Build system can be used in a project. It’s a project to create a PDF file from markdown. It’s basically a document processor, but built for code enthusiasts.

The project structure looks like the following…

❯ tree
.
├── .gitignore
├── build.sh
├── readme.md
└── readme.pdf

• To build the document, we render the markdown file and write the page to a PDF file.
• To test the document, we make sure we have all the necessary sections are in place. But as you can imagine, you can add tests to make sure that the document has no spelling errors, things like that.

A basic build.sh script for this project would be –

#!/usr/bin/env bash

case $1 in

test)
  echo add your tests here – for example a spellcheck
  ;;

build)
  docker run  – rm  – volume "`pwd`:/data"  – user `id -u`:`id -g` pandoc/latex:2.17-alpine readme.md -o readme.pdf
  ;;
esac

We have to handle exit codes, have the right set of options, and if your project gets complex we have to have a way of organizing various parts of the project.

Examples of some Build Systems

Just like everything, there are a ton of options available here, but my suggestion would be to keep things as general as possible. What are some of the options available?

• make: If you look at the chronology of things, this seems to be one of the most popular and well known build systems. It has some learning curve, but knowing sh or bash really helps. Doesn’t have any way of organizing multiple make files; it was built for single app repositories.
• Bazel by Google: Really popular and if things are true on the Web, their Build System for internal codebases too. It has a pythonic syntax and one of the most popular Build Systems. Has quite a few integrations and lots of community written plugins. Steep learning curve.
• Please.Build by Thought Machine: A dark horse really, I couldn’t find a whole lot of articles about Please on the Web. This also has a learning curve, but their code labs are much better compared to Bazel. For me, this looked like the right choice.

Microsoft’s Lage, Lerna, Pants are some of the other options. But there a lot more which solve specific problems.

Which Build System did I choose?

In this section I’ll list how I came to choose Thought Machine’s Please.Build as my go-to Build System going forward. I’ve been using Please.Build for a while, but if enough excitement is there, I can take a look at other Build Systems.

Reasons for choosing Please —

• Uses similar syntax and concepts as Bazel (used in many open source projects)
• Has better integration with Docker compared to Bazel.
• Easy learning as it uses Starlark which has a pythonic syntax.
• Great monorepo and cross language support.
• It wasn’t there before, but please now also has multi-arch support and now works on ARM. I have tested it on my Raspberry Pi.

Reasons why you might avoid Please.Build —

• Not a whole lot of discussions on the Web; so debugging might get a little tough.
• No support for Helm at the moment, but general rules can be created to overcome this (there’s an open Pull Request on Thought Machine’s pleasings repo, which is a repository to add extra features to Please, and held up by the community)
• No support for JavaScript/TypeScript projects, but again, general rules work fairly well and most of the projects I’ve used this for uses Docker, so it should be fine.

Creating a Please Project

Installation

If you’re on macOS/Linux (yes brew works on Linux 🤩), you can use brew to install please. Skip to the next step if you want to see an alternative way to do this.

❯ brew tap thought-machine/please
❯ brew install please
# works on both macOS and Linux

Initialize a Please.Build project

You’ll see that even if you don’t have Please installed, it’s easy to set it up with the `pleasew` wrapper. So your team members and even in CI, you can easily setup Please with the right version.

❯ plz init 
Wrote config template to ~/code/blog-build-system/please/.plzconfig, you're now ready to go!

Also wrote wrapper script to pleasew; users can invoke that directly to run Please, even without it installed.

Pleasings are a collection of auxiliary build rules that support other languages and technologies not present in the core please distribution.
For more information visit https://github.com/thought-machine/pleasings

❯ tree -a
.
├── .plzconfig
└── pleasew

As you can see .plzconfig and pleasew got created. .plzconfig is a file to configure Please and pass through environment variables to the Build System. As the builds are hermetic by default, environment variables aren’t passed through to the Build Agent.

1. Let’s write a hello world build target. We can write build targets in BUILD or BUILD.plz files.

❯ cat BUILD
sh_cmd(
  name="hello-world",
  cmd="echo hello world"
)

To run this use plz run /:hello-world– the syntax might look clunky, but it will make sense. Because we can have different packages in the same project, you can address any build target in the format /<path-from-root:build-target-name> which becomes /:hello-world in our case. The path is addressed from root, the project root, that is. If you’re in the same directory as the BUILD file, you can also run plz run :hello-world

❯ plz run /:hello-world
hello world

That was pretty simple, but a lot of things just happened. The run commands does two basic things in the background. First, a plz build /:hello-world ran, which generated the actual script which is executed for our sh_cmd rule. This is located at plz-out/bin/hello-world.sh.

From this point on, the script will not be generated again, until our source code changes, it is cached. The same is true if your build creates artefacts. This is really powerful, as even in CI, once your cache the plz-out directory, all your build cache will magically light up.

Checkout all the details at please.build/codelabs.html through their CodeLabs.

As this is a single app repository (not a monorepo), we created a BUILD file at the root of the repository. If you have a monorepo, each BUILD file creates an isolated package in your project.

Please.Build Terminologies

Most of the Build systems share these terminologies to address various parts of the project.

• rule: A rule is any function available either in Please natively or brought in through a third party project to help with any task. An example is the github_repo rule, which can be used to bring in a Github repository to use files from that repo as if they were in the current project.
• package: A package is any directory which has a BUILD or BUILD.plz file. A package is an isolated entity, which helps maintain hermeticity, but it can use items from other packages via dependencies, deps attribute in please.
• target: In each package, we can have multiple targets, which does specific things for that package. Example — a nodejs service package can have lint, test, build, dockerize, code_scan and deploy targets. (The target name was hello-world in the example we wrote earlier)

The ‘Paper’ Project – with Please.Build a.k.a plz

To use a Build System in our project, we can write a custom rule to render the page using pandoc. Now we have many ways to do this, but I always try to use Docker.

There are three main verbs in please — build, run and test (but there are more) — one to build a certain artefact; and another to run (if the artefact has that option) and the third to run tests on the project. I had created the bash script above to have similar verb names to make the transition simpler, and also easy to update in CI.

We are going to add a simple BUILD configuration to the Paper Project.

sh_cmd(
  name = "app",
  srcs = glob("readme.md"),
  # note the extra backslash before backtick, it is required to prevent 
  # substitution when please.build generates the script
  cmd = 'docker run  --rm  --volume "\`pwd\`:/data" pandoc/latex:2.17-alpine /data/readme.md -o readme.pdf'
)

gentest(
  name = "app_test",
  test_cmd = "echo add your tests here - for example a spellcheck",
  no_test_output = True
)

Well it’s as simple as running ./pleasew run :app or plz run :app if you have please installed to create the same PDF file. Because we are using docker, we can ignore please’s hermeticity. The sh_cmd will run the command directly in the directory where the BUILD file is. We can also run plz test which runs all test targets in the project (we have only one with a simple echo; try returning a non-zero exit code and see what happens if tests fail).

Summary

A build system is a tool to help convert some inputs to an output and create standards along the way. Apart from this, a typical build system also has dependency calculations, hermetic builds and cache management.

In simple terms, writing build configurations is like creating a Design System, but for building the project — the power of the Build System might not be immediately apparent in a simple example, but they will start lighting up as soon as your deployments start getting complex, and introducing this brings in a little order to the chaos.

You can see a more realistic and live project mrsauravsahu/payobills on Github to see how easy it gets, to manage a large project.

In the next part I’ll show specific examples of how I used Please in my projects.

– Sahu, S