Reproducible builds, part 2

Introduction

I’m looking to be able to effectively audit builds related to Ubuntu that are built using Launchpad. Launchpad is the web application and server software the Ubuntu uses for:

• code hosting
• project tracking
• bug tracking/managemnt
• code builds and packaging
• other stuff?

The use case is an Ubuntu user who would like to confirm the built packages that Launchpad produces by replicating the build locally and comparing the output executables.

Personal background

I’m relatively new to the build and packaging game and have a cursory knowledge of the tools and processes, but before diving into Launchpad I want to write down what I do know about it, especially the Debian and Ubuntu processes and tools, so I have something to refer back to.

Prior to this year my experience with building was twofold, one was writing C code and using make, the other was building gcc, perl, emacs, and other open source software before the age of binary distributions.

I wrote batch-oriented scientific code in the early 90s while doing research for my Ph.D. This was at first using the IBM xlC¹ compiler on AIX². AIX used the XCOFF³ executable file format. The structure of an executable file is important to understand when doing practical analysis of reproducible builds because if you haven’t got to the point of producing bitwise identicle outputs you can still distinguish between trivial differences and significant differences within the sections of the executable file.

Later I wrote batch-oriented C code that used Posix threads⁴ to allow for parallel execution in a single memory space on a COMA⁵ machine, the KSR-1⁶, in contrast to similar early experimental NUMA⁷ machines. Other users of the machine were trying to research algorithms that would work with such an architecture while I was just trying to calculate solutions to physics problems using numerical methods and trying to optimize what I could.

In parallel I did have to learn about open source building by learning to use the early versions of what is now GNU Autotools⁸. Just understanding the concept of the following sequence:

1. n=1, compile gcc_n=gcc₁ using xlC
2. Compile gcc_(n+1) using gcc_n
3. If gcc_(n+1) == gcc_n, install gcc_n, else n++ and loop back to step 2.

is a fundamental example of the need for reproducable builds.

In any event my experience was rather limited to using make and for builds and learning the XCOFF format.

Note about debbuild

My first Debian-style build was using the basics, debbuild. I’m going to just bullet-point a bunch of stuff I learned without trying to go deep into this since I want to, like I’ve said, not be precious about being complete in the short term in favour of getting stuff out of my head.

What I recall:

• Debian makes effort to start with a pristine source code tarball
  • pristine means it is a tar of the source code build environment provided by the upstream developers
    • if the upstream developers provide an actual tarball, that is what is preferred, since then it can be directly validated
    • otherwise a tarball of, say, the upstream repository tree (minus repository metadata) as of a particular tag is used, since the individual files can be directly validated
  • in fact, I think the goal, in principle at least, is to be able to associate a particular tarball from upstream, identified not only by name or source but by hash, with the associated Debian source package version
• the Debian build process is to then take a separately-maintained packaging tarball of a directory named “debian” that is intended to be extracted inside of the extracted source build tree of the upstream code
• anyone can then extract both the upstream tarball into a build directory and then extract the Debian build definition tarball into the top level of that build directory and then use the Debian build tools (debbuild) to build either the source or executable debian package
• the Debian tools use quilt⁹ to apply a series (pun intended) of localized patches to that pristine source prior to build
  • obvious the goal of Debian is to feed patches to the upstream community, but these patches might be ones that have not yet been accepted or, more likely, simply customisations of the configuration of the build process to suite the Debian distribution
• Ubuntu builds on this generally by maintained a series of patches on top of the Debian ones with the same goal of moving any appropriate patches to upstream first or to Debian second
• The Debian tooling includes all sorts of capabilities like a change log, signed builds of both source and executable packages, and so on

Note about schroot

I’ve already done a bit of that investigative work by doing builds of a couple Ubuntu packages (qemu-* and libvirt-*) using the schroot tool. But I still don’t have a good understanding of the layering of tools from the low-level Makefiles and compilers to the debian/* subdirectory and associated build tools (debbuild, …?) up to schroot, sbuild, and https://launchpad.net. I mean, I have a hand-wavy understanding but not a detailed understanding, and that makes me feel completely unmoored.

Note about sbuild

I have yet to really investigate sbuild other than seeing it in part of the Launchpad installation and setup and seeing that it does relate to and possibly use schroot.

Launchpad

I’ve installed Launchpad using these resources but have not got far into it. I was going to review that here, today, but I got bogged down in the above. I will continue tomorrow, hopefully.

Article series

• Part 1
• Part 2 (this article)
• Part 3