Jump to content

Reproducible builds and trust in free software

From IdeaWazaWiki

Why Software Anyone Can Rebuild Bit for Bit Is Harder to Secretly Backdoor

Most people who install open source software never compile it. They download a finished program built on someone else's computer and trust that it matches the public source code. Reproducible builds are a way to check that trust instead of assuming it. This page treats them as freedom tech and as a step from scarcity toward abundance: today, trustworthy software depends on a scarce resource, a small number of build servers and signing keys that attackers can target. Reproducible builds turn verification into something abundant, because any volunteer with an ordinary computer can rebuild a program and confirm that the published binary matches its source. The original angle here is that "open source" by itself protects users far less than most people think, while "rebuildable by anyone" protects them far more.

The gap between source code and the program you run

Source code is what humans read. The program a computer runs is a binary, produced from that source by compilers and build tools. When a project publishes both, there is a quiet gap: almost nobody checks that the binary really came from the published source. A backdoor placed in the binary, or inserted during the build, can sit in plain sight while the public source looks clean.

Ken Thompson described the deepest version of this problem in his 1984 Turing Award lecture, "Reflections on Trusting Trust." He showed how a compiler could be altered to insert a hidden password into the Unix login program, and then altered again so that it reinserted both tricks whenever it compiled a fresh copy of itself. After that, the malicious lines could be deleted from the compiler source, and the trap would survive with no trace in any source file. His stated moral was that no amount of source-level verification or scrutiny will protect you from using untrusted code.

What a reproducible build is

The Reproducible Builds project gives a precise definition: a build is reproducible if, given the same source code, build environment, and build instructions, any party can recreate bit-by-bit identical copies of the specified artifacts. Verification is done by comparing cryptographic hashes. If two strangers on two continents build the same release and get the same hash as the official download, an attacker would have needed to compromise all three builds in the same way.

Getting there is hard work. Builds often pick up timestamps, file ordering, user names, and machine paths, so two honest builds can differ for no sinister reason. Removing that noise is mostly patient engineering, not a breakthrough.

Three stories that show why it matters

In 2013, the Tor Project rebuilt its browser bundle so that it came out byte for byte identical no matter who built it or on what hardware. The developer behind that effort wrote that attacks on build processes could deliver malware to tens or hundreds of millions of machines in a single officially signed update, and that he no longer believed it was possible to keep signing keys or even an offline build machine secure against the adversaries Tor faces. His answer was to let any individual download the source privately, check it against public repositories, reproduce the build exactly, and raise the alarm if anything differed.

In 2020, that warning came true in a commercial product. CrowdStrike's analysis of the SolarWinds attack describes a tool called SUNSPOT, planted on a build server, that watched for compilation of the Orion product and swapped one source file for a backdoored version during the build, with safeguards to avoid breaking the build and alerting developers. The published source and the shipped program no longer matched. Orion was closed source, so outsiders could not rebuild it. That is the point: an independent rebuild from clean source, on a different machine, is exactly the kind of check that can expose this sort of tampering, but only when the source is available and the build is reproducible.

In March 2024, a developer noticed that ssh logins on a test system were using too much processor time and traced the cause to a backdoor in the xz compression library, later tracked as CVE-2024-3094. His public report explains that part of the attack lived only in the release tarballs and not in the project's version control history, while the rest hid inside binary test files in the repository. This case is an honest limit on the idea. A reproducible build of a poisoned tarball faithfully reproduces the poison. What would have flagged the gap is comparing the release tarball with the version control source, and building from that source directly.

Trusting the compiler itself

Thompson's attack seemed impossible to beat for decades, since checking a compiler requires another compiler. David A. Wheeler's 2009 dissertation proposed a counter called Diverse Double-Compiling. The compiler's source is compiled with a second, independent compiler, and the result is used to compile the compiler source again. If the final output is bit for bit identical to the suspect binary, the source accurately represents it. He demonstrated the method on several compilers, including GCC. The technique depends on reproducibility, because the comparison only works if honest builds come out identical.

The GNU Guix project pushed in a related direction. In April 2023 it announced a "Full-Source Bootstrap": a package graph of more than 22,000 nodes rooted in a 357-byte starting program, with each later tool built from readable source on top of that seed. The team wrote that, if one should always be able to build software from source, then the trusting trust attack is a symptom of a missing bootstrap story.

How far real systems have come

Debian runs a public rebuilder that tries to recreate each binary package in its archive using the record of the original build. In mid November 2026, its amd64 dashboard reported about 95,500 rebuilds that matched and about 3,100 that did not, with some packages still untested. The project page says it wants independent rebuilders in a diverse variety of setups, so that verification does not rest on one organization. The Guix team credits the Bitcoin community as one of the first to put bootstrappable builds into practice, which is fitting for software that holds money.

Why this is an abundance story

Trust in software has usually been rationed. Users trust a vendor's reputation, a signing key, or a security audit, and each of those is scarce, expensive, and attractive to attackers. Reproducible builds change the economics. Checking a release costs some electricity and a spare computer. Volunteers can run rebuilders the way others run Tor relays or Snowflake proxies, and each additional independent rebuilder makes a silent attack more expensive. For people who depend on freedom tech, such as journalists, activists, and anyone using privacy tools in a hostile country, this is a concrete defense against the most dangerous kind of attack: a targeted, signed, official update.

Honest limits

  • Reproducibility proves that a binary matches its source. It says nothing about whether the source itself is honest, as the xz case shows.
  • Release tarballs, generated files, and binary test data can hide changes that a casual reader of the repository never sees.
  • Verification only helps if people actually run independent rebuilders and someone acts on a mismatch.
  • Closed source software cannot be checked this way by outsiders, so the benefit is limited to projects that publish their source.
  • Hardware and firmware sit below the compiler. Thompson himself noted that microcode bugs would be almost impossible to detect.
  • Why does publishing source code protect users less than most people assume, and what does a reproducible build add?
  • Explain Thompson's trusting trust attack to a reader with no programming background, using a non-computer analogy.
  • Compare the SolarWinds and xz cases and decide which one reproducible builds would have helped with more, and why.
  • How could a library, university, or credit union run an independent rebuilder as a public service, and what would it need?
  • Explore whether verification by many volunteers is a better model of trust than certification by a few auditors.
  • What would it take for app stores and phone operating systems to show users whether an app was independently reproduced?
  • Discuss how the idea of a full-source bootstrap from a tiny seed program connects to long-term goals for software that no single company controls.

See also

Sources