<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Compilers on Random Stuff - Too Many Machines</title><link>https://random.too-many-machines.com/tags/compilers/</link><description>Recent content in Compilers on Random Stuff - Too Many Machines</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>© Igor Wawrzyniak</copyright><lastBuildDate>Fri, 20 Nov 2026 19:00:00 +0100</lastBuildDate><atom:link href="https://random.too-many-machines.com/tags/compilers/index.xml" rel="self" type="application/rss+xml"/><item><title>Reproducible builds - can you trust your distro's binaries?</title><link>https://random.too-many-machines.com/posts/reproducible-builds/</link><pubDate>Fri, 20 Nov 2026 19:00:00 +0100</pubDate><guid>https://random.too-many-machines.com/posts/reproducible-builds/</guid><description>&lt;p&gt;Open source means you can read the code but almost nobody does. And even if you do, how can you be sure that the code you&amp;rsquo;re running is the same? Compiling the whole operating system with libraries and applications directly from source is possible - that was the whole point, after all - and some people do it for education. But it&amp;rsquo;s not a practical way of keeping a fleet of servers updated. We need a way to prove the provenance of the package.&lt;/p&gt;
&lt;p&gt;Package signatures are one step - and an important one - but they only prove the binary came from the distribution. They don&amp;rsquo;t prove it was built from the source it claims to be built from. That&amp;rsquo;s the gap reproducible builds are trying to close.&lt;/p&gt;
&lt;p&gt;A build is reproducible if anyone who compiles the same source, with the same tools and parameters, gets a &lt;strong&gt;bit-for-bit identical&lt;/strong&gt; result. Then anyone can verify the build server: rebuild the package independently and compare the checksums. If they match, the binary really comes from that source.&lt;/p&gt;
&lt;p&gt;By default, the builds are not reproducible. Compilers and build systems embed timestamps, paths, usernames, file ordering and random keys into their output. Debian, Arch and Fedora now build the vast majority of their packages reproducibly. That took a lot of effort over the years.&lt;/p&gt;
&lt;h2 id="how-it-began-trusting-trust"&gt;How it began: Trusting Trust
&lt;/h2&gt;&lt;p&gt;In 1983 Ken Thompson gave his Turing Award lecture, &amp;ldquo;Reflections on Trusting Trust&amp;rdquo;. He described a compiler modified to do two things: insert a backdoor when compiling the &lt;code&gt;login&lt;/code&gt; program, and insert its own modification when compiling the compiler itself. Once you have such a compiler binary, you can remove the malicious code from the compiler source so there&amp;rsquo;s no trace - but every compiler built from clean source with the infected binary is infected too.&lt;/p&gt;
&lt;p&gt;That&amp;rsquo;s an extreme example of a supply chain attack. Is it possible in the real world? Ken Thompson created a proof of concept, which worked (ironically, the backdoor made the build non-reproducible in an unusual way: the compiler grew by one byte each time it compiled itself). There were similar cases in the wild: Induc virus infected Delphi compilers in 2009, and XCodeGhost in 2015 targeted, you guessed it, Xcode.&lt;/p&gt;
&lt;p&gt;Moving away from compilers: build servers, used (among others) by Linux distros, are a very attractive target. If you compromise one machine, your malware will be shipped to millions of users, carrying a valid signature and coming from a trusted source.&lt;/p&gt;
&lt;h2 id="why-arent-builds-reproducible-by-default"&gt;Why aren&amp;rsquo;t builds reproducible by default?
&lt;/h2&gt;&lt;h3 id="timestamps"&gt;Timestamps
&lt;/h3&gt;&lt;p&gt;Let&amp;rsquo;s try a simple example:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-c" data-lang="c"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&lt;span style="color:#75715e"&gt;#include&lt;/span&gt; &lt;span style="color:#75715e"&gt;&amp;lt;stdio.h&amp;gt;&lt;/span&gt;&lt;span style="color:#75715e"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&lt;span style="color:#66d9ef"&gt;int&lt;/span&gt; &lt;span style="color:#a6e22e"&gt;main&lt;/span&gt;(&lt;span style="color:#66d9ef"&gt;void&lt;/span&gt;) {
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; &lt;span style="color:#a6e22e"&gt;printf&lt;/span&gt;(&lt;span style="color:#e6db74"&gt;&amp;#34;Hello, built on %s %s&lt;/span&gt;&lt;span style="color:#ae81ff"&gt;\n&lt;/span&gt;&lt;span style="color:#e6db74"&gt;&amp;#34;&lt;/span&gt;, __DATE__, __TIME__);
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; &lt;span style="color:#66d9ef"&gt;return&lt;/span&gt; &lt;span style="color:#ae81ff"&gt;0&lt;/span&gt;;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;}
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Compile it twice, a few seconds apart:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ gcc -O2 -o hello1 hello.c
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ gcc -O2 -o hello2 hello.c
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ sha256sum hello1 hello2
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;e095fb1f0e1f44220468876b93075a4985c21f0a9f32c17c917a75f3a824246f hello1
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;0b3715369f9bcd163e85e0ff55b746301893f3ed264ec54d5b6df97b7cab0f33 hello2
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Different. Obviously, that&amp;rsquo;s because the program embeds the build time. Do you think I cheated, no real software does that? Quite the contrary, many popular apps have it in their version string. I bet you have at least two of those:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;igor@ood:/$ vim --version | head -n &lt;span style="color:#ae81ff"&gt;1&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;VIM - Vi IMproved 9.2 &lt;span style="color:#f92672"&gt;(&lt;/span&gt;&lt;span style="color:#ae81ff"&gt;2026&lt;/span&gt; Feb 14, compiled Jul &lt;span style="color:#ae81ff"&gt;27&lt;/span&gt; &lt;span style="color:#ae81ff"&gt;2026&lt;/span&gt; 10:11:19&lt;span style="color:#f92672"&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;igor@ood:/$ uname -v
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&lt;span style="color:#75715e"&gt;#1 SMP PREEMPT_DYNAMIC Debian 7.2.15-1 (2026-11-01)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;igor@ood:/$ python3 -VV
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Python 3.14.10 &lt;span style="color:#f92672"&gt;(&lt;/span&gt;main, Oct &lt;span style="color:#ae81ff"&gt;26&lt;/span&gt; 2026, 15:12:11&lt;span style="color:#f92672"&gt;)&lt;/span&gt; &lt;span style="color:#f92672"&gt;[&lt;/span&gt;GCC 16.2.0&lt;span style="color:#f92672"&gt;]&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;And the timestamps aren&amp;rsquo;t only there if the source code explicitly references them. A real app is not just a single C file that turns into a single binary. A deb or rpm package contains files, and they all have modification time. Worse, those files might be archives (e.g. jar) that contain more files.&lt;/p&gt;
&lt;h3 id="other-usual-suspects"&gt;Other usual suspects
&lt;/h3&gt;&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Build paths.&lt;/strong&gt; Debug information records the directory the code was compiled in. Build the same thing in &lt;code&gt;/home/alice/src&lt;/code&gt; and &lt;code&gt;/build/pkg-1234&lt;/code&gt; and you get different binaries.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;File ordering.&lt;/strong&gt; If a build script does the equivalent of &lt;code&gt;for f in *.o&lt;/code&gt; without sorting, the link order (and therefore the binary) can be different in every case.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Environment.&lt;/strong&gt; Locale, timezone, umask, hostname, username.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Randomness.&lt;/strong&gt; Hash table seeds, temporary file names that end up in the output, and the big one: &lt;strong&gt;randomly generated signing keys&lt;/strong&gt;. If you followed my &lt;a class="link" href="../compiling-linux-kernel/" &gt;kernel compiling post&lt;/a&gt;, the build generated a fresh module signing key, so every build of the same kernel is different. Not a bug, that&amp;rsquo;s the whole point of a throwaway key - but it does break reproducibility.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Parallelism.&lt;/strong&gt; Parallel builds, e.g. &lt;code&gt;make -j16&lt;/code&gt;, can finish things in a different order, and it can affect the binary.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="how-its-fixed"&gt;How it&amp;rsquo;s fixed
&lt;/h2&gt;&lt;p&gt;The Reproducible Builds project started in Debian around 2013 (Bitcoin had been doing deterministic builds from 2011 and Tor Browser started a bit earlier in 2013) and grew into a cross-distribution effort: Arch and a few others distros were next. Fedora joined later and is catching up.&lt;/p&gt;
&lt;h3 id="buildinfo-buildinfo-and-koji-buildinfo"&gt;.buildinfo, .BUILDINFO and &lt;code&gt;koji buildinfo&lt;/code&gt;
&lt;/h3&gt;&lt;p&gt;To reproduce the build, you need to use the same versions of the compiler, linker, documentation generators and other tools. And then you need to have the same path and environment variables. So the first step is writing down what the build environment actually was. Debian&amp;rsquo;s standard is the &lt;code&gt;.buildinfo&lt;/code&gt; file. It records all these plus the checksums of the resulting binaries and it&amp;rsquo;s published alongside the packages. Arch uses a &lt;code&gt;.BUILDINFO&lt;/code&gt; file inside every package. Despite the similar name, it&amp;rsquo;s a different format.&lt;/p&gt;
&lt;p&gt;Fedora has a different approach - its build tool &lt;code&gt;koji&lt;/code&gt; can query the build server to read the environment. If you rebuild a package with &lt;code&gt;fedora-repro-build&lt;/code&gt;, it will happen automatically, so the end result will have the same checksum.&lt;/p&gt;
&lt;p&gt;Some distros have rebuild servers in addition to the usual build servers. They continuously rebuild packages and would alert the developers if the checksum differed. But you can also get the environment information and do the same process yourself. It&amp;rsquo;s important for creating trust.&lt;/p&gt;
&lt;h3 id="build-time"&gt;Build time
&lt;/h3&gt;&lt;p&gt;Timestamps are the most important sources of non-reproducibility. The fix is the &lt;strong&gt;&lt;code&gt;SOURCE_DATE_EPOCH&lt;/code&gt;&lt;/strong&gt; environment variable, containing a Unix timestamp, that tells all build tools &amp;ldquo;pretend this is the current time&amp;rdquo;. GCC and Clang use it for &lt;code&gt;__DATE__&lt;/code&gt; and &lt;code&gt;__TIME__&lt;/code&gt;. Which means that if an app chooses to embed a build time in its version string, it will work - the time won&amp;rsquo;t be real, but who cares anyway? Debian and Fedora set the SOURCE_DATE_EPOCH to the date of the last changelog entry, so it&amp;rsquo;s fixed for a given source package. A long list of other tools (tar, zip, Sphinx, gzip wrappers, documentation generators) were patched to respect it too.&lt;/p&gt;
&lt;p&gt;Let&amp;rsquo;s test it on the same C file.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ export SOURCE_DATE_EPOCH&lt;span style="color:#f92672"&gt;=&lt;/span&gt;&lt;span style="color:#ae81ff"&gt;1700000000&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ gcc -O2 -o hello3 hello.c
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ gcc -O2 -o hello4 hello.c
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ sha256sum hello3 hello4
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;a8ad13ca37c3242c42466b3ac39da35e5f66423ba2dc9145d8cbfcac639bf30e hello3
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;a8ad13ca37c3242c42466b3ac39da35e5f66423ba2dc9145d8cbfcac639bf30e hello4
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ ./hello3
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;Hello, built on Nov &lt;span style="color:#ae81ff"&gt;14&lt;/span&gt; &lt;span style="color:#ae81ff"&gt;2023&lt;/span&gt; 22:13:20
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h3 id="build-path"&gt;Build path
&lt;/h3&gt;&lt;p&gt;Both GCC and Clang have a parameter &lt;strong&gt;&lt;code&gt;-ffile-prefix-map&lt;/code&gt;&lt;/strong&gt; to map directory paths. It was added specifically to make builds reproducible regardless of the current path. Let&amp;rsquo;s do the experiment again:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ &lt;span style="color:#f92672"&gt;(&lt;/span&gt;cd a &lt;span style="color:#f92672"&gt;&amp;amp;&amp;amp;&lt;/span&gt; gcc -g -O2 -o ../h1 $PWD/hello.c&lt;span style="color:#f92672"&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ &lt;span style="color:#f92672"&gt;(&lt;/span&gt;cd b &lt;span style="color:#f92672"&gt;&amp;amp;&amp;amp;&lt;/span&gt; gcc -g -O2 -o ../h2 $PWD/hello.c&lt;span style="color:#f92672"&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ sha256sum h1 h2
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;35564b676c8a1e0ea85d78b2a298912c3db87abf98847be1a6701acd7827769e h1
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;3300ea4ea1dcfba4ad1c7661895b0fff6a6f3aa117dbb3bea3d2cb6f213f114d h2
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ &lt;span style="color:#f92672"&gt;(&lt;/span&gt;cd a &lt;span style="color:#f92672"&gt;&amp;amp;&amp;amp;&lt;/span&gt; gcc -g -O2 -ffile-prefix-map&lt;span style="color:#f92672"&gt;=&lt;/span&gt;$PWD&lt;span style="color:#f92672"&gt;=&lt;/span&gt;. -o ../h1 $PWD/hello.c&lt;span style="color:#f92672"&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ &lt;span style="color:#f92672"&gt;(&lt;/span&gt;cd b &lt;span style="color:#f92672"&gt;&amp;amp;&amp;amp;&lt;/span&gt; gcc -g -O2 -ffile-prefix-map&lt;span style="color:#f92672"&gt;=&lt;/span&gt;$PWD&lt;span style="color:#f92672"&gt;=&lt;/span&gt;. -o ../h2 $PWD/hello.c&lt;span style="color:#f92672"&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;$ sha256sum h1 h2
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;632b8ebaaa01e74390f11f2d6277de330bd343caa8e7d8eced1512ce5f4c070d h1
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;632b8ebaaa01e74390f11f2d6277de330bd343caa8e7d8eced1512ce5f4c070d h2
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h3 id="special-case-the-linux-kernel"&gt;Special case: the Linux kernel
&lt;/h3&gt;&lt;p&gt;The Linux kernel embeds a few things about the environment: build time, username and hostname. It was a rational thing to do in the 1990s when people often compiled custom kernels: easier to tell which one you&amp;rsquo;re running if you had several to choose from, and if something didn&amp;rsquo;t work, you knew which admin to blame. Older Unix systems that Linux imitated did the same and for the same reasons. Today, when almost everyone uses a distribution&amp;rsquo;s kernel, it&amp;rsquo;s not needed anymore and of course it would totally break reproducibility. But all of these can be configured with environment variables: &lt;code&gt;KBUILD_BUILD_TIMESTAMP&lt;/code&gt;, &lt;code&gt;KBUILD_BUILD_USER&lt;/code&gt; and &lt;code&gt;KBUILD_BUILD_HOST&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Another problem is module signing. For the reproducible build, you need to provide a pre-generated key instead of letting the build create a random one - and then keep it secret, because anyone with the key can produce the same signed binary. That&amp;rsquo;s an interesting conflict: a signature is by definition something only one party can produce, so signed artefacts are verified by stripping or ignoring the signature and comparing the rest.&lt;/p&gt;
&lt;h3 id="everything-else"&gt;Everything else
&lt;/h3&gt;&lt;p&gt;For other problems there&amp;rsquo;s &lt;code&gt;strip-nondeterminism&lt;/code&gt; (Debian) or &lt;code&gt;add-determinism&lt;/code&gt; (Fedora): post-processing tools that normalise timestamps and ordering in archives after the build. And if that fails - patching of build scripts. Hundreds of upstream projects received such patches.&lt;/p&gt;
&lt;h3 id="tools-for-checking"&gt;Tools for checking
&lt;/h3&gt;&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;diffoscope&lt;/strong&gt; - &amp;ldquo;diff on steroids&amp;rdquo;. Give it two files (rpm/deb packages, ISOs, archives, binaries) and it recursively unpacks them and shows you &lt;em&gt;where&lt;/em&gt; they differ. Even if it means finding a specific file in a zip inside a jar inside a deb. Even if you don&amp;rsquo;t care about reproducible builds, it&amp;rsquo;s a great tool for other uses.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;reprotest&lt;/strong&gt; - builds the same thing twice while deliberately varying the environment (time, path, locale, user, timezone, file order, umask&amp;hellip;) to check for hidden dependencies.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;debrebuild&lt;/strong&gt; (Debian) and &lt;strong&gt;makerepropkg&lt;/strong&gt; (Arch) - rebuild an official package locally with exactly the same build environment as the original one, using the .buildinfo/.BUILDINFO file&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;koji&lt;/strong&gt; - a client for Fedora build system&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;rebuilderd&lt;/strong&gt; - an rebuilder software used by some distros, which continuously rebuilds packages and publishes the results.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="where-are-we-now"&gt;Where are we now?
&lt;/h2&gt;&lt;p&gt;Debian&amp;rsquo;s continuous testing at tests.reproducible-builds.org shows about 96% of packages in testing building reproducibly - and that&amp;rsquo;s while deliberately varying the environment between the two builds, which is harsher than a real-world rebuild. As is often the case, achieving the first 90% was easy, while the rest was the struggle. Arch Linux is behind with 74% (though the core repository is 93%, extras are the problem - and many of those are not non-reproducible, just not tested yet).&lt;/p&gt;
&lt;p&gt;Fedora reached about 90% reproducible package builds mostly with infrastructure changes. They initially aimed to achieve 99% in Fedora 43, but later moved the target to Fedora 46. RHEL is built from Fedora, so it benefits from the same work, but Red Hat doesn&amp;rsquo;t publish any reproducibility numbers for it. Ubuntu doesn&amp;rsquo;t publish any either: packages synced unchanged from Debian benefit from Debian&amp;rsquo;s fixes, but nobody continuously rebuilds Ubuntu to check, and Launchpad only started publishing the &lt;code&gt;.buildinfo&lt;/code&gt; files needed for independent rebuilds in 2024.&lt;/p&gt;
&lt;p&gt;NixOS and Guix get it largely for free from their design, since their builds already run in isolated, pinned environments; NixOS announced as early as 2021 that its minimal installation ISO was reproducible. Guix took it a step further with &lt;strong&gt;bootstrappable builds&lt;/strong&gt;: the whole distribution can be built starting from a tiny binary seed of a few hundred bytes, small enough to audit by hand (you don&amp;rsquo;t even need to trust a pre-built GCC). That&amp;rsquo;s the real answer to Trusting Trust.&lt;/p&gt;
&lt;h2 id="the-xz-backdoor-and-the-important-next-step"&gt;The xz backdoor and the important next step
&lt;/h2&gt;&lt;p&gt;In March 2024 Andres Freund noticed that SSH logins on his Debian sid machine were taking half a second longer than they should and found a backdoor in xz/liblzma (CVE-2024-3094). After years of patient social engineering, the attacker became a maintainer and slipped the payload in through &amp;ldquo;binary test files&amp;rdquo; plus a modified build script.&lt;/p&gt;
&lt;p&gt;Here&amp;rsquo;s the catch: the modified build script (&lt;code&gt;build-to-host.m4&lt;/code&gt;) wasn&amp;rsquo;t in the git repository. It was only in the &lt;strong&gt;release tarball&lt;/strong&gt;. Build servers commonly get the source from release files, while individual users often get it directly from git. Reproducible builds verify that the binary matches the source - and it did. The source that was used was the problem.&lt;/p&gt;
&lt;p&gt;What would have helped is comparing the tarball with the git tag, which is now a hot topic - some distributions are moving towards building from git instead of from tarballs, and reproducible builds people are working on making release tarballs themselves reproducible from the repository.&lt;/p&gt;
&lt;h2 id="what-it-doesnt-protect-from"&gt;What it doesn&amp;rsquo;t protect from
&lt;/h2&gt;&lt;p&gt;Having reproducible builds doesn&amp;rsquo;t automatically mean that it&amp;rsquo;s impossible to add backdoors to Linux packages. Only that it&amp;rsquo;s more likely to be quickly detected. And that&amp;rsquo;s when coupled with other things, such as package signatures and code reviews.&lt;/p&gt;
&lt;p&gt;Most supply chain attacks these days are aimed at other targets, such as Python or Node packages, not the Linux distros. But that doesn&amp;rsquo;t mean the effort was wasted - it means the distros are now a really tough target.&lt;/p&gt;
&lt;h2 id="should-you-care"&gt;Should you care?
&lt;/h2&gt;&lt;p&gt;As a user: not much you need to do, except appreciate that someone is doing the boring work that few people would even understand.&lt;/p&gt;
&lt;p&gt;As someone who builds software: set &lt;code&gt;SOURCE_DATE_EPOCH&lt;/code&gt;, don&amp;rsquo;t embed build timestamps and hostnames if you don&amp;rsquo;t have to, sort your file lists, use &lt;code&gt;-ffile-prefix-map&lt;/code&gt;. It costs almost nothing, and makes your releases verifiable. Or make your app so popular that it gets into Debian, then the package maintainer will do that for you.&lt;/p&gt;
&lt;h2 id="sources"&gt;Sources
&lt;/h2&gt;&lt;ul&gt;
&lt;li&gt;Ken Thompson, &lt;a class="link" href="https://doi.org/10.1145/358198.358210" target="_blank" rel="noopener"
 &gt;Reflections on Trusting Trust&lt;/a&gt;, Communications of the ACM, 1984&lt;/li&gt;
&lt;li&gt;Russ Cox, &lt;a class="link" href="https://research.swtch.com/nih" target="_blank" rel="noopener"
 &gt;Running the &amp;ldquo;Reflections on Trusting Trust&amp;rdquo; Compiler&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://en.wikipedia.org/wiki/XcodeGhost" target="_blank" rel="noopener"
 &gt;XcodeGhost&lt;/a&gt; on Wikipedia&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://reproducible-builds.org/" target="_blank" rel="noopener"
 &gt;Reproducible Builds project&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://reproducible-builds.org/specs/source-date-epoch/" target="_blank" rel="noopener"
 &gt;&lt;code&gt;SOURCE_DATE_EPOCH&lt;/code&gt; specification&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://diffoscope.org/" target="_blank" rel="noopener"
 &gt;diffoscope&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://tests.reproducible-builds.org/debian/reproducible.html" target="_blank" rel="noopener"
 &gt;Debian reproducibility status&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://reproducible.archlinux.org/" target="_blank" rel="noopener"
 &gt;Arch Linux reproducibility status&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://lwn.net/Articles/1014979/" target="_blank" rel="noopener"
 &gt;Fedora change aims for 99% package reproducibility&lt;/a&gt; (LWN, 2025) and the &lt;a class="link" href="https://fedoraproject.org/wiki/Changes/Package_builds_are_expected_to_be_reproducible" target="_blank" rel="noopener"
 &gt;Fedora change proposal&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://lists.snapcraft.io/archives/kernel-team/2024-January/148400.html" target="_blank" rel="noopener"
 &gt;buildinfo files are now available from Launchpad&lt;/a&gt; (Ubuntu kernel team list, 2024)&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://discourse.nixos.org/t/nixos-unstable-s-iso-minimal-x86-64-linux-is-100-reproducible/13723" target="_blank" rel="noopener"
 &gt;NixOS minimal ISO is 100% reproducible&lt;/a&gt; (NixOS Discourse, 2021)&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/" target="_blank" rel="noopener"
 &gt;The Full-Source Bootstrap&lt;/a&gt; (Guix blog, 2023) and &lt;a class="link" href="https://bootstrappable.org/" target="_blank" rel="noopener"
 &gt;bootstrappable.org&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;David A. Wheeler, &lt;a class="link" href="https://dwheeler.com/trusting-trust/" target="_blank" rel="noopener"
 &gt;Countering Trusting Trust through Diverse Double-Compiling&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Andres Freund, &lt;a class="link" href="https://www.openwall.com/lists/oss-security/2024/03/29/4" target="_blank" rel="noopener"
 &gt;backdoor in upstream xz/liblzma&lt;/a&gt; (oss-security, 2024)&lt;/li&gt;
&lt;li&gt;&lt;a class="link" href="https://nvd.nist.gov/vuln/detail/CVE-2024-3094" target="_blank" rel="noopener"
 &gt;CVE-2024-3094&lt;/a&gt; at NVD&lt;/li&gt;
&lt;/ul&gt;</description></item></channel></rss>