Reproducible Build Summit

Reproducible Build Summit

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 4947 bytes --]

Hello, Guix!

Manolis and I had the pleasure to attend the first “Reproducible Build
Summit” this week, wonderfully well organized by Debian hackers Lunar
and Holger, along with other brilliant people, and with the support of
the Linux Foundation, the Open Technology Fund, and Google.

  https://reproducible-builds.org/events/athens2015/

There were around 40 people, including contributors to a variety of free
operating systems and distros, which led to many very insightful
sessions and discussions.

Here are the highlights of the sessions I attended and discussions I
had, off the top of my head; Manolis can complement it:

 * Guix & Nix session

   Eelco Dolstra of Nix and myself held a “geek dating” session where we
   presented Nix/Guix to everyone by groups of 8 people.  That was a fun
   exercise, but also difficult to do without any demo/slides.

   I was glad to learn that Lamby, a Debian developer, had come up with
   <https://github.com/lamby/apt-challenge>.  :-)  A later session was
   dedicated to specifying a tool similar to ‘guix challenge’ and dubbed
   ‘reprotest’.

 * Shared reproducibility issue database

   Currently we have a bunch of patches and open reproducibility issues
   at <http://bugs.gnu.org/guix>, but they feel lonely!  The Debian
   folks have a Yaml database (recutils would be nice, but hey!) that is
   used to generate the pages at <https://reproducible.debian.org/>.
   They are kindly offering to open it to other distros, so that we can
   all work together instead of duplicating work.  Expect updates on the
   practical details soonish.

 * Distro bootstrapping

   We had a short session on distro bootstrapping, where we discussed
   how each project addresses it, and what we could do to improve the
   situation.  Most distros, including Debian, do not have a clearly
   specified set of bootstrap binaries like we have, but are willing to
   work towards having a more formal set of requirements, to allow the
   distro to be bootstrapped in different contexts.  There were
   discussions about diverse double compilation and whether/how it could
   help, as well as how we could reduce the bootstrap set (I suggested
   something like keeping TinyC, not GCC, as a bootstrap binary, used to
   build GCC 4.6-, used to build a more recent GCC that requires C++.)

 * Hacking sessions

   Eelco, Manolis, and I sat together for the hacking sessions.  I
   focused on shamelessly stealing the Nix daemon’s ability to rebuild a
   derivation and error out if the result differs (commits 07e70f4 and
   708d907.)

   I also discussed with Eelco the fact that the daemon was leaking the
   real name of the build directory, meaning that if a build machine
   runs:

     TMPDIR=/foo/bar guix-daemon

   and the other runs:

     TMPDIR=/tmp guix-daemon

   then the first build process will see a directory called
   /foo/bar/nix-build-xxx.  If it captures the build directory name,
   then we get a discrepancy.  Eelco quickly changed that, such that the
   build process always sees /tmp/nix-build-xxx:
   <https://github.com/NixOS/nix/commit/8063fc497ab78fa72962b93874fe25dcca2b55ed>.
   I’ll merge this commit soon.

   Of course the better fix is for packages to not capture the build
   directory name, and even to error out when that happens.  But that
   seems like a longer-term goal, potentially can-of-wormy (think of
   debugging symbols…)

   Besides, I packaged ‘findnewest’ (commit df7393b) by Thomas Klausner
   of NetBSD minutes after he had tagged it.  ;-) This tool is meant to
   help provide useful *and* reproducible timestamps for packages that
   want it.

 * Testing Guix reproducibility

   Holger of Debian said he would be happy to perform independent builds
   of Guix packages on the ProfitBricks-sponsored machines that are used
   for Debian and other distros.  Manolis will discuss with him to get
   that up and running.

 * Authenticating code from a Git repo

   The Qubes people have a sophisticated process to sign their code in
   Git.  Since they want signatures from both the author and the
   reviewers of a given patch series, they use signed tags (signed
   commits can only have one signature.)  Thus they end up automatically
   creating signed tags for each series of patches that is pushed (a bit
   heavyweight, but it does the job.)

     https://www.qubes-os.org/doc/verifying-signatures/

   I’d like to suggest that we do something similar.

I think that’s a lot of food for thought already!  There’ll be other
such meetings in the future, so I hope we can keep making progress
together.  I would also like to get the broader GNU involved in
addressing the issue.

Anyway, that was a great experience, with lots of friendly people to
chat with and have a walk in Athens by night!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Reproducible Build Summit

[Manolis Ragkousis]

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]

Hello Guix!

It was a great, productive, week and I would really like to thank
everyone that made this possible.

Now some things I want to add:

* Signatures challenges

  We had a session on how each project signs the binaries it
distributes and ways to achieve build system compromise detection. Georg
from the Tor project told us about this paper
<https://isis.poly.edu/~jcappos/papers/samuel_tuf_ccs_2010.pdf> which
describes how we can secure a system against such attacks.

I discussed with Ludovic the need to have an automated system that will
continuously verify the binary outputs from multiple build sources so we
can find any possible malicious compromises.
But in order to do that we need to increase our build servers and/or
implement the peer to peer binary distribution (Remi?). More machines -
more builds to compare.

The above will also help in testing Guix reproducibility. Finally I will
help Holger install Guix on the ProfitBricks-sponsored machines.

* Authenticating code from a Git repo

  Here I agree with Ludovic that we should find a way to do something
similar with Qubes, so we can at least be able to pinpoint a future
compromise if it happens.

That's what I wanted to add for now.

This was a great week, I learned a lot of new things from a great bunch
of smart and friendly people and I believe we should get a lot more
involved in this initiative.

Manolis.

[-- Attachment #2: 0x8DF54C32.asc --]
[-- Type: application/pgp-keys, Size: 3910 bytes --]

Re: Reproducible Build Summit

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

>    Eelco, Manolis, and I sat together for the hacking sessions.  I
>    focused on shamelessly stealing the Nix daemon’s ability to rebuild a
>    derivation and error out if the result differs (commits 07e70f4 and
>    708d907.)

Commit a8d6564 adds the --check option for ‘guix build’:

  ‘--check’
       Rebuild PACKAGE-OR-DERIVATION, which are already available in the
       store, and raise an error if the build results are not bit-for-bit
       identical.

       This mechanism allows you to check whether previously-installed
       substitutes are genuine (*note Substitutes::), or whether a
       package’s build result is deterministic.  *Note Invoking guix
       challenge::, for more background information and tools.

and commit 5b74fe0 adds --rounds:

  ‘--rounds=N’
       Build each derivation N times in a row, and raise an error if
       consecutive build results are not bit-for-bit identical.

       This is a useful way to detect non-deterministic builds processes.
       Non-deterministic build processes are a problem because they make
       it practically impossible for users to _verify_ whether third-party
       binaries are genuine.  *Note Invoking guix challenge::, for more.

       Note that, currently, the differing build results are not kept
       around, so you will have to manually investigate in case of an
       error—e.g., by stashing one of the build results with ‘guix archive
       --export’, then rebuilding, and finally comparing the two results.

I encourage you to use the latter when adding new packages and to
investigate any reproducibility issues!

>    I also discussed with Eelco the fact that the daemon was leaking the
>    real name of the build directory, meaning that if a build machine
>    runs:
>
>      TMPDIR=/foo/bar guix-daemon
>
>    and the other runs:
>
>      TMPDIR=/tmp guix-daemon
>
>    then the first build process will see a directory called
>    /foo/bar/nix-build-xxx.  If it captures the build directory name,
>    then we get a discrepancy.  Eelco quickly changed that, such that the
>    build process always sees /tmp/nix-build-xxx:
>    <https://github.com/NixOS/nix/commit/8063fc497ab78fa72962b93874fe25dcca2b55ed>.
>    I’ll merge this commit soon.

Done in cb96010.

Ludo’.

Shared reproducibility issue database

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

>  * Shared reproducibility issue database
>
>    Currently we have a bunch of patches and open reproducibility issues
>    at <http://bugs.gnu.org/guix>, but they feel lonely!  The Debian
>    folks have a Yaml database (recutils would be nice, but hey!) that is
>    used to generate the pages at <https://reproducible.debian.org/>.
>    They are kindly offering to open it to other distros, so that we can
>    all work together instead of duplicating work.  Expect updates on the
>    practical details soonish.

An update!  You can already do:

  git clone git://git.debian.org/git/reproducible/notes.git

to access the database, which may contain useful hints in packages.yml.

Furthermore, the fine hackers on #reproducible-builds on irc.oftc.net
said they’re happy to give accounts to people who ask.  So you can go to
alioth.debian.org, ask for an account, and then click on “Request to
join” at <https://alioth.debian.org/projects/reproducible>.

However, note that the database hasn’t been converted yet to the
distro-independent format that we discussed at the meeting.  This will
happen probably sometime in January.  Until then it probably doesn’t
make much sense for us to modify it.

Ludo’.

Re: Reproducible Build Summit

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Manolis and I had the pleasure to attend the first “Reproducible Build
> Summit” this week, wonderfully well organized by Debian hackers Lunar
> and Holger, along with other brilliant people, and with the support of
> the Linux Foundation, the Open Technology Fund, and Google.
>
>   https://reproducible-builds.org/events/athens2015/

Minutes are now available at this URL.

Ludo’.