Reproducible build: linux-libre kernel

Reproducible build: linux-libre kernel

[rain1]

[-- Attachment #1: Type: text/plain, Size: 700 bytes --]

Hello!

I noticed that the linux-libre package we have is not a reproducible 
build using the command: guix build linux-libre --check --no-substitutes

I did a bit of research into this these posts and blogs were helpful, as 
well as the mempo project:
* <https://lwn.net/Articles/437864/>
* <https://abraithwaite.net/2014/08/11/deterministic-kernel-builds/>
* 
<https://github.com/rfree/mempo-kernel/blob/master/kernel-build/linux-mempo/build.sh#L95>

So I've added these tweaks, performed the build twice and it seems to be 
reproducible now! Would anyone like to test it and see if they do get 
the same build? (On x86_64)

/gnu/store/zbzs26xa68znnlgvgqchm4y7lxbnl4gg-linux-libre-det-4.5

Cheers.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-packages-linux.scm-Added-setenv-calls-to-tune-th.patch --]
[-- Type: text/x-diff; name=0001-gnu-packages-linux.scm-Added-setenv-calls-to-tune-th.patch, Size: 1457 bytes --]

From 713017b65c9d5a03dcf709bb231113d2152558e1 Mon Sep 17 00:00:00 2001
From: rain1 <rain1@openmailbox.org>
Date: Wed, 30 Mar 2016 13:05:19 +0100
Subject: [PATCH] * gnu/packages/linux.scm: Added setenv calls to tune the
 build towards reproducibility.

---
 gnu/packages/linux.scm | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index fb2b671..34403ae 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -9,6 +9,7 @@
 ;;; Copyright © 2016 Christopher Allan Webber <cwebber@dustycloud.org>
 ;;; Copyright © 2016 Tobias Geerinckx-Rice <tobias.geerinckx.rice@gmail.com>
 ;;; Copyright © 2016 Alex Kost <alezost@gmail.com>
+;;; Copyright © 2016 Raymond Nicholson <rain1@openmailbox.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -223,6 +224,12 @@ for SYSTEM and optionally VARIANT, or #f if there is no such configuration."
   (let* ((version "4.5")
          (build-phase
           '(lambda* (#:key system inputs #:allow-other-keys #:rest args)
+             ;; Tuning for a reproducible build
+             (setenv "KCONFIG_NOTIMESTAMP" "1")
+             (setenv "KBUILD_BUILD_TIMESTAMP" "0")
+             (setenv "KBUILD_BUILD_USER" "guix")
+             (setenv "KBUILD_BUILD_HOST" "guix")
+             
              ;; Apply the neat patch.
              (system* "patch" "-p1" "--force"
                       "-i" (assoc-ref inputs "patch/freedo+gnu"))
-- 
2.7.3

Re: Reproducible build: linux-libre kernel

[Leo Famulari]

On Wed, Mar 30, 2016 at 01:16:13PM +0100, rain1@openmailbox.org wrote:
> Hello!
> 
> I noticed that the linux-libre package we have is not a reproducible build
> using the command: guix build linux-libre --check --no-substitutes
> 
> I did a bit of research into this these posts and blogs were helpful, as
> well as the mempo project:
> * <https://lwn.net/Articles/437864/>
> * <https://abraithwaite.net/2014/08/11/deterministic-kernel-builds/>
> * <https://github.com/rfree/mempo-kernel/blob/master/kernel-build/linux-mempo/build.sh#L95>
> 
> So I've added these tweaks, performed the build twice and it seems to be
> reproducible now! Would anyone like to test it and see if they do get the
> same build? (On x86_64)

Wow, thanks for taking this on! I'm testing it now... it might take a
little while.

>    (let* ((version "4.5")
>           (build-phase
>            '(lambda* (#:key system inputs #:allow-other-keys #:rest args)
> +             ;; Tuning for a reproducible build
> +             (setenv "KCONFIG_NOTIMESTAMP" "1")
> +             (setenv "KBUILD_BUILD_TIMESTAMP" "0")

For these timestamps, can you set them to value of the environment
variable SOURCE_DATE_EPOCH? [0] If you search the package tree you will
find a few examples. This variable is set automatically in the build
environment.

> +             (setenv "KBUILD_BUILD_USER" "guix")
> +             (setenv "KBUILD_BUILD_HOST" "guix")

Can you say if there is a canonical user name for this sort of thing, as
with SOURCE_DATE_EPOCH for dates? The site in [0] could be a good
resource. There is also the 'notes' git repo used to catalogue
reproducibility problems and solutions [1].


[0]
https://reproducible-builds.org/specs/source-date-epoch/

[1]
https://anonscm.debian.org/cgit/reproducible/notes.git

Re: Reproducible build: linux-libre kernel

[Mark H Weaver]

rain1@openmailbox.org writes:

> I noticed that the linux-libre package we have is not a reproducible
> build using the command: guix build linux-libre --check
> --no-substitutes
>
> I did a bit of research into this these posts and blogs were helpful,
> as well as the mempo project:
> * <https://lwn.net/Articles/437864/>
> * <https://abraithwaite.net/2014/08/11/deterministic-kernel-builds/>
> *
> <https://github.com/rfree/mempo-kernel/blob/master/kernel-build/linux-mempo/build.sh#L95>
>
> So I've added these tweaks, performed the build twice and it seems to
> be reproducible now!

That's great news, thank you!  Please see below for comments.

> From 713017b65c9d5a03dcf709bb231113d2152558e1 Mon Sep 17 00:00:00 2001
> From: rain1 <rain1@openmailbox.org>
> Date: Wed, 30 Mar 2016 13:05:19 +0100
> Subject: [PATCH] * gnu/packages/linux.scm: Added setenv calls to tune the
>  build towards reproducibility.

Your commit message is missing the initial summary line, and needs some
other changes as well to conform with our conventions.

Also, it turns out that our build container already ensures that the
build user and host are deterministic, so the timestamps were the only
issue.

So, how about something like this for the commit message:

--8<---------------cut here---------------start------------->8---
gnu: linux-libre: Avoid introducing timestamps into the build.

* gnu/packages/linux.scm (linux-libre)[build-phase]: Set the
KCONFIG_NOTIMESTAMP and KBUILD_BUILD_TIMESTAMP environment variables to
avoid introducing timestamps into the build outputs.
--8<---------------cut here---------------end--------------->8---

> diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
> index fb2b671..34403ae 100644
> --- a/gnu/packages/linux.scm
> +++ b/gnu/packages/linux.scm
> @@ -9,6 +9,7 @@
>  ;;; Copyright ?? 2016 Christopher Allan Webber <cwebber@dustycloud.org>
>  ;;; Copyright ?? 2016 Tobias Geerinckx-Rice <tobias.geerinckx.rice@gmail.com>
>  ;;; Copyright ?? 2016 Alex Kost <alezost@gmail.com>
> +;;; Copyright ?? 2016 Raymond Nicholson <rain1@openmailbox.org>

Your mail client doesn't seem to realize that this patch uses the UTF-8
encoding, so it's not including a header saying so, and the non-ASCII
copyright symbols are getting munged into two separate bytes.  When
trying to reply, my mail client is complaining about them.

> @@ -223,6 +224,12 @@ for SYSTEM and optionally VARIANT, or #f if there is no such configuration."
>    (let* ((version "4.5")
>           (build-phase
>            '(lambda* (#:key system inputs #:allow-other-keys #:rest args)
> +             ;; Tuning for a reproducible build

How about changing the comment to:

                ;; Avoid introducing timestamps
> +             (setenv "KCONFIG_NOTIMESTAMP" "1")
> +             (setenv "KBUILD_BUILD_TIMESTAMP" "0")

Also, Leo's suggestion to use SOURCE_DATE_EPOCH is a good one.

The following two settings can be dropped.

> +             (setenv "KBUILD_BUILD_USER" "guix")
> +             (setenv "KBUILD_BUILD_HOST" "guix")

What do you think?  Can you send an updated patch?

    Thank you!
       Mark

Re: Reproducible build: linux-libre kernel

[rain1]

[-- Attachment #1: Type: text/plain, Size: 108 bytes --]

Thank you both for looking at the patch and the feedback!

I've put these suggestions into this new version.

[-- Attachment #2: 0001-gnu-linux-libre-Avoid-introducing-timestamps-into-th.patch --]
[-- Type: application/octet-stream, Size: 1538 bytes --]

From 0e10fd8e7790e584c96a42a4c38484316d30bcf7 Mon Sep 17 00:00:00 2001
From: rain1 <rain1@openmailbox.org>
Date: Wed, 30 Mar 2016 13:05:19 +0100
Subject: [PATCH] gnu: linux-libre: Avoid introducing timestamps into the
 build.

* gnu/packages/linux.scm (linux-libre)[build-phase]: Set the
KCONFIG_NOTIMESTAMP and KBUILD_BUILD_TIMESTAMP environment variables to
avoid introducing timestamps into the build outputs.
---
 gnu/packages/linux.scm | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index fb2b671..1400110 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -9,6 +9,7 @@
 ;;; Copyright © 2016 Christopher Allan Webber <cwebber@dustycloud.org>
 ;;; Copyright © 2016 Tobias Geerinckx-Rice <tobias.geerinckx.rice@gmail.com>
 ;;; Copyright © 2016 Alex Kost <alezost@gmail.com>
+;;; Copyright © 2016 Raymond Nicholson <rain1@openmailbox.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -223,6 +224,10 @@ for SYSTEM and optionally VARIANT, or #f if there is no such configuration."
   (let* ((version "4.5")
          (build-phase
           '(lambda* (#:key system inputs #:allow-other-keys #:rest args)
+             ;; Avoid introducing timestamps
+             (setenv "KCONFIG_NOTIMESTAMP" "1")
+             (setenv "KBUILD_BUILD_TIMESTAMP" (getenv "SOURCE_DATE_EPOCH"))
+             
              ;; Apply the neat patch.
              (system* "patch" "-p1" "--force"
                       "-i" (assoc-ref inputs "patch/freedo+gnu"))
-- 
2.7.3

Re: Reproducible build: linux-libre kernel

[Mark H Weaver]

rain1 <rain1@openmailbox.org> writes:

> Thank you both for looking at the patch and the feedback!
>
> I've put these suggestions into this new version.

Looks good.  Pushed, thanks!

     Mark

Re: Reproducible build: linux-libre kernel

[Ludovic Courtès]

This is time-consuming but also very valuable work, so big thanks!

Ludo’.

Re: Reproducible build: linux-libre kernel

[Leo Famulari]

On Wed, Mar 30, 2016 at 11:25:54PM +0200, Ludovic Courtès wrote:
> This is time-consuming but also very valuable work, so big thanks!

Indeed! I'm really glad for this patch!

Building with --rounds=2 right now :)

Re: Reproducible build: linux-libre kernel

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 3061 bytes --]

Leo Famulari <leo@famulari.name> skribis:

> On Wed, Mar 30, 2016 at 11:25:54PM +0200, Ludovic Courtès wrote:
>> This is time-consuming but also very valuable work, so big thanks!
>
> Indeed! I'm really glad for this patch!
>
> Building with --rounds=2 right now :)

Almost there, but not quite!  :-)

I’ve just built it on x86_64, and:

--8<---------------cut here---------------start------------->8---
$ git describe 
v0.10.0-36-g4827b36
$ ./pre-inst-env guix challenge linux-libre
updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
updating list of substitutes from 'https://hydra.gnu.org'... 100.0%
/gnu/store/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5 contents differ:
  local hash: 0sh393xb88sybf2dnk73yya02s3bflb29qswj0fn9lzbgiwk000a
  https://mirror.hydra.gnu.org/nar/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5: 0l2la1l9q6qiyf4i6pmbvvmh94l70yvzx4kdvqjyl7d9irykivma
  https://hydra.gnu.org/nar/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5: 0l2la1l9q6qiyf4i6pmbvvmh94l70yvzx4kdvqjyl7d9irykivma
--8<---------------cut here---------------end--------------->8---

Closer inspection:

--8<---------------cut here---------------start------------->8---
$ wget -q -O - https://mirror.hydra.gnu.org/nar/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5 | bunzip2 | guix archive -x /tmp/linux-libre
$ diff -ru --no-dereference /tmp/linux-libre /gnu/store/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5
Binary files /tmp/linux-libre/bzImage and /gnu/store/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5/bzImage differ
$ ls -lrt  /tmp/linux-libre/bzImage  /gnu/store/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5/bzImage
-r--r--r-- 2 root guixbuild 6965120 Jan  1  1970 /gnu/store/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5/bzImage
-rw-r--r-- 1 ludo users     6965120 Mar 31 10:11 /tmp/linux-libre/bzImage
$ size  /tmp/linux-libre/bzImage  /gnu/store/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5/bzImage
size: /tmp/linux-libre/bzImage: Warning: Ignoring section flag IMAGE_SCN_MEM_NOT_PAGED in section .bss
   text    data     bss     dec     hex filename
6964608       0 16398464        23363072        1647e00 /tmp/linux-libre/bzImage
size: /gnu/store/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5/bzImage: Warning: Ignoring section flag IMAGE_SCN_MEM_NOT_PAGED in section .bss
6964608       0 16398464        23363072        1647e00 /gnu/store/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5/bzImage
--8<---------------cut here---------------end--------------->8---

The diffoscope output is huge and not very helpful at first glance, but
I’m attaching it anyway.

I don’t see anything about the kernel Linux in the reproducibility notes
at <git://git.debian.org/git/reproducible/notes.git>.

It would be worth asking on the mailing list at
<https://lists.reproducible-builds.org/listinfo/rb-general> though.
Others have probably looked into it already and could provide useful
insight.  Any taker?

Thanks for looking into it!

Ludo’.


[-- Attachment #2: diffoscope HTML output --]
[-- Type: application/octet-stream, Size: 219209 bytes --]

Re: Reproducible build: linux-libre kernel

[Leo Famulari]

On Thu, Mar 31, 2016 at 10:26:09AM +0200, Ludovic Courtès wrote:
> Leo Famulari <leo@famulari.name> skribis:
> 
> > On Wed, Mar 30, 2016 at 11:25:54PM +0200, Ludovic Courtès wrote:
> >> This is time-consuming but also very valuable work, so big thanks!
> >
> > Indeed! I'm really glad for this patch!
> >
> > Building with --rounds=2 right now :)
> 
> Almost there, but not quite!  :-)
> 
> I’ve just built it on x86_64, and:
> 
> --8<---------------cut here---------------start------------->8---
> $ git describe 
> v0.10.0-36-g4827b36
> $ ./pre-inst-env guix challenge linux-libre
> updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
> updating list of substitutes from 'https://hydra.gnu.org'... 100.0%
> /gnu/store/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5 contents differ:
>   local hash: 0sh393xb88sybf2dnk73yya02s3bflb29qswj0fn9lzbgiwk000a
>   https://mirror.hydra.gnu.org/nar/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5: 0l2la1l9q6qiyf4i6pmbvvmh94l70yvzx4kdvqjyl7d9irykivma
>   https://hydra.gnu.org/nar/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5: 0l2la1l9q6qiyf4i6pmbvvmh94l70yvzx4kdvqjyl7d9irykivma
> --8<---------------cut here---------------end--------------->8---

Would hydra have already built with the lastest patch?

Anyways, I seem to get contradictory results on different machines. For 
both machines, the desired output was
'/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5'.

When --rounds=x or --check fail, my workflow is to build twice, copy the
results of each build out of the store after each build, and compare
with diffoscope.

On my x86_64 workstation, the only difference seems to be that for one
of the two iterations of building linux-libre, the file
'lib/modules/4.5.0-gnu/build' is missing.

For the build that does have that file, it's a symlink that points to
'/tmp/guix-build-linux-libre-4.5.drv-0/linux-4.5'.

So, something is wrong there... and I did this "pair-wise comparison" 3
times, always with this result.

On this workstation machine, $TMPDIR is btrfs.

On my x86_64 low-power home server thing, building with --rounds=2
returned successfully, so I didn't do the rsync-and-diffoscope dance. On
this machine $TMPDIR is ext4.

I'm going to retry on the workstation with an ext4 $TMPDIR.

Re: Reproducible build: linux-libre kernel

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Thu, Mar 31, 2016 at 10:26:09AM +0200, Ludovic Courtès wrote:
>> Leo Famulari <leo@famulari.name> skribis:
>> 
>> > On Wed, Mar 30, 2016 at 11:25:54PM +0200, Ludovic Courtès wrote:
>> >> This is time-consuming but also very valuable work, so big thanks!
>> >
>> > Indeed! I'm really glad for this patch!
>> >
>> > Building with --rounds=2 right now :)
>> 
>> Almost there, but not quite!  :-)
>> 
>> I’ve just built it on x86_64, and:
>> 
>> --8<---------------cut here---------------start------------->8---
>> $ git describe 
>> v0.10.0-36-g4827b36
>> $ ./pre-inst-env guix challenge linux-libre
>> updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
>> updating list of substitutes from 'https://hydra.gnu.org'... 100.0%
>> /gnu/store/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5 contents differ:
>>   local hash: 0sh393xb88sybf2dnk73yya02s3bflb29qswj0fn9lzbgiwk000a
>>   https://mirror.hydra.gnu.org/nar/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5: 0l2la1l9q6qiyf4i6pmbvvmh94l70yvzx4kdvqjyl7d9irykivma
>>   https://hydra.gnu.org/nar/n20i0s96p29vkl07vlvxli5xql3m3hgi-linux-libre-4.5: 0l2la1l9q6qiyf4i6pmbvvmh94l70yvzx4kdvqjyl7d9irykivma
>> --8<---------------cut here---------------end--------------->8---
>
> Would hydra have already built with the lastest patch?

Yes, according to the ‘git describe’ output above.

It might be that this discrepancy shows up only when building on
machines with different CPU flavors, in which case --rounds doesn’t
help.

Ludo’.