Quick reproducible test for GNU Guix

Quick reproducible test for GNU Guix

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 2098 bytes --]

I did some quick reproducibility testing running GNU Guix, and so far
got pretty good results:

Using guix (and packages) built from commit:

  f83d07f7778b699d46741a5667113342f5f0a737

$ guix challenge --verbose --diff=diffoscope ...
2,463 store items were analyzed:
  - 2,016 (81.9%) were identical
  - 37 (1.5%) differed
  - 410 (16.6%) were inconclusive

The log file produced was 951MB compressed with lz4, including
diffoscope output.

On an x86_64 virtual machine with 8-12 cores and 16-32 GB of ram (I
upped the specs later in the build process), running Debian and some WIP
packages from:

  https://salsa.debian.org/vagrant/guix

The packages tested were from a minimal system using
gnu/system/examples/bare-bones.tmpl, diffoscope, a couple guix pulls,
all without using any substitutes; all the builds for this system should
be locally built.


At ~82% identical, that's not too bad. The ~17% inconclusive are likely
substitutes that haven't yet built or failed to build on the substitute
server, or some local builds that are not yet in guix (e.g. WIP builds
of reprotest!!). I wonder if "guix challenge" may compare some things
that are normally only built locally (e.g. grub.cfg or similar ?).


There are about 10,000 packages I haven't built and compared yet,
presuming "guix package -A | wc -l" is a reasonable guess, though there
are some hidden packages used as transitive build dependencies, if I
recall.


I did notice that when doing multiple builds in parallel, it uses a
different user for each build, though I don't know if that's normalized
within the build environment container. If not, that could fix quite a
few issues! See issues mentioning "user" at:

  https://salsa.debian.org/reproducible-builds/reproducible-notes/blob/master/issues.yml


Eventually, I'd like to do more systematic test of guix packages, with
published logs per-package, rather than whatever I happened to build on
the system so far, but this was a quick start to help flesh out ideas
for feature requests to "guix challenge" to make this all easier... more
on that soon!


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [rb-general] Quick reproducible test for GNU Guix

[Ludovic Courtès]

Hi Vagrant!

Vagrant Cascadian <vagrant@reproducible-builds.org> skribis:

> I did some quick reproducibility testing running GNU Guix, and so far
> got pretty good results:
>
> Using guix (and packages) built from commit:
>
>   f83d07f7778b699d46741a5667113342f5f0a737
>
> $ guix challenge --verbose --diff=diffoscope ...
> 2,463 store items were analyzed:
>   - 2,016 (81.9%) were identical
>   - 37 (1.5%) differed
>   - 410 (16.6%) were inconclusive

Woow, nice!  There are 12+K packages now, but at least that gives us a
good overview of the story for core packages.

> The log file produced was 951MB compressed with lz4, including
> diffoscope output.

Still reading it?  ;-)

> At ~82% identical, that's not too bad. The ~17% inconclusive are likely
> substitutes that haven't yet built or failed to build on the substitute
> server, or some local builds that are not yet in guix (e.g. WIP builds
> of reprotest!!). I wonder if "guix challenge" may compare some things
> that are normally only built locally (e.g. grub.cfg or similar ?).

Yes, “guix challenge” without any arguments compares all the items in
your store, which includes file specific to your machine, like
‘grub.cfg’, the initrd, etc.

If you want to focus on packages, you can run something like:

  guix challenge $(guix package -A | cut -f1)

> I did notice that when doing multiple builds in parallel, it uses a
> different user for each build, though I don't know if that's normalized
> within the build environment container. If not, that could fix quite a
> few issues! See issues mentioning "user" at:
>
>   https://salsa.debian.org/reproducible-builds/reproducible-notes/blob/master/issues.yml

The user name in the build environment is canonicalized:

  https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc#n1858

> Eventually, I'd like to do more systematic test of guix packages, with
> published logs per-package, rather than whatever I happened to build on
> the system so far, but this was a quick start to help flesh out ideas
> for feature requests to "guix challenge" to make this all easier... more
> on that soon!

That’d be great!

Related to that, Christopher Baines developed a nice feature for the
Guix Data Service during and after the summit: for each Guix revision,
there’s a page showing the overall package reproducibility status based
on the info obtained from our two independent build farms.  The URL is
something like
<http://data.guix.gnu.org/revision/e7ce4ef997c3db4d47d943c3e435f395b278bc50/package-reproducibility>
(right now it’s empty for some reason, but normally it shows
identical/different/inconclusive percentages.)

Thanks,
Ludo’.

Re: [rb-general] Quick reproducible test for GNU Guix

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

>> Eventually, I'd like to do more systematic test of guix packages, with
>> published logs per-package, rather than whatever I happened to build on
>> the system so far, but this was a quick start to help flesh out ideas
>> for feature requests to "guix challenge" to make this all easier... more
>> on that soon!
>
> That’d be great!
>
> Related to that, Christopher Baines developed a nice feature for the
> Guix Data Service during and after the summit: for each Guix revision,
> there’s a page showing the overall package reproducibility status based
> on the info obtained from our two independent build farms.  The URL is
> something like
> <http://data.guix.gnu.org/revision/e7ce4ef997c3db4d47d943c3e435f395b278bc50/package-reproducibility>
> (right now it’s empty for some reason, but normally it shows
> identical/different/inconclusive percentages.)

Hey,

This should be back working now. I changed how cross derivations are
stored and handled, and that broke this page. It's now back showing at
least something, but there's still some work to do before it's showing
representative data.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: [rb-general] Quick reproducible test for GNU Guix

[Holger Levsen]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

Hi Vagrant,

On Fri, Feb 07, 2020 at 03:08:59PM -0800, Vagrant Cascadian wrote:
> I did some quick reproducibility testing running GNU Guix, and so far
> got pretty good results:
> 
> Using guix (and packages) built from commit:
>   f83d07f7778b699d46741a5667113342f5f0a737
> 
> $ guix challenge --verbose --diff=diffoscope ...
> 2,463 store items were analyzed:
>   - 2,016 (81.9%) were identical
>   - 37 (1.5%) differed
>   - 410 (16.6%) were inconclusive

thanks for sharing these results, very nice! I've added this to the February
report and would be happy to receive future updates on this as well as setup
some continuous testing!


-- 
cheers,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]