* Proving bitwise reproducibility
@ 2025-02-03 21:28 Alexis Praga via Guix-Science
2025-02-04 8:30 ` Cayetano Santos
2025-02-04 11:45 ` Simon Tournier
0 siblings, 2 replies; 13+ messages in thread
From: Alexis Praga via Guix-Science @ 2025-02-03 21:28 UTC (permalink / raw)
To: guix-science@gnu.org
[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]
Dear Guix users,
I have discovered the world of functional package managers 2 years ago
through Nix and have since introduced Nix in my day job (running
genetics pipeline on our local cluster).
Julien Malkan (Telecom Paris) published a paper a few days ago in
which he checked the bitwise reproducibility of more than 700 000
packages in nixpkgs : https://hal.science/hal-04913007
In short, he used `nix-build --check` to compared a build to a
reference binary.
I find this approach really interesting and was wondering if something
similar was possible (and maybe planned) for guix ?
To quote the author :
Given the similarity in design choices, we do not have reasons
to believe that Guix would perform
any differently in terms of build reproducibility than Nix, but
we have not empirically verified it;
doing so would be useful complementary work.
Thanks,
Alexis Praga
Oncobiologie génétique bioinformatique
CHU Besançon
M. D, Ph. D
[-- Attachment #2: Type: text/html, Size: 2864 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-03 21:28 Proving bitwise reproducibility Alexis Praga via Guix-Science
@ 2025-02-04 8:30 ` Cayetano Santos
2025-02-04 10:30 ` Hugo Buddelmeijer
2025-02-04 16:10 ` Ludovic Courtès
2025-02-04 11:45 ` Simon Tournier
1 sibling, 2 replies; 13+ messages in thread
From: Cayetano Santos @ 2025-02-04 8:30 UTC (permalink / raw)
To: Alexis Praga via Guix-Science; +Cc: alexis
>lun. 03 févr. 2025 at 22:28, Alexis Praga via Guix-Science <guix-science@gnu.org> wrote:
> Julien Malkan (Telecom Paris) published a paper a few days ago in
> which he checked the bitwise reproducibility of more than 700 000
> packages in nixpkgs : https://hal.science/hal-04913007
For completeness, the summary blog post:
https://luj.fr/blog/is-nixos-truly-reproducible.html
Interesting enough, NixOS removed the “Reproducible Builds” terms from
their site.
https://github.com/NixOS/nixos-homepage/pull/1077
> In short, he used `nix-build --check` to compared a build to a
> reference binary.
>
> I find this approach really interesting and was wondering if something
> similar was possible (and maybe planned) for guix ?
>
> To quote the author :
>
> Given the similarity in design choices, we do not have reasons
> to believe that Guix would perform
>
> any differently in terms of build reproducibility than Nix, but
> we have not empirically verified it;
>
> doing so would be useful complementary work.
I had exactly the same reaction when reading the paper, and found the ci
testing reports in https://reproducible-builds.org/citests. Among them,
guix (link is very slow to load):
https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility
which seems unmaintained / outdated.
I was unable to find any more relevant information about this, though.
Best,
--
Cayetano Santos
cayetano.santos@cnrs.fr
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-04 8:30 ` Cayetano Santos
@ 2025-02-04 10:30 ` Hugo Buddelmeijer
2025-02-04 10:42 ` Alexis Praga via Guix-Science
2025-02-04 12:15 ` Simon Tournier
2025-02-04 16:10 ` Ludovic Courtès
1 sibling, 2 replies; 13+ messages in thread
From: Hugo Buddelmeijer @ 2025-02-04 10:30 UTC (permalink / raw)
To: Cayetano Santos; +Cc: Alexis Praga via Guix-Science, alexis
Hi,
FWIW, I was wondering how well GUIX is reproducible, so here are some
results of my mini investigation.
My conclusion is that the goal of GUIX is reproducibility, but that it
is not guaranteed:
https://guix.gnu.org/manual/en/html_node/Invoking-guix-challenge.html
> We try hard to remove sources of non-determinism in packages to make it easier to verify substitutes, but of course, this is a process that involves not just Guix, but a large part of the free software community. In the meantime, guix challenge is one tool to help address the problem.
https://guix.gnu.org/manual/en/html_node/Features.html
> [The purely functional approach] also helps maximize build reproducibility: thanks to the isolated build environments that are used, a given build is likely to yield bit-identical files when performed on different machines
> Because build results are usually bit-for-bit reproducible, users do not have to trust servers that provide substitutes
Running `guix challenge $(guix package -I | awk '{print $1}')` gave this output:
278 store items were analyzed:
- 192 (69.1%) were identical
- 5 (1.8%) differed
- 81 (29.1%) were inconclusive
The 81 inconclusive package did not have a local build. So 5 out of
197 differed:
/gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9
/gnu/store/lv80kc60hgms8bj8ypcsc3wplxgcx4z4-nss-3.99
/gnu/store/imz9c12jm9fmvcdfkk2fv2ypxhb0viab-autogen-5.18.16
/gnu/store/acykx90a2391l7xi67v19nz73gdgfq28-cc65-2.19
/gnu/store/a34kfcidwii9xvv47i7gip7jzy4acfcw-parallel-20250122
All these differences were between https://bordeaux.guix.gnu.org and
https://ci.guix.gnu.org ; maybe there is a way to compare those two
substitute servers directly.
While it is great that only 5 packages are indeterministic, I would
have expected such core packages as guile and autogen to be
deterministic. More details about the differences below.
Hugo
differing files for guile:
/lib/guile/3.0/ccache/ice-9/ftw.go
differing files for nss:
/lib/nss/libfreebl3.chk
/lib/nss/libfreeblpriv3.chk
/lib/nss/libnssdbm3.chk
/lib/nss/libsoftokn3.chk
differing file for autogen:
/bin/autogen
The differences for parallel were all pdf's, and for cc65 there were
many differences.
I tried the diffoscope, which sounds pretty nice in theory, but I did
not manage to gain more insights from that.
On Tue, 4 Feb 2025 at 09:30, Cayetano Santos <cayetano.santos@cnrs.fr> wrote:
>
>
> >lun. 03 févr. 2025 at 22:28, Alexis Praga via Guix-Science <guix-science@gnu.org> wrote:
>
> > Julien Malkan (Telecom Paris) published a paper a few days ago in
> > which he checked the bitwise reproducibility of more than 700 000
> > packages in nixpkgs : https://hal.science/hal-04913007
>
> For completeness, the summary blog post:
>
> https://luj.fr/blog/is-nixos-truly-reproducible.html
>
> Interesting enough, NixOS removed the “Reproducible Builds” terms from
> their site.
>
> https://github.com/NixOS/nixos-homepage/pull/1077
>
> > In short, he used `nix-build --check` to compared a build to a
> > reference binary.
> >
> > I find this approach really interesting and was wondering if something
> > similar was possible (and maybe planned) for guix ?
> >
> > To quote the author :
> >
> > Given the similarity in design choices, we do not have reasons
> > to believe that Guix would perform
> >
> > any differently in terms of build reproducibility than Nix, but
> > we have not empirically verified it;
> >
> > doing so would be useful complementary work.
>
> I had exactly the same reaction when reading the paper, and found the ci
> testing reports in https://reproducible-builds.org/citests. Among them,
> guix (link is very slow to load):
>
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility
>
> which seems unmaintained / outdated.
>
> I was unable to find any more relevant information about this, though.
>
> Best,
>
> --
> Cayetano Santos
>
> cayetano.santos@cnrs.fr
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-04 10:30 ` Hugo Buddelmeijer
@ 2025-02-04 10:42 ` Alexis Praga via Guix-Science
2025-02-04 11:02 ` Hugo Buddelmeijer
2025-02-04 14:37 ` Cayetano Santos
2025-02-04 12:15 ` Simon Tournier
1 sibling, 2 replies; 13+ messages in thread
From: Alexis Praga via Guix-Science @ 2025-02-04 10:42 UTC (permalink / raw)
To: Hugo Buddelmeijer; +Cc: Cayetano Santos, Alexis Praga via Guix-Science
[-- Attachment #1: Type: text/plain, Size: 5848 bytes --]
Hi Hugo & Cayetano,
As a follow-up, does guix challenge only compares binaries between 2
servers ? Or does it rebuild the derivation locally ?
If that's the latter, I was wondering how feasible it would be to guix
challenge on all packages in terms of storage and computational
power...
But maybe that's what already done at
https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility
Thanks,
On 2025-02-04T11:31:04.000+01:00, Hugo Buddelmeijer
<hugo@buddelmeijer.nl> wrote:
> Hi,
>
> FWIW, I was wondering how well GUIX is reproducible, so here are some
>
> results of my mini investigation.
>
> My conclusion is that the goal of GUIX is reproducibility, but that it
>
> is not guaranteed:
>
> https://guix.gnu.org/manual/en/html_node/Invoking-guix-challenge.html
>> We try hard to remove sources of non-determinism in packages to
>> make it easier to verify substitutes, but of course, this is a
>> process that involves not just Guix, but a large part of the free
>> software community. In the meantime, guix challenge is one tool to
>> help address the problem.
>
> https://guix.gnu.org/manual/en/html_node/Features.html
>
>> [The purely functional approach] also helps maximize build
>> reproducibility: thanks to the isolated build environments that
>> are used, a given build is likely to yield bit-identical files
>> when performed on different machines
>>
>> Because build results are usually bit-for-bit reproducible, users
>> do not have to trust servers that provide substitutes
>
> Running `guix challenge $(guix package -I | awk '{print $1}')` gave
> this output:
>
> 278 store items were analyzed:
>
> - 192 (69.1%) were identical
>
> - 5 (1.8%) differed
>
> - 81 (29.1%) were inconclusive
>
> The 81 inconclusive package did not have a local build. So 5 out of
>
> 197 differed:
>
> /gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9
>
> /gnu/store/lv80kc60hgms8bj8ypcsc3wplxgcx4z4-nss-3.99
>
> /gnu/store/imz9c12jm9fmvcdfkk2fv2ypxhb0viab-autogen-5.18.16
>
> /gnu/store/acykx90a2391l7xi67v19nz73gdgfq28-cc65-2.19
>
> /gnu/store/a34kfcidwii9xvv47i7gip7jzy4acfcw-parallel-20250122
>
> All these differences were between https://bordeaux.guix.gnu.org and
>
> https://ci.guix.gnu.org ; maybe there is a way to compare those two
>
> substitute servers directly.
>
> While it is great that only 5 packages are indeterministic, I would
>
> have expected such core packages as guile and autogen to be
>
> deterministic. More details about the differences below.
>
> Hugo
>
> differing files for guile:
>
> /lib/guile/3.0/ccache/ice-9/ftw.go
>
> differing files for nss:
>
> /lib/nss/libfreebl3.chk
>
> /lib/nss/libfreeblpriv3.chk
>
> /lib/nss/libnssdbm3.chk
>
> /lib/nss/libsoftokn3.chk
>
> differing file for autogen:
>
> /bin/autogen
>
> The differences for parallel were all pdf's, and for cc65 there were
>
> many differences.
>
> I tried the diffoscope, which sounds pretty nice in theory, but I
> did
>
> not manage to gain more insights from that.
>
> On Tue, 4 Feb 2025 at 09:30, Cayetano Santos
> <cayetano.santos@cnrs.fr> wrote:
>
>>> lun. 03 févr. 2025 at 22:28, Alexis Praga via Guix-Science
>>> <guix-science@gnu.org> wrote:
>>
>>> Julien Malkan (Telecom Paris) published a paper a few days ago
>>> in
>>>
>>> which he checked the bitwise reproducibility of more than 700
>>> 000
>>>
>>> packages in nixpkgs : https://hal.science/hal-04913007
>>
>> For completeness, the summary blog post:
>>
>> https://luj.fr/blog/is-nixos-truly-reproducible.html
>>
>> Interesting enough, NixOS removed the “Reproducible Builds”
>> terms from
>>
>> their site.
>>
>> https://github.com/NixOS/nixos-homepage/pull/1077
>>
>>> In short, he used `nix-build --check` to compared a build to a
>>>
>>> reference binary.
>>>
>>> I find this approach really interesting and was wondering if
>>> something
>>>
>>> similar was possible (and maybe planned) for guix ?
>>>
>>> To quote the author :
>>>
>>> Given the similarity in design choices, we do not have reasons
>>>
>>> to believe that Guix would perform
>>>
>>> any differently in terms of build reproducibility than Nix, but
>>>
>>> we have not empirically verified it;
>>>
>>> doing so would be useful complementary work.
>>
>> I had exactly the same reaction when reading the paper, and found
>> the ci
>>
>> testing reports in https://reproducible-builds.org/citests. Among
>> them,
>>
>> guix (link is very slow to load):
>>
>> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility
>>
>> which seems unmaintained / outdated.
>>
>> I was unable to find any more relevant information about this,
>> though.
>>
>> Best,
>>
>> --
>>
>> Cayetano Santos
>>
>> cayetano.santos@cnrs.fr
[-- Attachment #2: Type: text/html, Size: 9937 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-04 10:42 ` Alexis Praga via Guix-Science
@ 2025-02-04 11:02 ` Hugo Buddelmeijer
2025-02-04 14:08 ` Cayetano Santos
2025-02-04 14:37 ` Cayetano Santos
1 sibling, 1 reply; 13+ messages in thread
From: Hugo Buddelmeijer @ 2025-02-04 11:02 UTC (permalink / raw)
To: alexis; +Cc: Cayetano Santos, Alexis Praga via Guix-Science
Hi Alexis,
I am just a guix user, but I think the goal of `guix challenge` is
indeed to compare local builds. But I did not have any local builds
that were also available on the substitute servers, because well, I
would have used the binary if it was available.
I don't know how much processing power and data storage you'd need to
challenge all packages.
One thing I would like to add is that `guix build` has a `--rounds` option:
https://guix.gnu.org/manual/en/html_node/Common-Build-Options.html
> --rounds=n
>
> Build each derivation n times in a row, and raise an error if consecutive build results are not bit-for-bit identical.
>
> This is a useful way to detect non-deterministic builds processes. Non-deterministic build processes are a problem because they make it practically impossible for users to verify whether third-party binaries are genuine.
FWIW: I did a bit of investigation on the differences in autogen, and
it is only one byte. I'm sharing it below in case someone more
knowledgeable than me could do something sensible with that.
I don't think I can add much more to this discussion; I'm just sharing
what I learned because I was curious myself.
Hugo
Using xxd:
$ diff autogen.xxd.ci autogen.xxd.tobias
7378c7378
< 0001cd10: 48c7 8560 0500 000e 0000 00f6 8548 0100 H..`.........H..
---
> 0001cd10: 48c7 8560 0500 0018 0000 00f6 8548 0100 H..`.........H..
Using objdump --disassemble:
$ diff autogen.disassemble.tobias autogen.disassemble.ci
2c2
< tobias/tobias/bin/autogen: file format elf64-x86-64
---
> ci/ci/bin/autogen: file format elf64-x86-64
23522,23523c23522,23523
< 41cd10: 48 c7 85 60 05 00 00 movq $0x18,0x560(%rbp)
< 41cd17: 18 00 00 00
---
> 41cd10: 48 c7 85 60 05 00 00 movq $0xe,0x560(%rbp)
> 41cd17: 0e 00 00 00
On Tue, 4 Feb 2025 at 11:42, <alexis@praga.dev> wrote:
>
> Hi Hugo & Cayetano,
>
> As a follow-up, does guix challenge only compares binaries between 2 servers ? Or does it rebuild the derivation locally ?
>
> If that's the latter, I was wondering how feasible it would be to guix challenge on all packages in terms of storage and computational power...
> But maybe that's what already done at
>
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility
>
> Thanks,
>
> On 2025-02-04T11:31:04.000+01:00, Hugo Buddelmeijer <hugo@buddelmeijer.nl> wrote:
>
> Hi,
>
> FWIW, I was wondering how well GUIX is reproducible, so here are some
> results of my mini investigation.
>
> My conclusion is that the goal of GUIX is reproducibility, but that it
> is not guaranteed:
>
> https://guix.gnu.org/manual/en/html_node/Invoking-guix-challenge.html
>
> We try hard to remove sources of non-determinism in packages to make it easier to verify substitutes, but of course, this is a process that involves not just Guix, but a large part of the free software community. In the meantime, guix challenge is one tool to help address the problem.
>
>
> https://guix.gnu.org/manual/en/html_node/Features.html
>
> [The purely functional approach] also helps maximize build reproducibility: thanks to the isolated build environments that are used, a given build is likely to yield bit-identical files when performed on different machines
> Because build results are usually bit-for-bit reproducible, users do not have to trust servers that provide substitutes
>
>
> Running `guix challenge $(guix package -I | awk '{print $1}')` gave this output:
>
> 278 store items were analyzed:
> - 192 (69.1%) were identical
> - 5 (1.8%) differed
> - 81 (29.1%) were inconclusive
>
> The 81 inconclusive package did not have a local build. So 5 out of
> 197 differed:
>
> /gnu/store/mfkz7fvlfpv3ppwbkv0imb19nrf95akf-guile-3.0.9
> /gnu/store/lv80kc60hgms8bj8ypcsc3wplxgcx4z4-nss-3.99
> /gnu/store/imz9c12jm9fmvcdfkk2fv2ypxhb0viab-autogen-5.18.16
> /gnu/store/acykx90a2391l7xi67v19nz73gdgfq28-cc65-2.19
> /gnu/store/a34kfcidwii9xvv47i7gip7jzy4acfcw-parallel-20250122
>
> All these differences were between https://bordeaux.guix.gnu.org and
> https://ci.guix.gnu.org ; maybe there is a way to compare those two
> substitute servers directly.
>
> While it is great that only 5 packages are indeterministic, I would
> have expected such core packages as guile and autogen to be
> deterministic. More details about the differences below.
>
> Hugo
>
>
>
> differing files for guile:
> /lib/guile/3.0/ccache/ice-9/ftw.go
>
> differing files for nss:
> /lib/nss/libfreebl3.chk
> /lib/nss/libfreeblpriv3.chk
> /lib/nss/libnssdbm3.chk
> /lib/nss/libsoftokn3.chk
>
> differing file for autogen:
> /bin/autogen
>
> The differences for parallel were all pdf's, and for cc65 there were
> many differences.
>
> I tried the diffoscope, which sounds pretty nice in theory, but I did
> not manage to gain more insights from that.
>
>
>
> On Tue, 4 Feb 2025 at 09:30, Cayetano Santos <cayetano.santos@cnrs.fr> wrote:
>
>
>
> lun. 03 févr. 2025 at 22:28, Alexis Praga via Guix-Science <guix-science@gnu.org> wrote:
>
>
>
> Julien Malkan (Telecom Paris) published a paper a few days ago in
> which he checked the bitwise reproducibility of more than 700 000
> packages in nixpkgs : https://hal.science/hal-04913007
>
>
> For completeness, the summary blog post:
>
> https://luj.fr/blog/is-nixos-truly-reproducible.html
>
> Interesting enough, NixOS removed the “Reproducible Builds” terms from
> their site.
>
> https://github.com/NixOS/nixos-homepage/pull/1077
>
> In short, he used `nix-build --check` to compared a build to a
> reference binary.
>
> I find this approach really interesting and was wondering if something
> similar was possible (and maybe planned) for guix ?
>
> To quote the author :
>
> Given the similarity in design choices, we do not have reasons
> to believe that Guix would perform
>
> any differently in terms of build reproducibility than Nix, but
> we have not empirically verified it;
>
> doing so would be useful complementary work.
>
>
> I had exactly the same reaction when reading the paper, and found the ci
> testing reports in https://reproducible-builds.org/citests. Among them,
> guix (link is very slow to load):
>
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility
>
> which seems unmaintained / outdated.
>
> I was unable to find any more relevant information about this, though.
>
> Best,
>
> --
> Cayetano Santos
>
> cayetano.santos@cnrs.fr
>
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-03 21:28 Proving bitwise reproducibility Alexis Praga via Guix-Science
2025-02-04 8:30 ` Cayetano Santos
@ 2025-02-04 11:45 ` Simon Tournier
2025-02-04 14:26 ` Alexis Praga via Guix-Science
1 sibling, 1 reply; 13+ messages in thread
From: Simon Tournier @ 2025-02-04 11:45 UTC (permalink / raw)
To: alexis, guix-science@gnu.org
Hi Alexis,
Thanks for your interest with Guix. :-)
On Mon, 03 Feb 2025 at 22:28, Alexis Praga via Guix-Science <guix-science@gnu.org> wrote:
> I have discovered the world of functional package managers 2 years ago
> through Nix and have since introduced Nix in my day job (running
> genetics pipeline on our local cluster).
Yeah, your short talk at FOSDEM was nice!
https://archive.fosdem.org/2024/schedule/event/fosdem-2024-3058-nix-for-genetics-powering-a-bioinformatics-pipeline/
Well, my auto-promo ;-)
https://www.nature.com/articles/s41597-022-01720-9
A paper written by two Medical Doctors and one regular engineer to
explain the “problem“. Maybe it might help in convincing your
colleagues – if they are not already – that 1. The topic matters a lot
(else all is built on the top of sand) and 2. Nix/Guix paves the way as
tools helping in tackling such topic.
> Julien Malka (Telecom Paris) published a paper a few days ago in
> which he checked the bitwise reproducibility of more than 700 000
> packages in nixpkgs : https://hal.science/hal-04913007
Yeah, cool! Isn’t it?
Guess what? We are in touch. :-) For instance, we’ve discussed
yesterday on Mastodon what could be tried with Haskell.
https://social.sciences.re/@luj@chaos.social/113941327662536682
> I find this approach really interesting and was wondering if something
> similar was possible (and maybe planned) for guix ?
Yes something similar is possible with Guix. Run ‘guix build --check’
and voilà… Euh?! On paper, it’s “and voilà” but in real world it’s more
engineering than just that.
In order to compare, Guix has some tooling as “guix challenge” or “guix
build --check” and all the power of Guix as a Scheme library.
Please note that Guix is also pushing the issue of bootstrapping.
Janneke helped by others did an amazing and incredible work around
bootstrapping:
https://guix.gnu.org/en/blog/tags/bootstrapping/
https://archive.fosdem.org/2021/schedule/event/gnumes/
Bah auto-promo (again!) of some details I understood. Maybe it might
help in jumping deeper in the story. :-)
https://simon.tournier.info/posts/2023-10-01-bootstrapping.html
> doing so would be useful complementary work.
Now Julien has identified at scale the roadblocks, Guix folk need to
check them and fix if required.
Somehow, these fixes are part of the Reproducible Builds effort [1].
Food for thoughts, quoting Holger [2]: « many unreproducible builds are
not fully technical and fixing them means a policy – it’s a political
decision. »
That’s said, from my point of view, the next step is to rebuild at scale
using source code from Software Heritage [3]. Now, a bunch of
auto-promo. :-) The first brick is now in “production” [4,5], we need to
polish several hacks that had been identified, e.g.,
https://simon.tournier.info/posts/2023-06-23-hackathon-repro.html
https://simon.tournier.info/posts/2023-12-21-repro-paper.html
https://simon.tournier.info/posts/2024-04-11-rewrite-drv.html
One promising strategy is detailed here:
https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment/
All in all, the answers to your questions seems:
• Guix provides tools for comparing bit-to-bit reproducibility but
such comparison isn’t fully automated. The Guix Data Service helps
here for improving the story.
• Some of us are in touch with Julien, Théo and Stefano. :-)
Last but not least, feel free to join the “Guix in scientific context“
(namely GuixHPC) effort [6]. For instance, we are organizing regular
Café Guix [7] – informal French speaking discussion about Guix and
Reproducible Research.
And the real last, feel free to join the French Reproducible Research
network:
https://www.recherche-reproductible.fr
Cheers,
simon
1: https://reproducible-builds.org/ 2:
2: https://debconf24.debconf.org/talks/18-reproducible-builds-the-first-eleven-years/
3: https://www.softwareheritage.org/
4: https://dl.acm.org/doi/10.1145/3641525.3663622
5: https://fosdem.org/2025/schedule/event/fosdem-2025-5897-guix-software-heritage-source-code-archiving-to-the-rescue-of-reproducible-deployment/
6: https://hpc.guix.info/
7: https://hpc.guix.info/events/2024-2025/café-guix/
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-04 10:30 ` Hugo Buddelmeijer
2025-02-04 10:42 ` Alexis Praga via Guix-Science
@ 2025-02-04 12:15 ` Simon Tournier
1 sibling, 0 replies; 13+ messages in thread
From: Simon Tournier @ 2025-02-04 12:15 UTC (permalink / raw)
To: Hugo Buddelmeijer, Cayetano Santos; +Cc: Alexis Praga via Guix-Science, alexis
Hi Hugo,
On Tue, 04 Feb 2025 at 11:30, Hugo Buddelmeijer <hugo@buddelmeijer.nl> wrote:
> My conclusion is that the goal of GUIX is reproducibility, but that it
> is not guaranteed:
Well, I think there is a marketing issue. :-)
Julien explains it well in the blog post « Is NixOS truly
reproducible? » [1]:
Build reproducibility is often considered as a de facto feature provided
by functional package managers like Nix. Although the functional package
manager model has important assets in the quest for build
reproducibility (like reproducibility of build environments for
example1), it is clear among practitioners that Nix does not guarantee
that all its builds achieve bitwise reproducibility. In fact, it is not
complicated to write a Nix package that builds an artifact
non-deterministically:
let
pkgs = import <nixpkgs> { };
in
pkgs.runCommand "random" { } ''
echo $RANDOM > $out
''
Despite this, build reproducibility has historically been used as a
marketing argument by the NixOS community, with the catchphrase
“Reproducible builds and deployments” appearing as a headline of the
nixos.org page until 20232. This situation has even occasionally created
tensions with members of the reproducible-builds group who dedicate a
lot of time contributing patches in compilers and downstream projects to
make them bitwise reproducible for everyone, and prompted blog posts
such as “NixOS is not reproducible” by Foxboron.
And I recommend to read « NixOS is not reproducible » [2] by Foxboron.
The key point of Guix/Nix is the ability to deeply verify and/or trace
“how” the whole dependency graph is built.
The bit-to-bit reproducibility depends on the bit-to-bit reproducibility
of each node; yeah for sure, no magic! :-)
However, knowing one channels.scm file that pins a state (revision),
it’s easy to redeploy (reproduce) the same computational environment.
Here “same“ means using the exact same source code identified by SHA256
checksum, the exact same recipe (compiler options, etc.), whole
processed by the exact same driver (Guile scripts etc.), and that
recursively for all the graph of dependencies.
Hope that helps.
Cheers,
simon
1: https://luj.fr/blog/is-nixos-truly-reproducible.html
2: https://linderud.dev/blog/nixos-is-not-reproducible
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-04 11:02 ` Hugo Buddelmeijer
@ 2025-02-04 14:08 ` Cayetano Santos
0 siblings, 0 replies; 13+ messages in thread
From: Cayetano Santos @ 2025-02-04 14:08 UTC (permalink / raw)
To: Hugo Buddelmeijer; +Cc: alexis, Alexis Praga via Guix-Science
>mar. 04 févr. 2025 at 12:02, Hugo Buddelmeijer <hugo@buddelmeijer.nl> wrote:
> One thing I would like to add is that `guix build` has a `--rounds` option:
> https://guix.gnu.org/manual/en/html_node/Common-Build-Options.html
>
>> --rounds=n
Don’t forget the --no-drafts option if you build a local package to challenge:
guix build autogen --check --no-grafts
guix challenge autogen --diff=diffoscope --substitute-urls="https://bordeaux.guix.gnu.org https://ci.guix.gnu.org"
--
Cayetano Santos
cayetano.santos@cnrs.fr
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-04 11:45 ` Simon Tournier
@ 2025-02-04 14:26 ` Alexis Praga via Guix-Science
2025-02-06 14:12 ` Simon Tournier
0 siblings, 1 reply; 13+ messages in thread
From: Alexis Praga via Guix-Science @ 2025-02-04 14:26 UTC (permalink / raw)
To: Simon Tournier; +Cc: guix-science@gnu.org
[-- Attachment #1: Type: text/plain, Size: 6952 bytes --]
Hi Simon,
Thanks for you enthusiasm !
If the Guix folks are on it, great ! I think publishing the results
would be a major step and complement nicely your 2022 paper (which is
the best authorative resource on the subject I've found).
I can't wait for the results.
One question though : it looks like Guix and Nix are doing things
apart from the reproducible build effort. Is that correct ? Are there
plans to merge efforts ?
There are a lot to unpack in your mail, thanks for the links.
For your other remarks (this is not quite related to the main
discussion), the main blocking points in my settings are
1) a sysadmin willing to install the package manager
2) having everything ready to use by the bioinformaticians
3) a local "helpdesk" for nix/guix
Regarding 1), I chose Nix at the time based on the number of packages.
The sysadmins of our regional clusters agreed to test Nix so I went
this way. Otherwise, I would have most likely gone the Guix route.
At the moment, I'm packaging bioinformatics tools in Nix. Do you know
if there is a tool to convert a Nix derivation to Guix ?
Thanks for the links, I subscribed to the guix-related one as I was
already following the others.
Alexis
PS: thanks for following my talk ! This year, we did a Bird of a
Feather for bioinformatics at Fosdem but didn't get to talk about
reproducibility (yet).
On 2025-02-04T12:46:21.000+01:00, Simon Tournier
<zimon.toutoune@gmail.com> wrote:
> Hi Alexis,
>
> Thanks for your interest with Guix. :-)
>
> On Mon, 03 Feb 2025 at 22:28, Alexis Praga via Guix-Science <guix-science@gnu.org> wrote:
>
>> I have discovered the world of functional package managers 2
>> years ago
>>
>> through Nix and have since introduced Nix in my day job (running
>>
>> genetics pipeline on our local cluster).
>
> Yeah, your short talk at FOSDEM was nice!
>
> https://archive.fosdem.org/2024/schedule/event/fosdem-2024-3058-nix-for-genetics-powering-a-bioinformatics-pipeline/
>
> Well, my auto-promo ;-)
>
> https://www.nature.com/articles/s41597-022-01720-9
>
> A paper written by two Medical Doctors and one regular engineer to
>
> explain the “problem“. Maybe it might help in convincing your
>
> colleagues – if they are not already – that 1. The topic matters
> a lot
>
> (else all is built on the top of sand) and 2. Nix/Guix paves the way
> as
>
> tools helping in tackling such topic.
>
>> Julien Malka (Telecom Paris) published a paper a few days ago in
>>
>> which he checked the bitwise reproducibility of more than 700 000
>>
>> packages in nixpkgs : https://hal.science/hal-04913007
>
> Yeah, cool! Isn’t it?
>
> Guess what? We are in touch. :-) For instance, we’ve discussed
>
> yesterday on Mastodon what could be tried with Haskell.
>
> https://social.sciences.re/@luj@chaos.social/113941327662536682
>
>> I find this approach really interesting and was wondering if
>> something
>>
>> similar was possible (and maybe planned) for guix ?
>
> Yes something similar is possible with Guix. Run ‘guix build
> --check’
>
> and voilà… Euh?! On paper, it’s “and voilà” but in real
> world it’s more
>
> engineering than just that.
>
> In order to compare, Guix has some tooling as “guix challenge”
> or “guix
>
> build --check” and all the power of Guix as a Scheme library.
>
> Please note that Guix is also pushing the issue of bootstrapping.
>
> Janneke helped by others did an amazing and incredible work around
>
> bootstrapping:
>
> https://guix.gnu.org/en/blog/tags/bootstrapping/
>
> https://archive.fosdem.org/2021/schedule/event/gnumes/
>
> Bah auto-promo (again!) of some details I understood. Maybe it might
>
> help in jumping deeper in the story. :-)
>
> https://simon.tournier.info/posts/2023-10-01-bootstrapping.html
>
>> doing so would be useful complementary work.
>
> Now Julien has identified at scale the roadblocks, Guix folk need to
>
> check them and fix if required.
>
> Somehow, these fixes are part of the Reproducible Builds effort [1].
>
> Food for thoughts, quoting Holger [2]: « many unreproducible builds
> are
>
> not fully technical and fixing them means a policy – it’s a
> political
>
> decision. »
>
> That’s said, from my point of view, the next step is to rebuild at
> scale
>
> using source code from Software Heritage [3]. Now, a bunch of
>
> auto-promo. :-) The first brick is now in “production” [4,5], we
> need to
>
> polish several hacks that had been identified, e.g.,
>
> https://simon.tournier.info/posts/2023-06-23-hackathon-repro.html
>
> https://simon.tournier.info/posts/2023-12-21-repro-paper.html
>
> https://simon.tournier.info/posts/2024-04-11-rewrite-drv.html
>
> One promising strategy is detailed here:
>
> https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment/
>
> All in all, the answers to your questions seems:
>
> • Guix provides tools for comparing bit-to-bit reproducibility
> but
>
> such comparison isn’t fully automated. The Guix Data Service
> helps
>
> here for improving the story.
>
> • Some of us are in touch with Julien, Théo and Stefano. :-)
>
> Last but not least, feel free to join the “Guix in scientific
> context“
>
> (namely GuixHPC) effort [6]. For instance, we are organizing regular
>
> Café Guix [7] – informal French speaking discussion about Guix
> and
>
> Reproducible Research.
>
> And the real last, feel free to join the French Reproducible
> Research
>
> network:
>
> https://www.recherche-reproductible.fr
>
> Cheers,
>
> simon
>
> 1: https://reproducible-builds.org/ 2:
>
> 2:
> https://debconf24.debconf.org/talks/18-reproducible-builds-the-first-eleven-years/
>
> 3: https://www.softwareheritage.org/
>
> 4: https://dl.acm.org/doi/10.1145/3641525.3663622
>
> 5:
> https://fosdem.org/2025/schedule/event/fosdem-2025-5897-guix-software-heritage-source-code-archiving-to-the-rescue-of-reproducible-deployment/
>
> 6: https://hpc.guix.info/
>
> 7: https://hpc.guix.info/events/2024-2025/café-guix/
[-- Attachment #2: Type: text/html, Size: 13578 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-04 10:42 ` Alexis Praga via Guix-Science
2025-02-04 11:02 ` Hugo Buddelmeijer
@ 2025-02-04 14:37 ` Cayetano Santos
1 sibling, 0 replies; 13+ messages in thread
From: Cayetano Santos @ 2025-02-04 14:37 UTC (permalink / raw)
To: alexis; +Cc: Hugo Buddelmeijer, Alexis Praga via Guix-Science
[-- Attachment #1: Type: text/plain, Size: 704 bytes --]
>mar. 04 févr. 2025 at 11:42, alexis@praga.dev wrote:
> As a follow-up, does guix challenge only compares binaries between 2
> servers ? Or does it rebuild the derivation locally ?
>
> If that's the latter, I was wondering how feasible it would be to guix
> challenge on all packages in terms of storage and computational
> power...
You may try to challenge substitute servers with:
guix shell -DCPWN guix nss-certs -- \
guix challenge $(guix package -p $GUIX_PROFILE -I | awk '{print $1}') --verbose
which in an example profile (attached manifest) gives to me
48 store items were analyzed:
- 43 (89.6%) were identical
- 2 (4.2%) differed
- 3 (6.3%) were inconclusive
[-- Attachment #2: manifest --]
[-- Type: text/plain, Size: 1795 bytes --]
;; Ce fichier « manifeste » peut être passé à « guix package -m » pour
;; reproduire le contenu de votre profil. Son contenu est « symbolique » : il
;; ne spécifie que les noms des paquets. Pour reproduire exactement le même
;; profil, vous devez aussi retenir les canaux utilisés, ceux renvoyés par
;; « guix describe ». Voir la section « Répliquer Guix » dans le manuel.
(specifications->manifest
(list "python-ipdb"
"python-lsp-server"
"python-opt-einsum"
"python-tables"
"python-numexpr"
"python-rope"
"python-pytest-isort"
"python-flake8-isort"
"python-flake8-pyi"
"python-recommonmark"
"python-mistletoe"
"python-autoflake8"
"python-yapf"
"python-proselint"
"python-autopep8"
"python-grpcio"
"python-argcomplete"
"python-lsp-jsonrpc"
"python-pylint"
"python-isort"
"python-pydocstyle"
"python-wrapt"
"python"
"python-pytest"
"python-flake8"
"python-flake8-continuation"
"python-flake8-print"
"python-flake8-quotes"
"python-flake8-todo"
"python-flake8-bugbear"
"python-flake8-debugger"
"python-pytest-flake8"
"python-pytest-flakes"
"python-markdown"
"python-black"
"python-pycodestyle"
"python-pyflakes"
"python-pygments"
"python-flake8-pie"
"python-flake8-blind-except"
"python-flake8-implicit-str-concat"
"python-tenacity"
"python-termcolor"
"python-jedi"
"python-mccabe"
"python-pyasn1"
"python-pyasn1-modules"
"python-cachetools"))
[-- Attachment #3: Type: text/plain, Size: 45 bytes --]
--
Cayetano Santos
cayetano.santos@cnrs.fr
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-04 8:30 ` Cayetano Santos
2025-02-04 10:30 ` Hugo Buddelmeijer
@ 2025-02-04 16:10 ` Ludovic Courtès
2025-02-04 16:33 ` Cayetano Santos
1 sibling, 1 reply; 13+ messages in thread
From: Ludovic Courtès @ 2025-02-04 16:10 UTC (permalink / raw)
To: Cayetano Santos; +Cc: Alexis Praga via Guix-Science, alexis
Hello,
Cayetano Santos <cayetano.santos@cnrs.fr> skribis:
> I had exactly the same reaction when reading the paper, and found the ci
> testing reports in https://reproducible-builds.org/citests. Among them,
> guix (link is very slow to load):
>
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility
>
> which seems unmaintained / outdated.
The Guix Data Service is maintained, although largely by a single person
(we should give a hand!). Not only does it have data about package
reproducibility across the entire channel, it also has that information
for all commits (well, all commits that were pushed: there are holes in
between pushes).
Right now
<https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility>
shows little because the build farm at bordeaux.guix.gnu.org has been
lagging behind, so there’s few packages that can be compared with
ci.guix.gnu.org. Keep in mind that the “unknown” (gray) area represents
packages built on only one of the build farms, or none of them.
If you pick an older commit, you get more information:
https://data.guix.gnu.org/revision/20dbf225f332ccc707578263ed710dcf2a8fb78e/package-reproducibility
Here only 4% unknown on x86_64-linux, and 5% non reproducible.
You can follow the link to view non-reproducible packages:
https://data.guix.gnu.org/revision/20dbf225f332ccc707578263ed710dcf2a8fb78e/package-derivation-outputs?output_consistency=not-matching&system=x86_64-linux
HTH!
Ludo’.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-04 16:10 ` Ludovic Courtès
@ 2025-02-04 16:33 ` Cayetano Santos
0 siblings, 0 replies; 13+ messages in thread
From: Cayetano Santos @ 2025-02-04 16:33 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: Alexis Praga via Guix-Science, alexis
>mar. 04 févr. 2025 at 17:10, Ludovic Courtès <ludo@gnu.org> wrote:
> If you pick an older commit, you get more information:
> https://data.guix.gnu.org/revision/20dbf225f332ccc707578263ed710dcf2a8fb78e/package-reproducibility
>
> Here only 4% unknown on x86_64-linux, and 5% non reproducible.
>
> You can follow the link to view non-reproducible packages:
> https://data.guix.gnu.org/revision/20dbf225f332ccc707578263ed710dcf2a8fb78e/package-derivation-outputs?output_consistency=not-matching&system=x86_64-linux
>
> HTH!
A lot ! This is exactly what I was looking for, thanks !
Any tip on how to get/produce the same graphs with time, by the way ?
Best,
--
Cayetano Santos
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Proving bitwise reproducibility
2025-02-04 14:26 ` Alexis Praga via Guix-Science
@ 2025-02-06 14:12 ` Simon Tournier
0 siblings, 0 replies; 13+ messages in thread
From: Simon Tournier @ 2025-02-06 14:12 UTC (permalink / raw)
To: alexis; +Cc: guix-science@gnu.org
Hi Alexis,
On Tue, 04 Feb 2025 at 15:26, Alexis Praga via Guix-Science <guix-science@gnu.org> wrote:
> One question though : it looks like Guix and Nix are doing things
> apart from the reproducible build effort. Is that correct ? Are there
> plans to merge efforts ?
Well, from my point of view, having various flavours of the same topic
helps in tackling this very topic. For instance, give a look to all the
projects involved in the Reproducible Effort: it ranges from Alpine to
Yocto, via Debian or FreeBSD. And guess what? Guix and Nix too! :-)
https://reproducible-builds.org/who/projects/
Maybe I misread you. From my understanding, the efforts are already
“shared”. People working on Guix had attended to Reproducible Build
Summits. For instance, in Capitole du Libre in Toulouse on November, I
had a quick chat about Debian .buildinfo file which somehow has some
similarities with Guix .drv file.
https://guix.gnu.org/en/blog/2018/reproducible-builds-summit-4th-edition/
https://guix.gnu.org/en/blog/2019/reproducible-builds-summit-5th-edition/
https://guix.gnu.org/en/blog/tags/reproducible-builds//
> 1) a sysadmin willing to install the package manager
>
> 2) having everything ready to use by the bioinformaticians
>
> 3) a local "helpdesk" for nix/guix
Hire me! ;-) I’m kidding, I’m already happy with my day job. And
although I love Doubs, my partner is less fan. :-D
Joke aside, yeah these are some bottlenecks. But:
1. More and more sysadmin jumps in the adventure. :-) And more and
more sysadmins now constitutes a “network”: Grenoble, Bordeaux,
Nantes, Montpellier, etc.
2. Considering the R ecosystem, Guix is probably the best solution out
there. And since the move of the channel guix-science from GitHub
to CodeBerg, maybe we’ll see a boost: more specialized packages.
3. That’s one of the reasons for initiating Café Guix and the
Mattermost channel. Indeed one needs to accept to have the hand a
bit dirty, however all is improving day after day, IMHO.
> Regarding 1), I chose Nix at the time based on the number of packages.
> The sysadmins of our regional clusters agreed to test Nix so I went
> this way. Otherwise, I would have most likely gone the Guix route.
IMHO, the both routes are not orthogonal. :-)
> At the moment, I'm packaging bioinformatics tools in Nix. Do you know
> if there is a tool to convert a Nix derivation to Guix ?
To me knowledge, there is no interoperability. Maybe if tvix [1] come
to light: having a modular daemon which accepts different builders.
However, considering the implementation of the store, it would be very
hard to define a Nix expression that populates the store with one item
and then reuse that item from Guix. That’s because of the current
content-addressed store implementation.
1: https://tvix.dev
Cheers,
simon
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2025-02-06 16:27 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-02-03 21:28 Proving bitwise reproducibility Alexis Praga via Guix-Science
2025-02-04 8:30 ` Cayetano Santos
2025-02-04 10:30 ` Hugo Buddelmeijer
2025-02-04 10:42 ` Alexis Praga via Guix-Science
2025-02-04 11:02 ` Hugo Buddelmeijer
2025-02-04 14:08 ` Cayetano Santos
2025-02-04 14:37 ` Cayetano Santos
2025-02-04 12:15 ` Simon Tournier
2025-02-04 16:10 ` Ludovic Courtès
2025-02-04 16:33 ` Cayetano Santos
2025-02-04 11:45 ` Simon Tournier
2025-02-04 14:26 ` Alexis Praga via Guix-Science
2025-02-06 14:12 ` Simon Tournier
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).