* bug#42009: Determinism problem with guix pull
@ 2020-06-22 17:07 Marinus
2020-06-23 22:46 ` bug#42009: package.cache not deterministic zimoun
0 siblings, 1 reply; 12+ messages in thread
From: Marinus @ 2020-06-22 17:07 UTC (permalink / raw)
To: 42009
Hi,
Run into a determinism problem today with guix pull.
I run guix pull --rounds=3 but guix ended in error that the result
wasn't the same.
The error was this:
building package cache...
|output
‘/gnu/store/277s1r2kxw9pfw1g6wg3vf6wrkksj57y-guix-package-cache’ of
‘/gnu/store/m64b2g2h75xbbnrxxn3g1h4833i58yj4-guix-package-cache.drv’
differs from previous round build of
/gnu/store/m64b2g2h75xbbnrxxn3g1h4833i58yj4-guix-package-cache.drv
failed View build log at
'/var/log/guix/drvs/m6/4b2g2h75xbbnrxxn3g1h4833i58yj4-guix-package-cache.drv.bz2'.
cannot build derivation
`/gnu/store/bcpjm4yvslpf4858lx4pj89xj279z5nv-profile.drv': 1
dependencies couldn't be built guix pull: error: build of
`/gnu/store/bcpjm4yvslpf4858lx4pj89xj279z5nv-profile.drv' failed
The files are indeed different size with one being larger than the
other.
My system is this:
guix describe
Generation 7 Jun 18 2020 13:55:17 (current)
guix e418c3d
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: e418c3d076ec301a2deda42568035d75f5ed174d
Marinus
^ permalink raw reply [flat|nested] 12+ messages in thread* bug#42009: package.cache not deterministic 2020-06-22 17:07 bug#42009: Determinism problem with guix pull Marinus @ 2020-06-23 22:46 ` zimoun 2020-06-24 18:59 ` Msavoritias 2020-06-25 22:33 ` Tobias Geerinckx-Rice via Bug reports for GNU Guix 0 siblings, 2 replies; 12+ messages in thread From: zimoun @ 2020-06-23 22:46 UTC (permalink / raw) To: Marinus, 42009 Dear, Thank you for the bug report. It is something already noticed [1] but without a clean bug report to track it. :-) 1: http://issues.guix.gnu.org/issue/39258#86 On Mon, 22 Jun 2020 at 19:07, Marinus <marinus.savoritias@disroot.org> wrote: > Run into a determinism problem today with guix pull. > I run guix pull --rounds=3 but guix ended in error that the result > wasn't the same. For reproducing, the simplest is: --8<---------------cut here---------------start------------->8--- $ guix build --check --no-grafts -K \ $(guix gc --derivers \ $(readlink -f ~/.config/guix/current/lib/guix/package.cache)) The following profile hooks will be built: /gnu/store/l50sinckbl1y0fz2y4yk4vvfdvay9c8l-guix-package-cache.drv /gnu/store/h69hdf14c11q7dip0gssfd4cv0qw8j7k-guix-package-cache.drv building package cache... (repl-version 0 1 1) Generating package cache for '/gnu/store/67zi87xwv2d90kx8pzxsbw2q7qkh11ns-profile'... (values (value "/gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache")) guix build: error: derivation `/gnu/store/h69hdf14c11q7dip0gssfd4cv0qw8j7k-guix-package-cache.drv' may not be deterministic: output `/gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache' differs --8<---------------cut here---------------end--------------->8--- Then the usual "diffoscope": --8<---------------cut here---------------start------------->8--- diffoscope \ /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache \ /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache-check/lib/guix/package.cache\ | head -n50 --8<---------------cut here---------------end--------------->8--- outputs something like: --8<---------------cut here---------------start------------->8--- --- /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache +++ /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache-check/lib/guix/package.cache ├── readelf --wide --file-header {} │ @@ -6,15 +6,15 @@ │ OS/ABI: <unknown: ff> │ ABI Version: 0 │ Type: DYN (Shared object file) │ Machine: None │ Version: 0x1 │ Entry point address: 0x0 │ Start of program headers: 64 (bytes into file) │ - Start of section headers: 4900296 (bytes into file) │ + Start of section headers: 4900456 (bytes into file) │ Flags: 0x0 │ Size of this header: 64 (bytes) │ Size of program headers: 56 (bytes) │ Number of program headers: 3 │ Size of section headers: 64 (bytes) │ Number of section headers: 20 │ Section header string table index: 17 ├── readelf --wide --program-header {} │ @@ -1,16 +1,16 @@ │ │ Elf file type is DYN (Shared object file) │ Entry point 0x0 │ There are 3 program headers, starting at offset 64 │ │ Program Headers: │ Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align │ - LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x286a68 0x286a68 R 0x10000 │ - LOAD 0x290000 0x0000000000290000 0x0000000000290000 0x21c5c8 0x21c5c8 RW 0x10000 │ - DYNAMIC 0x286a08 0x0000000000286a08 0x0000000000286a08 0x000060 0x000060 R 0x8 │ + LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x286b78 0x286b78 R 0x10000 │ + LOAD 0x290000 0x0000000000290000 0x0000000000290000 0x21c668 0x21c668 RW 0x10000 │ + DYNAMIC 0x286b18 0x0000000000286b18 0x0000000000286b18 0x000060 0x000060 R 0x8 │ │ Section to Segment mapping: │ Segment Sections... │ 00 .rodata .rtl-text .dynamic │ 01 .data │ 02 .dynamic ├── readelf --wide --sections {} │┄ stderr from `readelf --wide --sections {}`: │┄ readelf: Warning: [ 5]: Link field (0) should index a string section. │ @@ -1,29 +1,29 @@ │ -There are 20 section headers, starting at offset 0x4ac5c8: │ +There are 20 section headers, starting at offset 0x4ac668: --8<---------------cut here---------------end--------------->8--- Well, I do not know what should the next step. I mean this "package.cache" file is created by the function gnu/packages.scm:(generate-package-cache) which reads: --8<---------------cut here---------------start------------->8--- ;; Store the cache as a '.go' file. This makes loading fast and reduces ;; heap usage since some of the static data is directly mmapped. (put-bytevector port (compile `'(,@exp) #:to 'bytecode #:opts '(#:to-file? #t))))) --8<---------------cut here---------------end--------------->8--- Then it is on the Guile side, isn't it? All the best, simon ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#42009: package.cache not deterministic 2020-06-23 22:46 ` bug#42009: package.cache not deterministic zimoun @ 2020-06-24 18:59 ` Msavoritias 2020-06-25 9:04 ` zimoun 2020-06-25 22:33 ` Tobias Geerinckx-Rice via Bug reports for GNU Guix 1 sibling, 1 reply; 12+ messages in thread From: Msavoritias @ 2020-06-24 18:59 UTC (permalink / raw) To: zimoun; +Cc: 42009 [-- Attachment #1: Type: text/plain, Size: 5683 bytes --] Hi, Yeah seems like it. Although I have to admit I'm pretty newbie in a lot of this stuff. Do you suggest this bug should be filed against Guile? Marinus On Wed, Jun 24, 2020 at 00:46, zimoun <zimon.toutoune@gmail.com> wrote: > Dear, > > Thank you for the bug report. It is something already noticed [1] but > without a clean bug report to track it. :-) > > 1: <http://issues.guix.gnu.org/issue/39258#86> > > > On Mon, 22 Jun 2020 at 19:07, Marinus <marinus.savoritias@disroot.org > <mailto:marinus.savoritias@disroot.org>> wrote: > >> Run into a determinism problem today with guix pull. >> I run guix pull --rounds=3 but guix ended in error that the result >> wasn't the same. > > For reproducing, the simplest is: > > --8<---------------cut here---------------start------------->8--- > $ guix build --check --no-grafts -K \ > $(guix gc --derivers \ > $(readlink -f ~/.config/guix/current/lib/guix/package.cache)) > The following profile hooks will be built: > /gnu/store/l50sinckbl1y0fz2y4yk4vvfdvay9c8l-guix-package-cache.drv > /gnu/store/h69hdf14c11q7dip0gssfd4cv0qw8j7k-guix-package-cache.drv > building package cache... > (repl-version 0 1 1) > Generating package cache for > '/gnu/store/67zi87xwv2d90kx8pzxsbw2q7qkh11ns-profile'... > (values (value > "/gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache")) > guix build: error: derivation > `/gnu/store/h69hdf14c11q7dip0gssfd4cv0qw8j7k-guix-package-cache.drv' > may not be deterministic: output > `/gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache' > differs > --8<---------------cut here---------------end--------------->8--- > > Then the usual "diffoscope": > > --8<---------------cut here---------------start------------->8--- > diffoscope \ > /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache > \ > /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache-check/lib/guix/package.cache\ > | head -n50 > --8<---------------cut here---------------end--------------->8--- > > outputs something like: > > --8<---------------cut here---------------start------------->8--- > --- > /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache/lib/guix/package.cache > +++ > /gnu/store/0009whxhfz00lm027wbars8q4wb3rvia-guix-package-cache-check/lib/guix/package.cache > ├── readelf --wide --file-header {} > │ @@ -6,15 +6,15 @@ > │ OS/ABI: <unknown: ff> > │ ABI Version: 0 > │ Type: DYN (Shared object file) > │ Machine: None > │ Version: 0x1 > │ Entry point address: 0x0 > │ Start of program headers: 64 (bytes into file) > │ - Start of section headers: 4900296 (bytes into file) > │ + Start of section headers: 4900456 (bytes into file) > │ Flags: 0x0 > │ Size of this header: 64 (bytes) > │ Size of program headers: 56 (bytes) > │ Number of program headers: 3 > │ Size of section headers: 64 (bytes) > │ Number of section headers: 20 > │ Section header string table index: 17 > ├── readelf --wide --program-header {} > │ @@ -1,16 +1,16 @@ > │ > │ Elf file type is DYN (Shared object file) > │ Entry point 0x0 > │ There are 3 program headers, starting at offset 64 > │ > │ Program Headers: > │ Type Offset VirtAddr PhysAddr > FileSiz MemSiz Flg Align > │ - LOAD 0x000000 0x0000000000000000 0x0000000000000000 > 0x286a68 0x286a68 R 0x10000 > │ - LOAD 0x290000 0x0000000000290000 0x0000000000290000 > 0x21c5c8 0x21c5c8 RW 0x10000 > │ - DYNAMIC 0x286a08 0x0000000000286a08 0x0000000000286a08 > 0x000060 0x000060 R 0x8 > │ + LOAD 0x000000 0x0000000000000000 0x0000000000000000 > 0x286b78 0x286b78 R 0x10000 > │ + LOAD 0x290000 0x0000000000290000 0x0000000000290000 > 0x21c668 0x21c668 RW 0x10000 > │ + DYNAMIC 0x286b18 0x0000000000286b18 0x0000000000286b18 > 0x000060 0x000060 R 0x8 > │ > │ Section to Segment mapping: > │ Segment Sections... > │ 00 .rodata .rtl-text .dynamic > │ 01 .data > │ 02 .dynamic > ├── readelf --wide --sections {} > │┄ stderr from `readelf --wide --sections {}`: > │┄ readelf: Warning: [ 5]: Link field (0) should index a string > section. > │ @@ -1,29 +1,29 @@ > │ -There are 20 section headers, starting at offset 0x4ac5c8: > │ +There are 20 section headers, starting at offset 0x4ac668: > --8<---------------cut here---------------end--------------->8--- > > Well, I do not know what should the next step. I mean this > "package.cache" file is created by the function > gnu/packages.scm:(generate-package-cache) which reads: > > --8<---------------cut here---------------start------------->8--- > ;; Store the cache as a '.go' file. This makes loading fast > and reduces > ;; heap usage since some of the static data is directly mmapped. > (put-bytevector port > (compile `'(,@exp) > #:to 'bytecode > #:opts '(#:to-file? #t))))) > --8<---------------cut here---------------end--------------->8--- > > Then it is on the Guile side, isn't it? > > > All the best, > simon [-- Attachment #2: Type: text/html, Size: 5778 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#42009: package.cache not deterministic 2020-06-24 18:59 ` Msavoritias @ 2020-06-25 9:04 ` zimoun 2020-06-25 14:53 ` Msavoritias 0 siblings, 1 reply; 12+ messages in thread From: zimoun @ 2020-06-25 9:04 UTC (permalink / raw) To: Msavoritias; +Cc: 42009 Dear, On Wed, 24 Jun 2020 at 20:59, Msavoritias <marinus.savoritias@disroot.org> wrote: > Do you suggest this bug should be filed against Guile? Well, first the origin of the bug should be confirmed. :-) And maybe there is an option for 'compile'. I do not know. All the best, simon ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#42009: package.cache not deterministic 2020-06-25 9:04 ` zimoun @ 2020-06-25 14:53 ` Msavoritias 0 siblings, 0 replies; 12+ messages in thread From: Msavoritias @ 2020-06-25 14:53 UTC (permalink / raw) To: zimoun; +Cc: 42009 [-- Attachment #1: Type: text/plain, Size: 504 bytes --] Is there any tests or any more information I can do on my side? Marinus On Thu, Jun 25, 2020 at 11:04, zimoun <zimon.toutoune@gmail.com> wrote: > Dear, > > On Wed, 24 Jun 2020 at 20:59, Msavoritias > <marinus.savoritias@disroot.org > <mailto:marinus.savoritias@disroot.org>> wrote: > >> Do you suggest this bug should be filed against Guile? > > Well, first the origin of the bug should be confirmed. :-) > And maybe there is an option for 'compile'. I do not know. > > All the best, > simon [-- Attachment #2: Type: text/html, Size: 737 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#42009: package.cache not deterministic 2020-06-23 22:46 ` bug#42009: package.cache not deterministic zimoun 2020-06-24 18:59 ` Msavoritias @ 2020-06-25 22:33 ` Tobias Geerinckx-Rice via Bug reports for GNU Guix 2020-06-25 23:19 ` zimoun 1 sibling, 1 reply; 12+ messages in thread From: Tobias Geerinckx-Rice via Bug reports for GNU Guix @ 2020-06-25 22:33 UTC (permalink / raw) To: Marinus; +Cc: 42009, zimoun [-- Attachment #1: Type: text/plain, Size: 340 bytes --] Quick and lazy reply before the sleep comes, Seems like a ‘simple’ case of the packages not being ordered (enough): https://www.tobias.gr/guix-package.cache.diffoscope.html/#lib---guix---package.cache---strings---all--- (Sorry for the ad-hoc URL, I'll keep it up until this bug report is closed!) Kind regards, T G-R [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 227 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#42009: package.cache not deterministic 2020-06-25 22:33 ` Tobias Geerinckx-Rice via Bug reports for GNU Guix @ 2020-06-25 23:19 ` zimoun 2020-06-28 20:29 ` Ludovic Courtès 0 siblings, 1 reply; 12+ messages in thread From: zimoun @ 2020-06-25 23:19 UTC (permalink / raw) To: Tobias Geerinckx-Rice, Marinus; +Cc: 42009 Hi Tobias, On Fri, 26 Jun 2020 at 00:33, Tobias Geerinckx-Rice <me@tobias.gr> wrote: > Seems like a ‘simple’ case of the packages not being ordered > (enough): So one culprit would be 'scandir*' from 'scheme-files' used by 'scheme-modules' used by 'all-modules' used by 'generate-package-cache'. The docstring says: "The returned list is sorted in alphabetical order." Well, I will try to investigate more. > https://www.tobias.gr/guix-package.cache.diffoscope.html/#lib---guix---package.cache---strings---all--- Interesting... Good night, simon ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#42009: package.cache not deterministic 2020-06-25 23:19 ` zimoun @ 2020-06-28 20:29 ` Ludovic Courtès 2020-06-29 10:03 ` zimoun 0 siblings, 1 reply; 12+ messages in thread From: Ludovic Courtès @ 2020-06-28 20:29 UTC (permalink / raw) To: zimoun; +Cc: 42009 Hi, Most likely the problem with non-reproducible .go files is that the fix for <https://bugs.gnu.org/20272> was incomplete. In particular, I think that gensyms are not reproducible when building things in parallel, because the gensym depends on what’s loaded vs. interpreted. Example of a simpler discrepancy: --8<---------------cut here---------------start------------->8--- $ guix challenge guile-gcrypt /gnu/store/i6zzgsjjlqyfn9zmanb55hqdackhc0mf-guile-gcrypt-0.3.0 contents differ: local hash: 0bkxim5532bd234iqxvxd31ybp6v7rmbrvyj3jrgpmgxbsw7hmxj https://ci.guix.gnu.org/nar/lzip/i6zzgsjjlqyfn9zmanb55hqdackhc0mf-guile-gcrypt-0.3.0: 00zsccxn1485bnvxvssgq50i2w69i38m9hnrk8z1sji9qcmmaxcs differing files: /lib/guile/3.0/site-ccache/gcrypt/pk-crypto.go /lib/guile/3.0/site-ccache/gcrypt/mac.go --8<---------------cut here---------------end--------------->8--- HTH! Ludo’. ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#42009: package.cache not deterministic 2020-06-28 20:29 ` Ludovic Courtès @ 2020-06-29 10:03 ` zimoun 2020-06-29 12:34 ` Ludovic Courtès 0 siblings, 1 reply; 12+ messages in thread From: zimoun @ 2020-06-29 10:03 UTC (permalink / raw) To: Ludovic Courtès; +Cc: 42009 Hi Ludo, On Sun, 28 Jun 2020 at 22:29, Ludovic Courtès <ludo@gnu.org> wrote: > Most likely the problem with non-reproducible .go files is that the fix > for <https://bugs.gnu.org/20272> was incomplete. In particular, I think > that gensyms are not reproducible when building things in parallel, > because the gensym depends on what’s loaded vs. interpreted. Thank you for the pointer. How can I test the "hypothesis" of "building things in parallel"? --8<---------------cut here---------------start------------->8--- guix describe -f channels > /tmp/chan.scm guix pull -C /tmp/chan.scm --cores=1 -p /tmp/repull1 guix build --check --no-grafts --cores=1 \ $(guix gc --derivers \ $(readlink -f /tmp/repull1/lib/guix/package.cache)) The following profile hook will be built: /gnu/store/qbrgxbnx0hi13xm36a6a0zijzc1rcz22-guix-package-cache.drv building package cache... (repl-version 0 1 1) Generating package cache for '/gnu/store/bgqy3mfpzbpyz3pysqxzkpch39q98yv3-profile'... (values (value "/gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache/lib/guix/package.cache")) guix build: error: derivation `/gnu/store/qbrgxbnx0hi13xm36a6a0zijzc1rcz22-guix-package-cache.drv' may not be deterministic: output `/gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache' differs --8<---------------cut here---------------end--------------->8--- BTW, I do not understand why the derivations have different hashes, containing derivations with different hashes and more importantly, why it is not the same order. --8<---------------cut here---------------start------------->8--- guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache) /gnu/store/0pmc85ni7zsd5jrflb0prrj7bhvn1m1y-guix-package-cache.drv cat $(guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache)) Derive ([("out","/gnu/store/pfpbh4v1m2dgn9dwiz6rsbqgx8lmd3ms-guix-package-cache","","")] ,[("/gnu/store/3pkfaqkdkaqy8khsfbsl0si3r9mydygl-profile.drv",["out"]) ,("/gnu/store/nih4g42d2da8p2b5dmxqb081bbpv9ax4-inferior-script.scm.drv",["out"]) ,("/gnu/store/x32cnfkd50fnxs10xp1jdn24h7ai2gxr-guile-3.0.2.drv",["out"])] ,["/gnu/store/50h7d8cx9k28gdbdzc9y615d1564m8ia-guix-package-cache-builder"] ,"x86_64-linux","/gnu/store/0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.0.2/bin/guile",["--no-auto-compile","/gnu/store/50h7d8cx9k28gdbdzc9y615d1564m8ia-guix-package-cache-builder"] ,[("guix properties","((type . profile-hook) (hook . package-cache))") ,("out","/gnu/store/pfpbh4v1m2dgn9dwiz6rsbqgx8lmd3ms-guix-package-cache") ,("preferLocalBuild","1")]) --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- guix gc --derivers $(readlink -f /tmp/repull1/lib/guix/package.cache) /gnu/store/qbrgxbnx0hi13xm36a6a0zijzc1rcz22-guix-package-cache.drv cat $(guix gc --derivers $(readlink -f /tmp/repull1/lib/guix/package.cache)) Derive ([("out","/gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache","","")] ,[("/gnu/store/b4dcaccqli2zdfalrn0lc0cz94gd80sk-inferior-script.scm.drv",["out"]) ,("/gnu/store/hm03mwl234lw43ivx33nsap0j4pjwqjp-profile.drv",["out"]) ,("/gnu/store/x32cnfkd50fnxs10xp1jdn24h7ai2gxr-guile-3.0.2.drv",["out"])] ,["/gnu/store/251jkjnw9zza2zwr1k45x1049d1axl5q-guix-package-cache-builder"] ,"x86_64-linux","/gnu/store/0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.0.2/bin/guile",["--no-auto-compile","/gnu/store/251jkjnw9zza2zwr1k45x1049d1axl5q-guix-package-cache-builder"] ,[("guix properties","((type . profile-hook) (hook . package-cache))") ,("out","/gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache") ,("preferLocalBuild","1")]) --8<---------------cut here---------------end--------------->8--- And I am confused because if I repull again with '--cores=2', then, --8<---------------cut here---------------start------------->8--- /tmp/repull1/bin/guix pull -C /tmp/chan.scm --cores=2 -p /tmp/repull2 md5sum \ $(readlink -f /tmp/repull2/lib/guix/package.cache) \ $(readlink -f /tmp/repull2/lib/guix/package.cache) 75f6feb9f52c312cc9cc8f73534926ba /gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache/lib/guix/package.cache 75f6feb9f52c312cc9cc8f73534926ba /gnu/store/15nnwjqrmh5w9hqy9yp4ycxsyfbsr0wi-guix-package-cache/lib/guix/package.cache --8<---------------cut here---------------end--------------->8--- But '--check' fails in all cases. What do I miss? All the best, simon ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#42009: package.cache not deterministic 2020-06-29 10:03 ` zimoun @ 2020-06-29 12:34 ` Ludovic Courtès 2020-06-29 15:39 ` zimoun 2020-07-30 17:22 ` Ludovic Courtès 0 siblings, 2 replies; 12+ messages in thread From: Ludovic Courtès @ 2020-06-29 12:34 UTC (permalink / raw) To: zimoun; +Cc: 42009 [-- Attachment #1: Type: text/plain, Size: 2070 bytes --] Hi Simon, zimoun <zimon.toutoune@gmail.com> skribis: > On Sun, 28 Jun 2020 at 22:29, Ludovic Courtès <ludo@gnu.org> wrote: > >> Most likely the problem with non-reproducible .go files is that the fix >> for <https://bugs.gnu.org/20272> was incomplete. In particular, I think >> that gensyms are not reproducible when building things in parallel, >> because the gensym depends on what’s loaded vs. interpreted. > > Thank you for the pointer. > > How can I test the "hypothesis" of "building things in parallel"? > > guix describe -f channels > /tmp/chan.scm > guix pull -C /tmp/chan.scm --cores=1 -p /tmp/repull1 > > guix build --check --no-grafts --cores=1 \ > $(guix gc --derivers \ > $(readlink -f /tmp/repull1/lib/guix/package.cache)) > The following profile hook will be built: > /gnu/store/qbrgxbnx0hi13xm36a6a0zijzc1rcz22-guix-package-cache.drv I realize I was a bit off-topic: I was commenting on the more general issue of .go non-reproducibility. The problem with the package cache seems to be different. Sorry for the confusion! After --check, we can compare both caches like this: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> (define a (load-compiled "/gnu/store/0yd3kaar87zyxhbrjqjypp5rar3zj4gb-guix-package-cache/lib/guix/package.cache")) scheme@(guile-user)> (define b (load-compiled "/gnu/store/0yd3kaar87zyxhbrjqjypp5rar3zj4gb-guix-package-cache-check/lib/guix/package.cache")) scheme@(guile-user)> (length a) $2 = 13949 scheme@(guile-user)> (length b) $3 = 13949 scheme@(guile-user)> ,use(srfi srfi-1) scheme@(guile-user)> (lset= equal? a b) $4 = #f scheme@(guile-user)> (car (lset-difference equal? a b)) $5 = #("python2" "2.7.17" (gnu packages python) python-2 ("out" "tk") #t #f "gnu/packages/python.scm" 107 2) --8<---------------cut here---------------end--------------->8--- So, surprisingly, it’s not just an ordering issue: the caches do contain different pieces of information. This patch solves the ordering issue: [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: Type: text/x-patch, Size: 1320 bytes --] diff --git a/gnu/packages.scm b/gnu/packages.scm index d22c992bb1..5154d73deb 100644 --- a/gnu/packages.scm +++ b/gnu/packages.scm @@ -407,13 +407,25 @@ reducing the memory footprint." (_ result+seen))) + (define entry-key + (match-lambda + (#(name version module symbol outputs supported? deprecated? + file line column) + (string-append name version (or file "") + (if line (number->string line) ""))))) + + (define (entry<? a b) + (string<? (entry-key a) (entry-key b))) + (define exp - (first - (fold-module-public-variables* expand-cache - (cons '() vlist-null) - (all-modules (%package-module-path) - #:warn - warn-about-load-error)))) + (sort + (first + (fold-module-public-variables* expand-cache + (cons '() vlist-null) + (all-modules (%package-module-path) + #:warn + warn-about-load-error))) + entry<?)) (mkdir-p (dirname cache-file)) (call-with-output-file cache-file [-- Attachment #3: Type: text/plain, Size: 191 bytes --] But it’s not quite right because the order in which variables are traversed is still non-deterministic, so between two runs of ‘generate-package-cache’, caches differ like this: [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #4: Type: text/x-patch, Size: 1796 bytes --] --- a 2020-06-29 14:27:32.291028456 +0200 +++ b 2020-06-29 14:27:37.930993059 +0200 @@ -8581,7 +8581,7 @@ #("clang-runtime" "9.0.1" (gnu packages llvm) - clang-runtime + clang-runtime-9 ("out") #t #f @@ -26511,7 +26511,7 @@ #("gcc-objc++" "7.5.0" (gnu packages gcc) - gcc-objc++-7 + gcc-objc++ ("out" "lib" "debug") #t #f @@ -26641,7 +26641,7 @@ #("gcc-toolchain" "7.5.0" (gnu packages commencement) - gcc-toolchain + gcc-toolchain-7 ("out" "debug" "static") #t #f @@ -33311,7 +33311,7 @@ #("ghc" "8.6.5" (gnu packages haskell) - ghc-8.6 + ghc-8 ("out" "doc") #t #f @@ -41876,7 +41876,7 @@ #("icedtea" "3.7.0" (gnu packages java) - icedtea-8 + icedtea ("out" "jdk" "doc") #t #f @@ -54376,7 +54376,7 @@ #("linux-libre-headers" "5.4.20" (gnu packages linux) - linux-libre-headers-5.4.20 + linux-libre-headers ("out") #t #f @@ -54636,7 +54636,7 @@ #("llvm" "9.0.1" (gnu packages llvm) - llvm-9 + llvm ("out" "opt-viewer") #t #f @@ -61826,7 +61826,7 @@ #("ocaml" "4.09.0" (gnu packages ocaml) - ocaml + ocaml-4.09 ("out") #t #f @@ -62256,7 +62256,7 @@ #("opencl-headers" "2.2.0-0.e986688" (gnu packages opencl) - opencl-headers + opencl-headers-2.2 ("out") #t #f @@ -92636,7 +92636,7 @@ #("python2" "2.7.17" (gnu packages python) - python-2 + python-2.7 ("out" "tk") #t #f @@ -92646,7 +92646,7 @@ #("python" "3.8.2" (gnu packages python) - python-3 + python ("out" "tk") #t #f @@ -123676,7 +123676,7 @@ #("rust" "1.39.0" (gnu packages rust) - rust-1.39 + rust ("out" "doc" "cargo") #t #f [-- Attachment #5: Type: text/plain, Size: 176 bytes --] The problem has to do with aliases: we don’t always see the same variable first. So we have to sort before calling ‘expand-cache’. To be continued… Ludo’. ^ permalink raw reply related [flat|nested] 12+ messages in thread
* bug#42009: package.cache not deterministic 2020-06-29 12:34 ` Ludovic Courtès @ 2020-06-29 15:39 ` zimoun 2020-07-30 17:22 ` Ludovic Courtès 1 sibling, 0 replies; 12+ messages in thread From: zimoun @ 2020-06-29 15:39 UTC (permalink / raw) To: Ludovic Courtès; +Cc: 42009 Hi Ludo, On Mon, 29 Jun 2020 at 14:34, Ludovic Courtès <ludo@gnu.org> wrote: > I realize I was a bit off-topic: I was commenting on the more general > issue of .go non-reproducibility. > > The problem with the package cache seems to be different. Sorry for the > confusion! Well, the .go non-reproducibility is nonetheless interesting. :-) > So, surprisingly, it’s not just an ordering issue: the caches do contain > different pieces of information. Interestingly, on my machine with 8ea91d0, I have 12 differences but none of them is Python. Instead, I have: "rust", "ocaml", "mingw-w64-i686", "clang-toolchain", "clang-runtime", "linux-libre", "icedtea", "gcc-objc++", "gcc-objc", "bdb", "rust-lazy-static", "gcc-toolchain". All are aliases and python-2 is too but does not appear... On the other hand, these 2 aliases do not appear either in your list. --8<---------------cut here---------------start------------->8--- (define-public clang clang-9) (define-public clang-toolchain clang-toolchain-9) --8<---------------cut here---------------end--------------->8--- The question is: do we need to declare public e.g. clang-9? Declare clang is it not enough already? For example, ocaml-4.09 is not used outside gnu/packages/ocalm.scm. Idem for ghc-8.6, etc.. > This patch solves the ordering issue The 'sort' will not help for improving the speed of "guix pull". ;-) > But it’s not quite right because the order in which variables are > traversed is still non-deterministic, so between two runs of > ‘generate-package-cache’, caches differ like this: It depends on the eddy current in the upper atmosphere. :-) https://xkcd.com/378/ Well, it finds one or the other first and 'expand-cache' stores the first considering the second as "seen", isn't it? > The problem has to do with aliases: we don’t always see the same > variable first. So we have to sort before calling ‘expand-cache’. The question is why it is not always the same variable first. Well, IMHO, it could comes from: - either 'scandir*' in 'scheme-files' because all the other functions seem "pure" and this one not; - either 'fold-module-public-variables*' where 'seen' is one or the other. Or if the .go files are not deterministic, especially about 'gensym', it should explain why one symbol appears sometimes first. Cheers, simon ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#42009: package.cache not deterministic 2020-06-29 12:34 ` Ludovic Courtès 2020-06-29 15:39 ` zimoun @ 2020-07-30 17:22 ` Ludovic Courtès 1 sibling, 0 replies; 12+ messages in thread From: Ludovic Courtès @ 2020-07-30 17:22 UTC (permalink / raw) To: zimoun; +Cc: 42009-done Hi, Ludovic Courtès <ludo@gnu.org> skribis: > But it’s not quite right because the order in which variables are > traversed is still non-deterministic, so between two runs of > ‘generate-package-cache’, caches differ like this: > > --- a 2020-06-29 14:27:32.291028456 +0200 > +++ b 2020-06-29 14:27:37.930993059 +0200 > @@ -8581,7 +8581,7 @@ > #("clang-runtime" > "9.0.1" > (gnu packages llvm) > - clang-runtime > + clang-runtime-9 > ("out") > #t > #f > @@ -26511,7 +26511,7 @@ > #("gcc-objc++" > "7.5.0" > (gnu packages gcc) > - gcc-objc++-7 > + gcc-objc++ > ("out" "lib" "debug") > #t > #f > @@ -26641,7 +26641,7 @@ > #("gcc-toolchain" > "7.5.0" > (gnu packages commencement) > - gcc-toolchain > + gcc-toolchain-7 > ("out" "debug" "static") > #t > #f > @@ -33311,7 +33311,7 @@ > #("ghc" > "8.6.5" > (gnu packages haskell) > - ghc-8.6 > + ghc-8 > ("out" "doc") > #t > #f > @@ -41876,7 +41876,7 @@ > #("icedtea" > "3.7.0" > (gnu packages java) > - icedtea-8 > + icedtea > ("out" "jdk" "doc") > #t > #f > @@ -54376,7 +54376,7 @@ > #("linux-libre-headers" > "5.4.20" > (gnu packages linux) > - linux-libre-headers-5.4.20 > + linux-libre-headers > ("out") > #t > #f > @@ -54636,7 +54636,7 @@ > #("llvm" > "9.0.1" > (gnu packages llvm) > - llvm-9 > + llvm > ("out" "opt-viewer") > #t > #f > @@ -61826,7 +61826,7 @@ > #("ocaml" > "4.09.0" > (gnu packages ocaml) > - ocaml > + ocaml-4.09 > ("out") > #t > #f > @@ -62256,7 +62256,7 @@ > #("opencl-headers" > "2.2.0-0.e986688" > (gnu packages opencl) > - opencl-headers > + opencl-headers-2.2 > ("out") > #t > #f > @@ -92636,7 +92636,7 @@ > #("python2" > "2.7.17" > (gnu packages python) > - python-2 > + python-2.7 > ("out" "tk") > #t > #f > @@ -92646,7 +92646,7 @@ > #("python" > "3.8.2" > (gnu packages python) > - python-3 > + python > ("out" "tk") > #t > #f > @@ -123676,7 +123676,7 @@ > #("rust" > "1.39.0" > (gnu packages rust) > - rust-1.39 > + rust > ("out" "doc" "cargo") > #t > #f > > > The problem has to do with aliases: we don’t always see the same > variable first. So we have to sort before calling ‘expand-cache’. Fixed in a127e52f601ee73f675d5d28eac2388bb1ad11b0! Ludo’. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2020-07-30 17:24 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-06-22 17:07 bug#42009: Determinism problem with guix pull Marinus 2020-06-23 22:46 ` bug#42009: package.cache not deterministic zimoun 2020-06-24 18:59 ` Msavoritias 2020-06-25 9:04 ` zimoun 2020-06-25 14:53 ` Msavoritias 2020-06-25 22:33 ` Tobias Geerinckx-Rice via Bug reports for GNU Guix 2020-06-25 23:19 ` zimoun 2020-06-28 20:29 ` Ludovic Courtès 2020-06-29 10:03 ` zimoun 2020-06-29 12:34 ` Ludovic Courtès 2020-06-29 15:39 ` zimoun 2020-07-30 17:22 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).