unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / Atom feed
From: "Ludovic Courtès" <ludo@gnu.org>
To: zimoun <zimon.toutoune@gmail.com>
Cc: 42009@debbugs.gnu.org
Subject: bug#42009: package.cache not deterministic
Date: Mon, 29 Jun 2020 14:34:40 +0200	[thread overview]
Message-ID: <875zbacaov.fsf@gnu.org> (raw)
In-Reply-To: <86o8p2b34l.fsf@gmail.com> (zimoun's message of "Mon, 29 Jun 2020 12:03:22 +0200")

[-- Attachment #1: Type: text/plain, Size: 2070 bytes --]

Hi Simon,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sun, 28 Jun 2020 at 22:29, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Most likely the problem with non-reproducible .go files is that the fix
>> for <https://bugs.gnu.org/20272> was incomplete.  In particular, I think
>> that gensyms are not reproducible when building things in parallel,
>> because the gensym depends on what’s loaded vs. interpreted.
>
> Thank you for the pointer.
>
> How can I test the "hypothesis" of "building things in parallel"?
>
> guix describe -f channels > /tmp/chan.scm
> guix pull -C /tmp/chan.scm --cores=1 -p /tmp/repull1
>
> guix build --check --no-grafts --cores=1 \
>      $(guix gc --derivers \
>             $(readlink -f /tmp/repull1/lib/guix/package.cache))
> The following profile hook will be built:
>    /gnu/store/qbrgxbnx0hi13xm36a6a0zijzc1rcz22-guix-package-cache.drv

I realize I was a bit off-topic: I was commenting on the more general
issue of .go non-reproducibility.

The problem with the package cache seems to be different.  Sorry for the
confusion!

After --check, we can compare both caches like this:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (define a (load-compiled "/gnu/store/0yd3kaar87zyxhbrjqjypp5rar3zj4gb-guix-package-cache/lib/guix/package.cache"))
scheme@(guile-user)> (define b (load-compiled "/gnu/store/0yd3kaar87zyxhbrjqjypp5rar3zj4gb-guix-package-cache-check/lib/guix/package.cache"))
scheme@(guile-user)> (length a)
$2 = 13949
scheme@(guile-user)> (length b)
$3 = 13949
scheme@(guile-user)> ,use(srfi srfi-1)
scheme@(guile-user)> (lset= equal? a b)
$4 = #f
scheme@(guile-user)> (car (lset-difference equal? a b))
$5 = #("python2" "2.7.17" (gnu packages python) python-2 ("out" "tk") #t #f "gnu/packages/python.scm" 107 2)
--8<---------------cut here---------------end--------------->8---

So, surprisingly, it’s not just an ordering issue: the caches do contain
different pieces of information.

This patch solves the ordering issue:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1320 bytes --]

diff --git a/gnu/packages.scm b/gnu/packages.scm
index d22c992bb1..5154d73deb 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -407,13 +407,25 @@ reducing the memory footprint."
       (_
        result+seen)))
 
+  (define entry-key
+    (match-lambda
+      (#(name version module symbol outputs supported? deprecated?
+              file line column)
+       (string-append name version (or file "")
+                      (if line (number->string line) "")))))
+
+  (define (entry<? a b)
+    (string<? (entry-key a) (entry-key b)))
+
   (define exp
-    (first
-     (fold-module-public-variables* expand-cache
-                                    (cons '() vlist-null)
-                                    (all-modules (%package-module-path)
-                                                 #:warn
-                                                 warn-about-load-error))))
+    (sort
+     (first
+      (fold-module-public-variables* expand-cache
+                                     (cons '() vlist-null)
+                                     (all-modules (%package-module-path)
+                                                  #:warn
+                                                  warn-about-load-error)))
+     entry<?))
 
   (mkdir-p (dirname cache-file))
   (call-with-output-file cache-file

[-- Attachment #3: Type: text/plain, Size: 191 bytes --]


But it’s not quite right because the order in which variables are
traversed is still non-deterministic, so between two runs of
‘generate-package-cache’, caches differ like this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: Type: text/x-patch, Size: 1796 bytes --]

--- a	2020-06-29 14:27:32.291028456 +0200
+++ b	2020-06-29 14:27:37.930993059 +0200
@@ -8581,7 +8581,7 @@
  #("clang-runtime"
    "9.0.1"
    (gnu packages llvm)
-   clang-runtime
+   clang-runtime-9
    ("out")
    #t
    #f
@@ -26511,7 +26511,7 @@
  #("gcc-objc++"
    "7.5.0"
    (gnu packages gcc)
-   gcc-objc++-7
+   gcc-objc++
    ("out" "lib" "debug")
    #t
    #f
@@ -26641,7 +26641,7 @@
  #("gcc-toolchain"
    "7.5.0"
    (gnu packages commencement)
-   gcc-toolchain
+   gcc-toolchain-7
    ("out" "debug" "static")
    #t
    #f
@@ -33311,7 +33311,7 @@
  #("ghc"
    "8.6.5"
    (gnu packages haskell)
-   ghc-8.6
+   ghc-8
    ("out" "doc")
    #t
    #f
@@ -41876,7 +41876,7 @@
  #("icedtea"
    "3.7.0"
    (gnu packages java)
-   icedtea-8
+   icedtea
    ("out" "jdk" "doc")
    #t
    #f
@@ -54376,7 +54376,7 @@
  #("linux-libre-headers"
    "5.4.20"
    (gnu packages linux)
-   linux-libre-headers-5.4.20
+   linux-libre-headers
    ("out")
    #t
    #f
@@ -54636,7 +54636,7 @@
  #("llvm"
    "9.0.1"
    (gnu packages llvm)
-   llvm-9
+   llvm
    ("out" "opt-viewer")
    #t
    #f
@@ -61826,7 +61826,7 @@
  #("ocaml"
    "4.09.0"
    (gnu packages ocaml)
-   ocaml
+   ocaml-4.09
    ("out")
    #t
    #f
@@ -62256,7 +62256,7 @@
  #("opencl-headers"
    "2.2.0-0.e986688"
    (gnu packages opencl)
-   opencl-headers
+   opencl-headers-2.2
    ("out")
    #t
    #f
@@ -92636,7 +92636,7 @@
  #("python2"
    "2.7.17"
    (gnu packages python)
-   python-2
+   python-2.7
    ("out" "tk")
    #t
    #f
@@ -92646,7 +92646,7 @@
  #("python"
    "3.8.2"
    (gnu packages python)
-   python-3
+   python
    ("out" "tk")
    #t
    #f
@@ -123676,7 +123676,7 @@
  #("rust"
    "1.39.0"
    (gnu packages rust)
-   rust-1.39
+   rust
    ("out" "doc" "cargo")
    #t
    #f

[-- Attachment #5: Type: text/plain, Size: 176 bytes --]


The problem has to do with aliases: we don’t always see the same
variable first.  So we have to sort before calling ‘expand-cache’.

To be continued…

Ludo’.

  reply	other threads:[~2020-06-29 12:36 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-22 17:07 bug#42009: Determinism problem with guix pull Marinus
2020-06-23 22:46 ` bug#42009: package.cache not deterministic zimoun
2020-06-24 18:59   ` Msavoritias
2020-06-25  9:04     ` zimoun
2020-06-25 14:53       ` Msavoritias
2020-06-25 22:33   ` Tobias Geerinckx-Rice via Bug reports for GNU Guix
2020-06-25 23:19     ` zimoun
2020-06-28 20:29       ` Ludovic Courtès
2020-06-29 10:03         ` zimoun
2020-06-29 12:34           ` Ludovic Courtès [this message]
2020-06-29 15:39             ` zimoun
2020-07-30 17:22             ` Ludovic Courtès

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=875zbacaov.fsf@gnu.org \
    --to=ludo@gnu.org \
    --cc=42009@debbugs.gnu.org \
    --cc=zimon.toutoune@gmail.com \
    --subject='Re: bug#42009: package.cache not deterministic' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).