bug#57005: PETSc is not reproducible

bug#57005: PETSc is not reproducible

[Ludovic Courtès]

PETSc 3.16.1 is not reproducible as it captures the kernel version and
number of processors of the machine it’s built on:

--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 223	Aug 01 2022 00:30:24	(current)
  guix ab59155
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: ab59155c5a38dda7efaceb47c7528578fcf0def4
$ guix challenge petsc --diff="$(type -P diff) -ru"
/gnu/store/4dii69ddwxjiyf2pnay82l4w64s03n60-petsc-3.16.1 contents differ:
  no local build for '/gnu/store/4dii69ddwxjiyf2pnay82l4w64s03n60-petsc-3.16.1'
  https://ci.guix.gnu.org/nar/lzip/4dii69ddwxjiyf2pnay82l4w64s03n60-petsc-3.16.1: 1fqg24x0qq0vww8a21a4ifyxlwy0l7bg8zzrzvg024lk76n54kxh
  https://bordeaux.guix.gnu.org/nar/lzip/4dii69ddwxjiyf2pnay82l4w64s03n60-petsc-3.16.1: 162fp2wsxxjjl1pd42522jb3i37hpq8lnfs12qaipzhkzqda5h34
 bordeaux.guix.gnu.org  10.3MiB                                                                                                                      9.7MiB/s 00:01 [##################] 100.0%diff -ru /tmp/guix-directory.4kiUDI/include/petscmachineinfo.h /tmp/guix-directory.0i2wFi/include/petscmachineinfo.h
--- /tmp/guix-directory.4kiUDI/include/petscmachineinfo.h	1970-01-01 01:00:01.000000000 +0100
+++ /tmp/guix-directory.0i2wFi/include/petscmachineinfo.h	1970-01-01 01:00:01.000000000 +0100
@@ -1,6 +1,6 @@
 static const char *petscmachineinfo = "\n"
 "-----------------------------------------\n"
-"Machine characteristics: Linux-5.15.10-gnu-x86_64-with-glibc2.33\n"
+"Machine characteristics: Linux-5.15.11-gnu-x86_64-with-glibc2.33\n"
 "Using PETSc directory: /gnu/store/4dii69ddwxjiyf2pnay82l4w64s03n60-petsc-3.16.1\n"
 "Using PETSc arch: \n"
 "-----------------------------------------\n";
Binary files /tmp/guix-directory.4kiUDI/lib/libpetsc.so and /tmp/guix-directory.0i2wFi/lib/libpetsc.so differ
Binary files /tmp/guix-directory.4kiUDI/lib/libpetsc.so.3.16 and /tmp/guix-directory.0i2wFi/lib/libpetsc.so.3.16 differ
Binary files /tmp/guix-directory.4kiUDI/lib/libpetsc.so.3.16.1 and /tmp/guix-directory.0i2wFi/lib/libpetsc.so.3.16.1 differ
diff -ru /tmp/guix-directory.4kiUDI/lib/petsc/conf/petscvariables /tmp/guix-directory.0i2wFi/lib/petsc/conf/petscvariables
--- /tmp/guix-directory.4kiUDI/lib/petsc/conf/petscvariables	1970-01-01 01:00:01.000000000 +0100
+++ /tmp/guix-directory.0i2wFi/lib/petsc/conf/petscvariables	1970-01-01 01:00:01.000000000 +0100
@@ -107,10 +107,10 @@
 OMAKE_PRINTDIR  = /gnu/store/55cbpsi18mahg131nmiya6km5b4mscfa-make-4.3/bin/make  --print-directory
 OMAKE = /gnu/store/55cbpsi18mahg131nmiya6km5b4mscfa-make-4.3/bin/make  --no-print-directory
 MAKE_PAR_OUT_FLG = --output-sync=recurse
-MAKE_NP = 40
-MAKE_TEST_NP = 40
-MAKE_LOAD = 96.0
-NPMAX = 64
+MAKE_NP = 24
+MAKE_TEST_NP = 24
+MAKE_LOAD = 48.0
+NPMAX = 32
 SONAME_FUNCTION = $(1).$(SL_LINKER_SUFFIX).$(2)
 SL_LINKER_FUNCTION =  -shared  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3 -O0 -Wl,-soname,$(call SONAME_FUNCTION,$(notdir $(1)),$(2))
 BUILDSHAREDLIB = yes

1 store items were analyzed:
  - 0 (0.0%) were identical
  - 1 (100.0%) differed
  - 0 (0.0%) were inconclusive
--8<---------------cut here---------------end--------------->8---

Ludo’.

bug#57005: [PATCH v1] gnu: petsc: scrub build machine specifics

[Lars Bilke]

'Libraries compiled on' does not have to be removed since PETSc 3.9:

https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f

Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68

---
 gnu/packages/maths.scm | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 1f61ee0583..f09cbee48b 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -3455,18 +3455,21 @@ (define-public petsc
                 (substitute* "config/example_template.py"
                   (("#!/usr/bin/env bash")
                    (string-append "#!" (which "bash")))))))
-          (add-after 'configure 'clean-local-references
+          (add-after 'install 'clean-local-references
             (lambda* (#:key outputs #:allow-other-keys)
               (let ((out (assoc-ref outputs "out")))
                 (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
                   ;; Prevent build directory from leaking into compiled code
                   (((getcwd)) out)
-                  ;; Scrub timestamp for reproducibility
-                  ((".*Libraries compiled on.*") ""))
+                  ;; Scrub build machine characteristic
+                  (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
+                    "Machine characteristics: Linux-x.x.x"))
                 (substitute* (find-files "." "petscvariables")
                   ;; Do not expose build machine characteristics, set to defaults.
-                  (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
-                  (("NPMAX = [:digit:]+") "NPMAX = 2")))))
+                  (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
+                  (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
+                  (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
+                  (("NPMAX = [[:digit:]]+") "NPMAX = 2")))))
           (add-after 'install 'clean-install
             ;; Try to keep installed files from leaking build directory names.
             (lambda* (#:key inputs outputs #:allow-other-keys)

base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
-- 
2.43.0

bug#57005: [PATCH v1] gnu: petsc: scrub build machine specifics

[Lars Bilke]

[-- Attachment #1.1: Type: text/plain, Size: 2691 bytes --]

This patch makes these files deterministic:

- petscmachineinfo.h
- petscvariables

Unfortunately libpetsc.so.3.16.1 still differs between builds, see e.g. this diffoscope output:

https://jsfiddle.net/dLx15obp/show (click on run)

Maybe someone has an idea how to also fix the library?

Sincerely,
Lars

On 29 Feb 2024, at 10:46, Lars Bilke wrote:

> 'Libraries compiled on' does not have to be removed since PETSc 3.9:
>
> https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f
>
> Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68
>
> ---
>  gnu/packages/maths.scm | 13 ++++++++-----
>  1 file changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
> index 1f61ee0583..f09cbee48b 100644
> --- a/gnu/packages/maths.scm
> +++ b/gnu/packages/maths.scm
> @@ -3455,18 +3455,21 @@ (define-public petsc
>                  (substitute* "config/example_template.py"
>                    (("#!/usr/bin/env bash")
>                     (string-append "#!" (which "bash")))))))
> -          (add-after 'configure 'clean-local-references
> +          (add-after 'install 'clean-local-references
>              (lambda* (#:key outputs #:allow-other-keys)
>                (let ((out (assoc-ref outputs "out")))
>                  (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
>                    ;; Prevent build directory from leaking into compiled code
>                    (((getcwd)) out)
> -                  ;; Scrub timestamp for reproducibility
> -                  ((".*Libraries compiled on.*") ""))
> +                  ;; Scrub build machine characteristic
> +                  (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
> +                    "Machine characteristics: Linux-x.x.x"))
>                  (substitute* (find-files "." "petscvariables")
>                    ;; Do not expose build machine characteristics, set to defaults.
> -                  (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
> -                  (("NPMAX = [:digit:]+") "NPMAX = 2")))))
> +                  (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
> +                  (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
> +                  (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
> +                  (("NPMAX = [[:digit:]]+") "NPMAX = 2")))))
>            (add-after 'install 'clean-install
>              ;; Try to keep installed files from leaking build directory names.
>              (lambda* (#:key inputs outputs #:allow-other-keys)
>
> base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
> -- 
> 2.43.0

[-- Attachment #1.2: Type: text/html, Size: 3724 bytes --]

[-- Attachment #2: S/MIME digital signature --]
[-- Type: application/pkcs7-signature, Size: 5435 bytes --]

bug#57005: Suggestion

[Andreas Enge]

Hello Lars,

thanks for the patch! Concerning "Machine characteristics", I see it only
in petscmachineinfo.h, so we can probably drop looking for petscconf.h
in find-files.

There is already the cleaning phase 'clean-install after 'install.
So I would either suggest to keep the phase 'clean-local-references
where it is, or to merge its content with 'clean-install (I do not see
why one should be preferred over the other).

Andreas

bug#57005: [PATCH v2] gnu: petsc: scrub build machine specifics

[Lars Bilke]

'Libraries compiled on' does not have to be removed since PETSc 3.9:

https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f

Also merged all add-after 'install phases into `clean-install.

Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68
---
 gnu/packages/maths.scm | 43 ++++++++++++++++++------------------------
 1 file changed, 18 insertions(+), 25 deletions(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 1f61ee0583..4b4d033506 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -3455,30 +3455,34 @@ (define-public petsc
                 (substitute* "config/example_template.py"
                   (("#!/usr/bin/env bash")
                    (string-append "#!" (which "bash")))))))
-          (add-after 'configure 'clean-local-references
-            (lambda* (#:key outputs #:allow-other-keys)
+          (add-after 'install 'clean-install
+            ;; Try to keep installed files from leaking build directory names.
+            (lambda* (#:key inputs outputs #:allow-other-keys)
               (let ((out (assoc-ref outputs "out")))
-                (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
+                (substitute* (find-files "." "petscmachineinfo.h")
                   ;; Prevent build directory from leaking into compiled code
                   (((getcwd)) out)
-                  ;; Scrub timestamp for reproducibility
-                  ((".*Libraries compiled on.*") ""))
+                  ;; Scrub build machine characteristic
+                  (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
+                    "Machine characteristics: Linux-x.x.x"))
                 (substitute* (find-files "." "petscvariables")
                   ;; Do not expose build machine characteristics, set to defaults.
-                  (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
-                  (("NPMAX = [:digit:]+") "NPMAX = 2")))))
-          (add-after 'install 'clean-install
-            ;; Try to keep installed files from leaking build directory names.
-            (lambda* (#:key inputs outputs #:allow-other-keys)
-              (let ((out (assoc-ref outputs "out")))
+                  (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
+                  (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
+                  (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
+                  (("NPMAX = [[:digit:]]+") "NPMAX = 2"))
                 (substitute* (map (lambda (file)
                                     (string-append out "/lib/petsc/conf/" file))
                                   '("petscvariables"))
                   (((getcwd)) out))
-                ;; Make compiler references point to the store
                 (substitute* (string-append out "/lib/petsc/conf/petscvariables")
-                  (("= (gcc|g\\+\\+|gfortran)" _ compiler)
-                   (string-append "= " (which compiler))))
+                  ;; Do not retain a reference to GCC and other build only inputs.
+                  (("([[:graph:]]+)/bin/gcc") "gcc")
+                  (("([[:graph:]]+)/bin/g\\+\\+") "g++")
+                  (("([[:graph:]]+)/bin/make") "make")
+                  (("([[:graph:]]+)/bin/diff") "diff")
+                  (("([[:graph:]]+)/bin/sed") "sed")
+                  (("([[:graph:]]+)/bin/gfortran") "gfortran"))
                 ;; PETSc installs some build logs, which aren't necessary.
                 (for-each (lambda (file)
                             (let ((f (string-append out "/lib/petsc/conf/" file)))
@@ -3490,17 +3494,6 @@ (define-public petsc
                             "configure-hash"
                             ;; Once installed, should uninstall with Guix
                             "uninstall.py")))))
-          (add-after 'clean-install 'clear-reference-to-compiler
-            (lambda* (#:key inputs outputs #:allow-other-keys)
-              ;; Do not retain a reference to GCC and other build only inputs.
-              (let ((out (assoc-ref outputs "out")))
-              (substitute* (string-append out "/lib/petsc/conf/petscvariables")
-                (("([[:graph:]]+)/bin/gcc") "gcc")
-                (("([[:graph:]]+)/bin/g\\+\\+") "g++")
-                (("([[:graph:]]+)/bin/make") "make")
-                (("([[:graph:]]+)/bin/diff") "diff")
-                (("([[:graph:]]+)/bin/sed") "sed")
-                (("([[:graph:]]+)/bin/gfortran") "gfortran")))))
           (add-after 'install 'move-examples
             (lambda* (#:key outputs #:allow-other-keys)
               (let* ((out (assoc-ref outputs "out"))

base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
-- 
2.43.0

bug#57005: [PATCH v3] gnu: petsc: scrub build machine specifics

[Lars Bilke]

'Libraries compiled on' does not have to be removed since PETSc 3.9:

https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f

Also merged all add-after 'install phases into `clean-install.

Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68

sort source files in generated makefiles for reproducibility

Change-Id: I86e5b9cd0d7184019617bd8ae9a23eaf459891d7
---
 gnu/packages/maths.scm | 46 +++++++++++++++++++-----------------------
 1 file changed, 21 insertions(+), 25 deletions(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 1f61ee0583..b828d5d5cb 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -3447,6 +3447,9 @@ (define-public petsc
               (let* ((prefix (assoc-ref outputs "out"))
                      (flags `(,(string-append "--prefix=" prefix)
                               ,@configure-flags)))
+                ;; sort source files in configure (for reproducibility)
+                (substitute* "config/gmakegen.py"
+                   (("join\\(srcs\\[lang\\]\\)") "join(sorted(srcs[lang]))"))
                 (format #t "build directory: ~s~%" (getcwd))
                 (format #t "configure flags: ~s~%" flags)
                 (apply invoke "./configure" flags)
@@ -3455,30 +3458,34 @@ (define-public petsc
                 (substitute* "config/example_template.py"
                   (("#!/usr/bin/env bash")
                    (string-append "#!" (which "bash")))))))
-          (add-after 'configure 'clean-local-references
-            (lambda* (#:key outputs #:allow-other-keys)
+          (add-after 'install 'clean-install
+            ;; Try to keep installed files from leaking build directory names.
+            (lambda* (#:key inputs outputs #:allow-other-keys)
               (let ((out (assoc-ref outputs "out")))
-                (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
+                (substitute* (find-files "." "petscmachineinfo.h")
                   ;; Prevent build directory from leaking into compiled code
                   (((getcwd)) out)
-                  ;; Scrub timestamp for reproducibility
-                  ((".*Libraries compiled on.*") ""))
+                  ;; Scrub build machine characteristic
+                  (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
+                    "Machine characteristics: Linux-x.x.x"))
                 (substitute* (find-files "." "petscvariables")
                   ;; Do not expose build machine characteristics, set to defaults.
-                  (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
-                  (("NPMAX = [:digit:]+") "NPMAX = 2")))))
-          (add-after 'install 'clean-install
-            ;; Try to keep installed files from leaking build directory names.
-            (lambda* (#:key inputs outputs #:allow-other-keys)
-              (let ((out (assoc-ref outputs "out")))
+                  (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
+                  (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
+                  (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
+                  (("NPMAX = [[:digit:]]+") "NPMAX = 2"))
                 (substitute* (map (lambda (file)
                                     (string-append out "/lib/petsc/conf/" file))
                                   '("petscvariables"))
                   (((getcwd)) out))
-                ;; Make compiler references point to the store
                 (substitute* (string-append out "/lib/petsc/conf/petscvariables")
-                  (("= (gcc|g\\+\\+|gfortran)" _ compiler)
-                   (string-append "= " (which compiler))))
+                  ;; Do not retain a reference to GCC and other build only inputs.
+                  (("([[:graph:]]+)/bin/gcc") "gcc")
+                  (("([[:graph:]]+)/bin/g\\+\\+") "g++")
+                  (("([[:graph:]]+)/bin/make") "make")
+                  (("([[:graph:]]+)/bin/diff") "diff")
+                  (("([[:graph:]]+)/bin/sed") "sed")
+                  (("([[:graph:]]+)/bin/gfortran") "gfortran"))
                 ;; PETSc installs some build logs, which aren't necessary.
                 (for-each (lambda (file)
                             (let ((f (string-append out "/lib/petsc/conf/" file)))
@@ -3490,17 +3497,6 @@ (define-public petsc
                             "configure-hash"
                             ;; Once installed, should uninstall with Guix
                             "uninstall.py")))))
-          (add-after 'clean-install 'clear-reference-to-compiler
-            (lambda* (#:key inputs outputs #:allow-other-keys)
-              ;; Do not retain a reference to GCC and other build only inputs.
-              (let ((out (assoc-ref outputs "out")))
-              (substitute* (string-append out "/lib/petsc/conf/petscvariables")
-                (("([[:graph:]]+)/bin/gcc") "gcc")
-                (("([[:graph:]]+)/bin/g\\+\\+") "g++")
-                (("([[:graph:]]+)/bin/make") "make")
-                (("([[:graph:]]+)/bin/diff") "diff")
-                (("([[:graph:]]+)/bin/sed") "sed")
-                (("([[:graph:]]+)/bin/gfortran") "gfortran")))))
           (add-after 'install 'move-examples
             (lambda* (#:key outputs #:allow-other-keys)
               (let* ((out (assoc-ref outputs "out"))

base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
-- 
2.43.0

bug#57005: [PATCH v3] gnu: petsc: scrub build machine specifics

[Lars Bilke]

[-- Attachment #1: Type: text/plain, Size: 5824 bytes --]

v3 of this patch finally makes PETSc fully reproducible!

On 5 Mar 2024, at 15:58, Lars Bilke wrote:

> 'Libraries compiled on' does not have to be removed since PETSc 3.9:
>
> https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f
>
> Also merged all add-after 'install phases into `clean-install.
>
> Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68
>
> sort source files in generated makefiles for reproducibility
>
> Change-Id: I86e5b9cd0d7184019617bd8ae9a23eaf459891d7
> ---
>  gnu/packages/maths.scm | 46 +++++++++++++++++++-----------------------
>  1 file changed, 21 insertions(+), 25 deletions(-)
>
> diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
> index 1f61ee0583..b828d5d5cb 100644
> --- a/gnu/packages/maths.scm
> +++ b/gnu/packages/maths.scm
> @@ -3447,6 +3447,9 @@ (define-public petsc
>                (let* ((prefix (assoc-ref outputs "out"))
>                       (flags `(,(string-append "--prefix=" prefix)
>                                ,@configure-flags)))
> +                ;; sort source files in configure (for reproducibility)
> +                (substitute* "config/gmakegen.py"
> +                   (("join\\(srcs\\[lang\\]\\)") "join(sorted(srcs[lang]))"))
>                  (format #t "build directory: ~s~%" (getcwd))
>                  (format #t "configure flags: ~s~%" flags)
>                  (apply invoke "./configure" flags)
> @@ -3455,30 +3458,34 @@ (define-public petsc
>                  (substitute* "config/example_template.py"
>                    (("#!/usr/bin/env bash")
>                     (string-append "#!" (which "bash")))))))
> -          (add-after 'configure 'clean-local-references
> -            (lambda* (#:key outputs #:allow-other-keys)
> +          (add-after 'install 'clean-install
> +            ;; Try to keep installed files from leaking build directory names.
> +            (lambda* (#:key inputs outputs #:allow-other-keys)
>                (let ((out (assoc-ref outputs "out")))
> -                (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
> +                (substitute* (find-files "." "petscmachineinfo.h")
>                    ;; Prevent build directory from leaking into compiled code
>                    (((getcwd)) out)
> -                  ;; Scrub timestamp for reproducibility
> -                  ((".*Libraries compiled on.*") ""))
> +                  ;; Scrub build machine characteristic
> +                  (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
> +                    "Machine characteristics: Linux-x.x.x"))
>                  (substitute* (find-files "." "petscvariables")
>                    ;; Do not expose build machine characteristics, set to defaults.
> -                  (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
> -                  (("NPMAX = [:digit:]+") "NPMAX = 2")))))
> -          (add-after 'install 'clean-install
> -            ;; Try to keep installed files from leaking build directory names.
> -            (lambda* (#:key inputs outputs #:allow-other-keys)
> -              (let ((out (assoc-ref outputs "out")))
> +                  (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
> +                  (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
> +                  (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
> +                  (("NPMAX = [[:digit:]]+") "NPMAX = 2"))
>                  (substitute* (map (lambda (file)
>                                      (string-append out "/lib/petsc/conf/" file))
>                                    '("petscvariables"))
>                    (((getcwd)) out))
> -                ;; Make compiler references point to the store
>                  (substitute* (string-append out "/lib/petsc/conf/petscvariables")
> -                  (("= (gcc|g\\+\\+|gfortran)" _ compiler)
> -                   (string-append "= " (which compiler))))
> +                  ;; Do not retain a reference to GCC and other build only inputs.
> +                  (("([[:graph:]]+)/bin/gcc") "gcc")
> +                  (("([[:graph:]]+)/bin/g\\+\\+") "g++")
> +                  (("([[:graph:]]+)/bin/make") "make")
> +                  (("([[:graph:]]+)/bin/diff") "diff")
> +                  (("([[:graph:]]+)/bin/sed") "sed")
> +                  (("([[:graph:]]+)/bin/gfortran") "gfortran"))
>                  ;; PETSc installs some build logs, which aren't necessary.
>                  (for-each (lambda (file)
>                              (let ((f (string-append out "/lib/petsc/conf/" file)))
> @@ -3490,17 +3497,6 @@ (define-public petsc
>                              "configure-hash"
>                              ;; Once installed, should uninstall with Guix
>                              "uninstall.py")))))
> -          (add-after 'clean-install 'clear-reference-to-compiler
> -            (lambda* (#:key inputs outputs #:allow-other-keys)
> -              ;; Do not retain a reference to GCC and other build only inputs.
> -              (let ((out (assoc-ref outputs "out")))
> -              (substitute* (string-append out "/lib/petsc/conf/petscvariables")
> -                (("([[:graph:]]+)/bin/gcc") "gcc")
> -                (("([[:graph:]]+)/bin/g\\+\\+") "g++")
> -                (("([[:graph:]]+)/bin/make") "make")
> -                (("([[:graph:]]+)/bin/diff") "diff")
> -                (("([[:graph:]]+)/bin/sed") "sed")
> -                (("([[:graph:]]+)/bin/gfortran") "gfortran")))))
>            (add-after 'install 'move-examples
>              (lambda* (#:key outputs #:allow-other-keys)
>                (let* ((out (assoc-ref outputs "out"))
>
> base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
> -- 
> 2.43.0

[-- Attachment #2: S/MIME digital signature --]
[-- Type: application/pkcs7-signature, Size: 5435 bytes --]

bug#57005: [PATCH v4] gnu: petsc: scrub build machine specifics

[Lars Bilke]

'Libraries compiled on' does not have to be removed since PETSc 3.9:

https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f

Also merged all add-after 'install phases into `clean-install.

Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68

sort source files in generated makefiles for reproducibility

Change-Id: I86e5b9cd0d7184019617bd8ae9a23eaf459891d7
---
 gnu/packages/maths.scm | 56 ++++++++++++++++++------------------------
 1 file changed, 24 insertions(+), 32 deletions(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 1f61ee0583..1ea5ccee74 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -3447,38 +3447,41 @@ (define-public petsc
               (let* ((prefix (assoc-ref outputs "out"))
                      (flags `(,(string-append "--prefix=" prefix)
                               ,@configure-flags)))
+                ;; sort source files in configure (for reproducibility)
+                (substitute* "config/gmakegen.py"
+                   (("join\\(srcs\\[lang\\]\\)") "join(sorted(srcs[lang]))"))
                 (format #t "build directory: ~s~%" (getcwd))
                 (format #t "configure flags: ~s~%" flags)
                 (apply invoke "./configure" flags)
-
+                ;; Prevent build directory Linux version from leaking into
+                ;; compiled code
+                (substitute* (find-files "." "petscmachineinfo.h")
+                  (((getcwd)) out)
+                  ;; Scrub build machine characteristic
+                  (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
+                    "Machine characteristics: Linux-x.x.x"))
                 ;; Generate test scripts with the right shebang.
                 (substitute* "config/example_template.py"
                   (("#!/usr/bin/env bash")
                    (string-append "#!" (which "bash")))))))
-          (add-after 'configure 'clean-local-references
-            (lambda* (#:key outputs #:allow-other-keys)
-              (let ((out (assoc-ref outputs "out")))
-                (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
-                  ;; Prevent build directory from leaking into compiled code
-                  (((getcwd)) out)
-                  ;; Scrub timestamp for reproducibility
-                  ((".*Libraries compiled on.*") ""))
-                (substitute* (find-files "." "petscvariables")
-                  ;; Do not expose build machine characteristics, set to defaults.
-                  (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
-                  (("NPMAX = [:digit:]+") "NPMAX = 2")))))
           (add-after 'install 'clean-install
-            ;; Try to keep installed files from leaking build directory names.
             (lambda* (#:key inputs outputs #:allow-other-keys)
               (let ((out (assoc-ref outputs "out")))
-                (substitute* (map (lambda (file)
-                                    (string-append out "/lib/petsc/conf/" file))
-                                  '("petscvariables"))
-                  (((getcwd)) out))
-                ;; Make compiler references point to the store
                 (substitute* (string-append out "/lib/petsc/conf/petscvariables")
-                  (("= (gcc|g\\+\\+|gfortran)" _ compiler)
-                   (string-append "= " (which compiler))))
+                  ;; Try to keep installed files from leaking build directory names.
+                  (((getcwd)) out)
+                  ;; Do not expose build machine characteristics, set to defaults.
+                  (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
+                  (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
+                  (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
+                  (("NPMAX = [[:digit:]]+") "NPMAX = 2")
+                  ;; Do not retain a reference to GCC and other build only inputs.
+                  (("([[:graph:]]+)/bin/gcc") "gcc")
+                  (("([[:graph:]]+)/bin/g\\+\\+") "g++")
+                  (("([[:graph:]]+)/bin/make") "make")
+                  (("([[:graph:]]+)/bin/diff") "diff")
+                  (("([[:graph:]]+)/bin/sed") "sed")
+                  (("([[:graph:]]+)/bin/gfortran") "gfortran"))
                 ;; PETSc installs some build logs, which aren't necessary.
                 (for-each (lambda (file)
                             (let ((f (string-append out "/lib/petsc/conf/" file)))
@@ -3490,17 +3493,6 @@ (define-public petsc
                             "configure-hash"
                             ;; Once installed, should uninstall with Guix
                             "uninstall.py")))))
-          (add-after 'clean-install 'clear-reference-to-compiler
-            (lambda* (#:key inputs outputs #:allow-other-keys)
-              ;; Do not retain a reference to GCC and other build only inputs.
-              (let ((out (assoc-ref outputs "out")))
-              (substitute* (string-append out "/lib/petsc/conf/petscvariables")
-                (("([[:graph:]]+)/bin/gcc") "gcc")
-                (("([[:graph:]]+)/bin/g\\+\\+") "g++")
-                (("([[:graph:]]+)/bin/make") "make")
-                (("([[:graph:]]+)/bin/diff") "diff")
-                (("([[:graph:]]+)/bin/sed") "sed")
-                (("([[:graph:]]+)/bin/gfortran") "gfortran")))))
           (add-after 'install 'move-examples
             (lambda* (#:key outputs #:allow-other-keys)
               (let* ((out (assoc-ref outputs "out"))

base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
-- 
2.43.0

bug#57005: [PATCH v4] gnu: petsc: scrub build machine specifics

[Lars Bilke]

[-- Attachment #1.1: Type: text/plain, Size: 6208 bytes --]

v4 should (hopefully) fix all issues regarding reprocuibility of petsc:

I have tested v4 on multiple machines with exact identical output (ran `guix hash -S nar ...` on the packages) and also tested on one machine with `guix build --check`

On 6 Mar 2024, at 12:51, Lars Bilke wrote:

> 'Libraries compiled on' does not have to be removed since PETSc 3.9:
>
> https: //gitlab.com/petsc/petsc/-/commit/a970bd748978a4e12790be7c36b10e832ec0149f
>
> Also merged all add-after 'install phases into `clean-install.
>
> Change-Id: I41d82339e43187d7dc7e3fc189a338784c363c68
>
> sort source files in generated makefiles for reproducibility
>
> Change-Id: I86e5b9cd0d7184019617bd8ae9a23eaf459891d7
> ---
>  gnu/packages/maths.scm | 56 ++++++++++++++++++------------------------
>  1 file changed, 24 insertions(+), 32 deletions(-)
>
> diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
> index 1f61ee0583..1ea5ccee74 100644
> --- a/gnu/packages/maths.scm
> +++ b/gnu/packages/maths.scm
> @@ -3447,38 +3447,41 @@ (define-public petsc
>                (let* ((prefix (assoc-ref outputs "out"))
>                       (flags `(,(string-append "--prefix=" prefix)
>                                ,@configure-flags)))
> +                ;; sort source files in configure (for reproducibility)
> +                (substitute* "config/gmakegen.py"
> +                   (("join\\(srcs\\[lang\\]\\)") "join(sorted(srcs[lang]))"))
>                  (format #t "build directory: ~s~%" (getcwd))
>                  (format #t "configure flags: ~s~%" flags)
>                  (apply invoke "./configure" flags)
> -
> +                ;; Prevent build directory Linux version from leaking into
> +                ;; compiled code
> +                (substitute* (find-files "." "petscmachineinfo.h")
> +                  (((getcwd)) out)
> +                  ;; Scrub build machine characteristic
> +                  (("Machine characteristics: Linux-[0-9]+\\.[0-9]+\\.[0-9]+")
> +                    "Machine characteristics: Linux-x.x.x"))
>                  ;; Generate test scripts with the right shebang.
>                  (substitute* "config/example_template.py"
>                    (("#!/usr/bin/env bash")
>                     (string-append "#!" (which "bash")))))))
> -          (add-after 'configure 'clean-local-references
> -            (lambda* (#:key outputs #:allow-other-keys)
> -              (let ((out (assoc-ref outputs "out")))
> -                (substitute* (find-files "." "^petsc(conf|machineinfo).h$")
> -                  ;; Prevent build directory from leaking into compiled code
> -                  (((getcwd)) out)
> -                  ;; Scrub timestamp for reproducibility
> -                  ((".*Libraries compiled on.*") ""))
> -                (substitute* (find-files "." "petscvariables")
> -                  ;; Do not expose build machine characteristics, set to defaults.
> -                  (("MAKE_NP = [:digit:]+") "MAKE_NP = 2")
> -                  (("NPMAX = [:digit:]+") "NPMAX = 2")))))
>            (add-after 'install 'clean-install
> -            ;; Try to keep installed files from leaking build directory names.
>              (lambda* (#:key inputs outputs #:allow-other-keys)
>                (let ((out (assoc-ref outputs "out")))
> -                (substitute* (map (lambda (file)
> -                                    (string-append out "/lib/petsc/conf/" file))
> -                                  '("petscvariables"))
> -                  (((getcwd)) out))
> -                ;; Make compiler references point to the store
>                  (substitute* (string-append out "/lib/petsc/conf/petscvariables")
> -                  (("= (gcc|g\\+\\+|gfortran)" _ compiler)
> -                   (string-append "= " (which compiler))))
> +                  ;; Try to keep installed files from leaking build directory names.
> +                  (((getcwd)) out)
> +                  ;; Do not expose build machine characteristics, set to defaults.
> +                  (("MAKE_NP = [[:digit:]]+") "MAKE_NP = 2")
> +                  (("MAKE_TEST_NP = [[:digit:]]+") "MAKE_TEST_NP = 2")
> +                  (("MAKE_LOAD = [[:digit:]]+\\.[[:digit:]]+") "MAKE_LOAD = 2.0")
> +                  (("NPMAX = [[:digit:]]+") "NPMAX = 2")
> +                  ;; Do not retain a reference to GCC and other build only inputs.
> +                  (("([[:graph:]]+)/bin/gcc") "gcc")
> +                  (("([[:graph:]]+)/bin/g\\+\\+") "g++")
> +                  (("([[:graph:]]+)/bin/make") "make")
> +                  (("([[:graph:]]+)/bin/diff") "diff")
> +                  (("([[:graph:]]+)/bin/sed") "sed")
> +                  (("([[:graph:]]+)/bin/gfortran") "gfortran"))
>                  ;; PETSc installs some build logs, which aren't necessary.
>                  (for-each (lambda (file)
>                              (let ((f (string-append out "/lib/petsc/conf/" file)))
> @@ -3490,17 +3493,6 @@ (define-public petsc
>                              "configure-hash"
>                              ;; Once installed, should uninstall with Guix
>                              "uninstall.py")))))
> -          (add-after 'clean-install 'clear-reference-to-compiler
> -            (lambda* (#:key inputs outputs #:allow-other-keys)
> -              ;; Do not retain a reference to GCC and other build only inputs.
> -              (let ((out (assoc-ref outputs "out")))
> -              (substitute* (string-append out "/lib/petsc/conf/petscvariables")
> -                (("([[:graph:]]+)/bin/gcc") "gcc")
> -                (("([[:graph:]]+)/bin/g\\+\\+") "g++")
> -                (("([[:graph:]]+)/bin/make") "make")
> -                (("([[:graph:]]+)/bin/diff") "diff")
> -                (("([[:graph:]]+)/bin/sed") "sed")
> -                (("([[:graph:]]+)/bin/gfortran") "gfortran")))))
>            (add-after 'install 'move-examples
>              (lambda* (#:key outputs #:allow-other-keys)
>                (let* ((out (assoc-ref outputs "out"))
>
> base-commit: 75bad75367fcf2c289fae3b40dbcc850f92177be
> -- 
> 2.43.0

[-- Attachment #1.2: Type: text/html, Size: 7533 bytes --]

[-- Attachment #2: S/MIME digital signature --]
[-- Type: application/pkcs7-signature, Size: 5435 bytes --]

bug#57005: Close

[Andreas Enge]

Hello Lars,

apologies for my lack of engagement.

If I understood things correctly, Ludovic's recent patch from
https://issues.guix.gnu.org/72877 solves the reproducibility issue.
So I am closing this bug. If I am mistaken, please feel free to reopen it
with an updated patch.

Thanks,

Andreas