bug#22304: [Nicolò Balzarotti] Re: Julia v1.4.1 still not reproducible

[anothersms@gmail.com (Nicolò Balzarotti)]

[-- Attachment #1: Type: text/plain, Size: 75 bytes --]


Ops, I sent it to 22304@debuugs.gnu.org instead of 22304@debbugs.gnu.org


[-- Attachment #2: Type: message/rfc822, Size: 8372 bytes --]

[-- Attachment #2.1.1: Type: text/plain, Size: 826 bytes --]

Hi Efraim, Simon,

The attached patches provide Simon's suggested julia-deterministic
build.

With this, I get deterministic julia build, *but* only on a single
machine.  Building on a different machine I get a different output.  I
still need to investigate why.  Also, some of the patches are probably
not needed at all.  In the near future, as discussed previously, I aim
at submitting required patches upstream.

I disabled tests in julia-deterministic, so it get built quite fast
(~10minutes).  If you want to try to see if this is reproducible//know
how to figure out what might be happening on different machines,
feedback is wellcome!

My ouptut is (patches applied on 75741af9b2):
guix hash -r /gnu/store/mp8hl7j09nfnsxbb3a6ka23ww44gq0z2-julia-deterministic-1.4.1
1k6mak38nkiaj0b9l6rcmxfcjwvbijmczxs703hgawwi6z0jikxl


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2.1.2: 0001-gnu-julia-Specify-correct-llvm-version.patch --]
[-- Type: text/x-patch, Size: 789 bytes --]

From baf3f3853079ae375342f626516945998c15f1ca Mon Sep 17 00:00:00 2001
From: nixo <nicolo@nixo.xyz>
Date: Thu, 14 May 2020 17:17:08 +0200
Subject: [PATCH 1/2] gnu: julia: Specify correct llvm version.

---
 gnu/packages/julia.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index c636fa756b..8db5e66131 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -416,7 +416,7 @@ libraries.  It is also a bit like @code{ldd} and @code{otool -L}.")
                         (assoc-ref %build-inputs "utf8proc")
                         "/include")
          "USE_SYSTEM_LLVM=1"
-         "LLVM_VER=6.0.1"
+         "LLVM_VER=8.0.0"
 
          "USE_LLVM_SHLIB=1"
          "USE_SYSTEM_LIBUNWIND=1"
-- 
2.26.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2.1.3: 0002-gnu-Add-julia-deterministic.patch --]
[-- Type: text/x-patch, Size: 4580 bytes --]

From fc0bd3f4e8d83898e64fe8cae8c4c61defa7ba4d Mon Sep 17 00:00:00 2001
From: nixo <nicolo@nixo.xyz>
Date: Sat, 16 May 2020 11:56:26 +0200
Subject: [PATCH 2/2] gnu: Add julia-deterministic.

* gnu/packages/julia.scm (julia-deterministic): New variable.
---
 gnu/packages/julia.scm | 64 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)

diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index 8db5e66131..0b4cb82966 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -493,3 +493,67 @@ computing environments.  It provides a sophisticated compiler, distributed
 parallel execution, numerical accuracy, and an extensive mathematical function
 library.")
     (license license:expat)))
+
+;; Deterministic build of the julia executable
+(define-public julia-deterministic
+  (package
+    (inherit julia)
+    (name "julia-deterministic")
+    (source
+     (origin
+       (inherit (package-source julia))
+       ;; I wrote this patches to get a deterministic build.
+       ;; We need to discuss the best way to add them upstream.
+       (patches
+        (list
+         (origin ; timefuncs: support SOURCE_DATE_EPOCH for reproducible builds
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/1c56fb0975c7701596b5bdc9ad4e716a3e2fcfc9.patch")
+           (sha256 (base32 "0zw5554s0k38kgq8bgyxcjflrbjk6xqfrh8kidwsh6zvmpkmgnf5")))
+         (origin                ; don't use jl_hrtime to store module build_id
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/c13691df6747849dec8f654180dabc55e8044a64.patch")
+           (sha256 (base32 "0n7mnbwzr411j9z557jrcib5vr8v8mgqbj5xdrxv7b2n0mrfcp5d")))
+         (origin ; contrib: generate_precompile: do not use random temporary files
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/ae2929a5d3d1f724b421a7f04b6c7b84c9987583.patch")
+           (sha256 (base32 "00vpj4m9f4wzfkvd4flarki58jbfhi0zngw35zrhaph53mfi6y27")))
+         ;; (origin                        ; base: Base: deterministic random
+         ;;   (method url-fetch)
+         ;;   (uri "https://github.com/nico202/julia/commit/49a4edc26e9db110586c590c17df276301c344a7.patch")
+         ;;   (sha256 (base32 "16p4axwr3shf2jp6hq8dk19c9s1l667ki5a8bv3dbv5b0ji9w7ra")))
+         (origin                        ; remove some use of time()
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/5561587256b29761fe3a7d8fac169b4305afa240.patch")
+           (sha256 (base32 "0s9b90x61jjrc057myqdck52yj4lb4xlm7ypjn5qvpypnxrbkz6b")))
+         (origin                  ; contrib: generate_precompile: bare minimum
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/5cd454695e7f6a7e448c19d8f9b2d41fcdeb812e.patch")
+           (sha256 (base32 "0c9pz9ligd6wfhvra5wn7k2k2qr1zw3x67b2y9dhf108xf4z79sm")))
+         (origin          ; src: codegen: don't dump time to precompile stream
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/5445b5dc2ea30f2f39acb0fb45ba552122a7e6f5.patch")
+           (sha256 (base32 "0sa1a73ajk8cm3a5l25hyc2bg8kq6l16r2vvdn2qbcnpnjff9skx")))
+         (origin                        ; do not import repl
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/9a45d44ba387d7386b14abfe76e18f1f000eecc2.patch")
+           (sha256 (base32 "0k6yyxr8rd3dcj8rh32pjzlyc0izvi34xglmxsdhqq6q3j0rf0wb")))
+         (origin                        ; no more interactive
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/e78f791e048ed5192d6cb3166fe089207e2e994d.patch")
+           (sha256 (base32 "1bvgiishhc0csypymknff46cik5dnanyx4q5kgipjk0a47wi80x2")))
+         (origin                        ; generate precompile: sort modules
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/e5359e1ce383d549db87659ecd72dcb615eb649d.patch")
+           (sha256 (base32 "0slw4jczba88hpqyd0gq9x7jnw75yvq8wmbrl7dd8f83jmci8lzz")))))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments julia)
+       ((#:phases phases '%standard-phases)
+        `(modify-phases ,phases
+           (delete 'check)
+           (add-after 'unpack 'generic-target
+             (lambda _
+               ;; Not sure is needed/is the best target
+               (substitute* "contrib/generate_precompile.jl"
+                 (("native") "generic"))
+               #t))))))))
-- 
2.26.2


[-- Attachment #2.1.4: Type: text/plain, Size: 942 bytes --]


Thanks, Nicolò

Efraim Flashner <efraim@flashner.co.il> writes:

> On Wed, May 13, 2020 at 08:43:08PM +0200, zimoun wrote:
>> Hi Efraim,
>> 
>> On Wed, 13 May 2020 at 13:04, Efraim Flashner <efraim@flashner.co.il> wrote:
>> 
>> > guix build: error: derivation `/gnu/store/z21c12ijspavaclcins5pja8z1s8wbm0-julia-1.4.1.drv' may not be deterministic: output `/gnu/store/ri931fk8hf5wvagk76zhmwbphq2pwgnp-julia-1.4.1' differs from ?/gnu/store/ri931fk8hf5wvagk76zhmwbphq2pwgnp-julia-1.4.1-check?
>> >
>> > I built it on bayfront twice. I'll get the diffoscope output once it
>> > finally builds.
>
> I wasn't able to provide the output of diffoscope, it turns out that the
> output was too large to attach.
>
> -- 
> Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted