bug#22304: Build for Julia is not reproducible

bug#22304: Build for Julia is not reproducible

[Ricardo Wurmus]

When building “julia” twice we get different binaries for

  $out/lib/julia/sys.so

and

  $out/lib/julia/sys-debug.so

I could not determine the exact differences, because instead of a diff I
just see something like this:

--- /gnu/store/6njfyp46b22b0wb0r0ccg2wdzjx7066g-julia-0.4.2/lib/julia/sys.so
+++ 6njfyp46b22b0wb0r0ccg2wdzjx7066g-julia-0.4.2/lib/julia/sys.so
├── objdump --disassemble --full-contents {}
│ @@ -99994,8 +99994,8 @@
│   186a00 05a3d38a 01488985 38ffffff 488b0515  .....H..8...H...
│   186a10 5b8b0148 898540ff ffff4c89 ad48ffff  [..H..@...L..H..
│   186a20 ff31ffba 03000000 4c8db538 ffffff4c  .1......L..8...L
│   186a30 89f64989 f4e8b653 f0ff4889 8538ffff  ..I....S..H..8..
│   186a40 ff31ffba 03000000 488d9d28 ffffff49  .1......H..(...I
│   186a50 89de4c89 f6e89653 f0ff4889 8528ffff  ..L....S..H..(..
│   186a60 ff488b05 38d08a01 48898530 ffffff48  .H..8...H..0...H
│ -[ Too much input for diff ]                     
│ +[ Too much input for diff ]                         

...

--- /gnu/store/6njfyp46b22b0wb0r0ccg2wdzjx7066g-julia-0.4.2/lib/julia/sys-debug.so
+++ 6njfyp46b22b0wb0r0ccg2wdzjx7066g-julia-0.4.2/lib/julia/sys-debug.so
├── objdump --disassemble --full-contents {}
│ @@ -99994,8 +99994,8 @@
│   1869f0 01488b38 be370200 00e84257 f0ffe89d  .H.8.7....BW....
│   186a00 56f0ff66 6666662e 0f1f8400 00000000  V..ffff.........
│   186a10 554889e5 488b3e48 8b7608e8 e0fdffff  UH..H.>H.v......
│   186a20 4889c7e8 2856f0ff 5dc3660f 1f440000  H...(V..].f..D..
│   186a30 554889e5 4883ec10 64488b04 25280000  UH..H...dH..%(..
│   186a40 00488945 f848833d fb4d8d01 00740948  .H.E.H.=.M...t.H
│   186a50 8b0df24d 8d01eb1f 488d3591 ac190048  ...M....H.5....H
│ -[ Too much input for diff ]                     
│ +[ Too much input for diff ]                         
╵

~~ Ricardo

bug#22304: Build for Julia is not reproducible

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> When building “julia” twice we get different binaries for
>
>   $out/lib/julia/sys.so
>
> and
>
>   $out/lib/julia/sys-debug.so

Apparently there’s no __DATE__ and __TIME__ in the C code.

I would check if the order of object files on the command line that
links those shared objects is always the same.

Ludo’.

bug#22304: Build for Julia is not reproducible

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
>
>> When building “julia” twice we get different binaries for
>>
>>   $out/lib/julia/sys.so
>>
>> and
>>
>>   $out/lib/julia/sys-debug.so
>
> Apparently there’s no __DATE__ and __TIME__ in the C code.
>
> I would check if the order of object files on the command line that
> links those shared objects is always the same.

As far as I could tell the order of shared objects in two consecutive
runs is the same.

~~ Ricardo

bug#22304: Julia not reproducible

[zimoun]

Hi,

This bug [1] is still present even with the version 1.1.1 of Julia.

[1] http://issues.guix.gnu.org/issue/22304


--8<---------------cut here---------------start------------->8---
guix describe
Generation 57   Nov 25 2019 14:26:15    (current)
  guix b5d4d5b
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: b5d4d5b9bcf267fddd02fcc14b88eac0bebf979f
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
guix build julia # populate the store with dependencies
guix build julia --check -K --no-grafts
[...]
guix build: error: derivation
`/gnu/store/xdy3jjz7dzg1lr231b8zbss8xn44ldjj-julia-1.1.1.drv' may not
be deterministic: output
`/gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1' differs from
‘/gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check’
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
diff -r /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1
/gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check
Binary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/lib/julia/sys.so
and /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/lib/julia/sys.so
differ
Binary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Bar/HXSAn.ji
and /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Bar/HXSAn.ji
differ
Binary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Baz/rONVA.ji
and /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Baz/rONVA.ji
differ
Binary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Foo/MYb1d.ji
and /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Foo/MYb1d.ji
differ
Binary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Foo/TeeT6.ji
and /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Foo/TeeT6.ji
differ
Binary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Qux/YFfiR.ji
and /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Qux/YFfiR.ji
differ
--8<---------------cut here---------------end--------------->8---


Well, then I try to explore with diffoscope but hum? I do not do
correctly because the resulting diff is 845MB ouch!


All the best,
simon

bug#22304: Julia not reproducible

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Binary files /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1/share/julia/test/depot/compiled/v1.1/Qux/YFfiR.ji
> and /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1-check/share/julia/test/depot/compiled/v1.1/Qux/YFfiR.ji
> differ
>
>
> Well, then I try to explore with diffoscope but hum? I do not do
> correctly because the resulting diff is 845MB ouch!

Pro tip: just run

  diffoscope \
  /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1{,-check}/share/julia/test/depot/compiled/v1.1/Qux/YFfiR.ji

That’ll already be enough info for a start!

Ludo’.

bug#22304: Julia not reproducible

[zimoun]

Hi Ludo,

On Fri, 29 Nov 2019 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote:

> Pro tip: just run

I think that I have already did the pro tip. :-)

--8<---------------cut here---------------end--------------->8---
diffoscope /gnu/store/s2vj70fgv4v4wq66dbi797ss15f8xd3b-julia-1.1.1{,-check}/lib/julia/sys.so
\
   > /tmp/sys.diffoscope
du -sh /tmp/sys.diffoscope
845M    /tmp/sys.diffoscope
--8<---------------cut here---------------end--------------->8---

Maybe I am doing incorrectly...

If I understand well [1], the file .ji are "precompiled" (analogy: .pyc). Hum?!

[1] https://docs.julialang.org/en/v1/devdocs/sysimg/


Cheers,
simon

bug#22304: Julia not reproducible

[zimoun]

Hi,

On Mon, 2 Dec 2019 at 18:28, zimoun <zimon.toutoune@gmail.com> wrote:

> If I understand well [1], the file .ji are "precompiled" (analogy: .pyc). Hum?!

The non-determinism seems coming from these "precompiled" .ji files
and 2 bugs are opened upstream.

https://github.com/JuliaLang/julia/issues/25900
https://github.com/JuliaLang/julia/issues/34115

Thanks to Nicolò to pointing out these [1] when working on the version update.

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38546#80

With lengthy and clear details. :-)


Let start the container with everything useful.

--8<---------------cut here---------------start------------->8---
mkdir -p /tmp/test-julia && cd /tmp/test-julia

guix time-machine --commit=b5d4d5b \
   environment -u nixbld --no-cwd --container --network \
   --expose=$(realpath determinism) --share=/tmp/test-julia \
   --ad-hoc julia coreutils nss-certs curl tar gzip
--8<---------------cut here---------------end--------------->8---

Then, the path to reproduce is odd because there is no Julia packages
in Guix. So we need to use 'Pkg' from Julia which downloads from
elsewhere and the reproducibility of such is not well controlled but
it gives an idea. :-)

--8<---------------cut here---------------start------------->8---
export SSL_CERT_DIR="$GUIX_ENVIRONMENT/etc/ssl/certs"
export SSL_CERT_FILE="$SSL_CERT_DIR/ca-certificates.crt"
export GIT_SSL_CAINFO="$SSL_CERT_FILE"

# populate the Julia package manager
julia --startup-file=no -q -e 'using Pkg; Pkg.instantiate()'

# install something
julia --startup-file=no -q -e 'using Pkg; Pkg.add("Compat")'

# use something / precompiled it
julia --startup-file=no -q -e 'using Compat'

# check the .ji file
md5sum ~/.julia/compiled/v1.1/Compat/*
--8<---------------cut here---------------end--------------->8---


Then each time one runs this block, one gets different hash.


All the best,
simon

bug#22304: Build for Julia is not reproducible

[Ludovic Courtès]

Hi!

FWIW Julia 1.5.3 is also not reproducible:

--8<---------------cut here---------------start------------->8---
$ guix challenge julia
/gnu/store/d077a9ggqlcyhnhcg4ng2ibnlarxq8zf-julia-1.5.3 contents differ:
  no local build for '/gnu/store/d077a9ggqlcyhnhcg4ng2ibnlarxq8zf-julia-1.5.3'
  https://ci.guix.gnu.org/nar/lzip/d077a9ggqlcyhnhcg4ng2ibnlarxq8zf-julia-1.5.3: 097czl5kmmdbgljqjrmamkpzsfmkkpy578i1niwm2fizxslyxws6
  https://bordeaux.guix.gnu.org/nar/lzip/d077a9ggqlcyhnhcg4ng2ibnlarxq8zf-julia-1.5.3: 047bd3qgar4c17i9xp5g564xac4vsqy5h7hzd0ss824f367idh9c
  differing files:
    /share/julia/test/depot/compiled/v1.5/Qux/YFfiR_w3IH9.ji
    /share/julia/test/depot/compiled/v1.5/Foo/TeeT6_w3IH9.ji
    /share/julia/test/depot/compiled/v1.5/Foo/MYb1d_w3IH9.ji
    /share/julia/test/depot/compiled/v1.5/Baz/rONVA_w3IH9.ji
    /share/julia/test/depot/compiled/v1.5/Bar/HXSAn_w3IH9.ji
    /share/julia/stdlib/v1.5/REPL/src/Terminals.jl.6161.mem
    /share/julia/stdlib/v1.5/REPL/src/Terminals.jl.6152.mem
    /share/julia/stdlib/v1.5/REPL/src/Terminals.jl.6143.mem
    /share/julia/stdlib/v1.5/Logging/src/ConsoleLogger.jl.6161.mem
    /share/julia/stdlib/v1.5/Logging/src/ConsoleLogger.jl.6152.mem
    /share/julia/stdlib/v1.5/Logging/src/ConsoleLogger.jl.6143.mem
    /lib/julia/sys.so

1 store items were analyzed:
  - 0 (0.0%) were identical
  - 1 (100.0%) differed
  - 0 (0.0%) were inconclusive
$ guix describe
Generacio 186	Jun 25 2021 11:36:00	(nuna)
  guix e4305e4
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: e4305e4562f1bcedf888d8a5473aad256df38e81
--8<---------------cut here---------------end--------------->8---

Ludo’.

bug#22304: [Nicolò Balzarotti] Re: Julia v1.4.1 still not reproducible

[Nicolò Balzarotti]

[-- Attachment #1: Type: text/plain, Size: 75 bytes --]


Ops, I sent it to 22304@debuugs.gnu.org instead of 22304@debbugs.gnu.org


[-- Attachment #2: Type: message/rfc822, Size: 8372 bytes --]

[-- Attachment #2.1.1: Type: text/plain, Size: 826 bytes --]

Hi Efraim, Simon,

The attached patches provide Simon's suggested julia-deterministic
build.

With this, I get deterministic julia build, *but* only on a single
machine.  Building on a different machine I get a different output.  I
still need to investigate why.  Also, some of the patches are probably
not needed at all.  In the near future, as discussed previously, I aim
at submitting required patches upstream.

I disabled tests in julia-deterministic, so it get built quite fast
(~10minutes).  If you want to try to see if this is reproducible//know
how to figure out what might be happening on different machines,
feedback is wellcome!

My ouptut is (patches applied on 75741af9b2):
guix hash -r /gnu/store/mp8hl7j09nfnsxbb3a6ka23ww44gq0z2-julia-deterministic-1.4.1
1k6mak38nkiaj0b9l6rcmxfcjwvbijmczxs703hgawwi6z0jikxl


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2.1.2: 0001-gnu-julia-Specify-correct-llvm-version.patch --]
[-- Type: text/x-patch, Size: 789 bytes --]

From baf3f3853079ae375342f626516945998c15f1ca Mon Sep 17 00:00:00 2001
From: nixo <nicolo@nixo.xyz>
Date: Thu, 14 May 2020 17:17:08 +0200
Subject: [PATCH 1/2] gnu: julia: Specify correct llvm version.

---
 gnu/packages/julia.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index c636fa756b..8db5e66131 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -416,7 +416,7 @@ libraries.  It is also a bit like @code{ldd} and @code{otool -L}.")
                         (assoc-ref %build-inputs "utf8proc")
                         "/include")
          "USE_SYSTEM_LLVM=1"
-         "LLVM_VER=6.0.1"
+         "LLVM_VER=8.0.0"
 
          "USE_LLVM_SHLIB=1"
          "USE_SYSTEM_LIBUNWIND=1"
-- 
2.26.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2.1.3: 0002-gnu-Add-julia-deterministic.patch --]
[-- Type: text/x-patch, Size: 4580 bytes --]

From fc0bd3f4e8d83898e64fe8cae8c4c61defa7ba4d Mon Sep 17 00:00:00 2001
From: nixo <nicolo@nixo.xyz>
Date: Sat, 16 May 2020 11:56:26 +0200
Subject: [PATCH 2/2] gnu: Add julia-deterministic.

* gnu/packages/julia.scm (julia-deterministic): New variable.
---
 gnu/packages/julia.scm | 64 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)

diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index 8db5e66131..0b4cb82966 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -493,3 +493,67 @@ computing environments.  It provides a sophisticated compiler, distributed
 parallel execution, numerical accuracy, and an extensive mathematical function
 library.")
     (license license:expat)))
+
+;; Deterministic build of the julia executable
+(define-public julia-deterministic
+  (package
+    (inherit julia)
+    (name "julia-deterministic")
+    (source
+     (origin
+       (inherit (package-source julia))
+       ;; I wrote this patches to get a deterministic build.
+       ;; We need to discuss the best way to add them upstream.
+       (patches
+        (list
+         (origin ; timefuncs: support SOURCE_DATE_EPOCH for reproducible builds
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/1c56fb0975c7701596b5bdc9ad4e716a3e2fcfc9.patch")
+           (sha256 (base32 "0zw5554s0k38kgq8bgyxcjflrbjk6xqfrh8kidwsh6zvmpkmgnf5")))
+         (origin                ; don't use jl_hrtime to store module build_id
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/c13691df6747849dec8f654180dabc55e8044a64.patch")
+           (sha256 (base32 "0n7mnbwzr411j9z557jrcib5vr8v8mgqbj5xdrxv7b2n0mrfcp5d")))
+         (origin ; contrib: generate_precompile: do not use random temporary files
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/ae2929a5d3d1f724b421a7f04b6c7b84c9987583.patch")
+           (sha256 (base32 "00vpj4m9f4wzfkvd4flarki58jbfhi0zngw35zrhaph53mfi6y27")))
+         ;; (origin                        ; base: Base: deterministic random
+         ;;   (method url-fetch)
+         ;;   (uri "https://github.com/nico202/julia/commit/49a4edc26e9db110586c590c17df276301c344a7.patch")
+         ;;   (sha256 (base32 "16p4axwr3shf2jp6hq8dk19c9s1l667ki5a8bv3dbv5b0ji9w7ra")))
+         (origin                        ; remove some use of time()
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/5561587256b29761fe3a7d8fac169b4305afa240.patch")
+           (sha256 (base32 "0s9b90x61jjrc057myqdck52yj4lb4xlm7ypjn5qvpypnxrbkz6b")))
+         (origin                  ; contrib: generate_precompile: bare minimum
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/5cd454695e7f6a7e448c19d8f9b2d41fcdeb812e.patch")
+           (sha256 (base32 "0c9pz9ligd6wfhvra5wn7k2k2qr1zw3x67b2y9dhf108xf4z79sm")))
+         (origin          ; src: codegen: don't dump time to precompile stream
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/5445b5dc2ea30f2f39acb0fb45ba552122a7e6f5.patch")
+           (sha256 (base32 "0sa1a73ajk8cm3a5l25hyc2bg8kq6l16r2vvdn2qbcnpnjff9skx")))
+         (origin                        ; do not import repl
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/9a45d44ba387d7386b14abfe76e18f1f000eecc2.patch")
+           (sha256 (base32 "0k6yyxr8rd3dcj8rh32pjzlyc0izvi34xglmxsdhqq6q3j0rf0wb")))
+         (origin                        ; no more interactive
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/e78f791e048ed5192d6cb3166fe089207e2e994d.patch")
+           (sha256 (base32 "1bvgiishhc0csypymknff46cik5dnanyx4q5kgipjk0a47wi80x2")))
+         (origin                        ; generate precompile: sort modules
+           (method url-fetch)
+           (uri "https://github.com/nico202/julia/commit/e5359e1ce383d549db87659ecd72dcb615eb649d.patch")
+           (sha256 (base32 "0slw4jczba88hpqyd0gq9x7jnw75yvq8wmbrl7dd8f83jmci8lzz")))))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments julia)
+       ((#:phases phases '%standard-phases)
+        `(modify-phases ,phases
+           (delete 'check)
+           (add-after 'unpack 'generic-target
+             (lambda _
+               ;; Not sure is needed/is the best target
+               (substitute* "contrib/generate_precompile.jl"
+                 (("native") "generic"))
+               #t))))))))
-- 
2.26.2


[-- Attachment #2.1.4: Type: text/plain, Size: 942 bytes --]


Thanks, Nicolò

Efraim Flashner <efraim@flashner.co.il> writes:

> On Wed, May 13, 2020 at 08:43:08PM +0200, zimoun wrote:
>> Hi Efraim,
>> 
>> On Wed, 13 May 2020 at 13:04, Efraim Flashner <efraim@flashner.co.il> wrote:
>> 
>> > guix build: error: derivation `/gnu/store/z21c12ijspavaclcins5pja8z1s8wbm0-julia-1.4.1.drv' may not be deterministic: output `/gnu/store/ri931fk8hf5wvagk76zhmwbphq2pwgnp-julia-1.4.1' differs from ?/gnu/store/ri931fk8hf5wvagk76zhmwbphq2pwgnp-julia-1.4.1-check?
>> >
>> > I built it on bayfront twice. I'll get the diffoscope output once it
>> > finally builds.
>
> I wasn't able to provide the output of diffoscope, it turns out that the
> output was too large to attach.
>
> -- 
> Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted

bug#22304: Julia

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

Just checking this, looks like 1.8.3 is still not reproducible, just one
file differs though:

→ guix challenge --diff=simple julia
/gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3 contents differ:
  no local build for '/gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3'
  https://ci.guix.gnu.org/nar/lzip/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3: 0jzli7ym7ihj7zhqwqp8lgwn6c38n44ld1rdn39pa7n8qp1q154r
  https://bordeaux.guix.gnu.org/nar/lzip/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3: 0616kaa8pbhzrkvcnj191yw1pcd4gy73rvmcr2cv6j7pfxyrzj7y
  differing file:
    /lib/julia/sys.so

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]