[bug#38546] [00/11] Update Julia, Fix precompilation, add HTTP.jl

["Nicolò Balzarotti" <anothersms@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2268 bytes --]

Hi Simon,

Sorry, I forgot to send the dsfmt patch.  Also, julia's
SOURCE_DATE_EPOCH patch was named differently. I've fixed this in theattached patches.  You need to apply Add-dsfmt.patch, Update-to-1.3.1
and then julia-use-SOURCE_DATE_EPOCH.

About reproducibility: if I'm not wrong, sys.so contains Base library
precompiled ([1]).  Precompilation is still non deterministic (here's
[2] an issue on github). Something I did to check precompilation:

Since I'm tired of this, I've started investigating better.

Loaded a guix container
#+begin_src bash
mkdir -p /tmp/jlprecompiled/
./pre-inst-env guix environment -u nixbld --no-cwd --container --network --expose=$(realpath determinism) --share=/tmp/jlprecompiled/ --ad-hoc julia coreutils nss-certs curl tar gzip libfaketime
#+end_src

from this container, run some time this:

#+begin_src bash
export SSL_CERT_DIR="$GUIX_ENVIRONMENT/etc/ssl/certs"
export SSL_CERT_FILE="$SSL_CERT_DIR/ca-certificates.crt"
export GIT_SSL_CAINFO="$SSL_CERT_FILE"

SOURCE_DATE_EPOCH=1 julia --startup-file=no -q -e 'using Pkg; Pkg.instantiate()'
SOURCE_DATE_EPOCH=1 faketime '2008-12-24 08:15:42' julia --startup-file=no -q -e 'using Random; using Pkg; Pkg.add("Compat"); Random.seed!(0); using Compat'

cp -r ~/.julia/compiled/v1.3/Compat/ /tmp/jlprecompiled/$RANDOM
sha256sum ~/.julia/compiled/v1.3/Compat/*
# 6c10d65f6cd9cdd5b98e4d5e2b82b7bd00508f0daccef0967dec92cb6468052e  /home/nixbld/.julia/compiled/v1.3/Compat/GSFWK_Pua4I.ji
ls -lah ~/.julia/compiled/v1.3/Compat/*
# -rw------- 1 nixbld users 102K Jan 21 07:39 /home/nixbld/.julia/compiled/v1.3/Compat/GSFWK_Pua4I.ji
#+end_src

I could not get the same results twice (also, size differs).  I'll work
on this on some spare time (for example, there are other places where
SOURCE_DATE_EPOCH can be used, but this [3] is a problem I need to
solve first).

Maybe 1.3.1 (when reviewed) can be merged, since we have
the same problem with julia 1.1, but we can wait for the
source-date-epoch and julia-xyz patches until we solve this.

What do you think?

Thanks, Nicolò

[1] https://docs.julialang.org/en/v1/devdocs/sysimg/
[2] https://github.com/JuliaLang/julia/issues/25900
[3] https://github.com/JuliaLang/julia/issues/34115


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-dsfmt.patch --]
[-- Type: text/x-patch, Size: 3796 bytes --]

From 337ddd64ce4f3a00f33a90496822500dc6df2010 Mon Sep 17 00:00:00 2001
From: nixo <nicolo@nixo.xyz>
Date: Sat, 28 Dec 2019 00:10:48 +0100
Subject: [PATCH] gnu: Add dsfmt.

* gnu/packages/maths.scm (dsfmt): New variable.
---
 gnu/packages/maths.scm | 63 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 8364d528fd..1d35d880bd 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -372,6 +372,69 @@ semiconductors.")
     (license license:gpl3+)
     (home-page "https://www.gnu.org/software/dionysus/")))
 
+(define-public dsfmt
+  (package
+    (name "dsfmt")
+    (version "2.2.3")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append
+                "http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/SFMT/dSFMT-src-"
+                version ".tar.gz"))
+              (sha256
+               (base32
+                "03kaqbjbi6viz0n33dk5jlf6ayxqlsq4804n7kwkndiga9s4hd42"))
+              ;; julia needs those (and is the only program depending on dsfmt now)
+              (patches
+               (list
+                (origin (method url-fetch)
+                        (uri
+                         "https://raw.githubusercontent.com/JuliaLang/julia/v1.3.0/deps/patches/dSFMT.c.patch")
+                        (sha256 (base32 "09mhv11bms8jsmkmdqvlcgljwhzw3b6n9nncpi2b6dla9798hw2y"))
+                        (file-name "dSFMT.c.patch"))
+                (origin (method url-fetch)
+                        (uri
+                         "https://raw.githubusercontent.com/JuliaLang/julia/v1.3.0/deps/patches/dSFMT.h.patch")
+                        (sha256 (base32 "1py5rd0yxic335lzka23f6x2dhncrpizpyrk57gi2f28c0p98y5n"))
+                        (file-name "dSFMT.h.patch"))))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (delete 'configure)            ;no configure script
+         (replace 'build
+           ;; upstream Makefile does not build a shared library
+           (lambda _
+             (invoke
+              "gcc"
+              ;; flags copied from julia
+              ;; https://github.com/JuliaLang/julia/blob/v1.3.0/deps/dsfmt.mk
+              "-DNDEBUG" "-DDSFMT_MEXP=19937"
+              "-fPIC" "-DDSFMT_DO_NOT_USE_OLD_NAMES"
+              "-O3" "-finline-functions" "-fomit-frame-pointer"
+              "-fno-strict-aliasing" "--param" "max-inline-insns-single=1800"
+              "-Wmissing-prototypes" "-Wall" "-std=c99" "-shared" "dSFMT.c"
+              "-o" "libdSFMT.so")))
+         (replace 'install              ;no "install" target
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out")
+                                       "/lib"))
+                   (inc (string-append (assoc-ref outputs "out")
+                                       "/include")))
+               (install-file "libdSFMT.so" lib)
+               (install-file "dSFMT.h" inc)
+               #t))))))
+    (synopsis "Double precision SIMD-oriented Fast Mersenne Twister")
+    (description "Speed up Fast Mersenne Twister generation by avoiding the
+expensive conversion of integer to double (floating point). dSFMT directly
+generates double precision floating point pseudorandom numbers which have the
+IEEE Standard for Binary Floating-Point Arithmetic (ANSI/IEEE Std 754-1985)
+format. dSFMT is only available on the CPUs which use IEEE 754 format double
+precision floating point numbers.")
+    (home-page "http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/SFMT/")
+    (license license:bsd-3)))
+
 (define-public gsl
   (package
     (name "gsl")
-- 
2.25.0


[-- Attachment #3: 0001-gnu-julia-Update-to-1.3.1.patch --]
[-- Type: text/x-patch, Size: 21843 bytes --]

From ae58a1dd7b24989e281b07d9a90e1bb6de5199e2 Mon Sep 17 00:00:00 2001
From: nixo <nicolo@nixo.xyz>
Date: Sat, 18 Jan 2020 15:14:32 +0100
Subject: [PATCH] gnu: julia: Update to 1.3.1.

* gnu/packages/julia.scm (julia): Update to 1.3.1.
---
 gnu/packages/julia.scm | 281 +++++++++++++++++------------------------
 1 file changed, 115 insertions(+), 166 deletions(-)

diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index 65a5e42beb..3455dc6239 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015, 2016, 2017 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2016 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2020 Nicolò Balzarotti <nicolo@nixo.xyz>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -28,6 +29,7 @@
   #:use-module (gnu packages algebra)
   #:use-module (gnu packages base)
   #:use-module (gnu packages compression)
+  #:use-module (gnu packages curl)
   #:use-module (gnu packages elf)
   #:use-module (gnu packages gcc)
   #:use-module (gnu packages llvm)
@@ -43,18 +45,30 @@
   #:use-module (gnu packages textutils)
   #:use-module (gnu packages tls)
   #:use-module (gnu packages version-control)
+  #:use-module (gnu packages ssh)
   #:use-module (gnu packages wget)
   #:use-module (ice-9 match))
 
+(define (julia-patch-url version name)
+  (string-append "https://raw.githubusercontent.com/JuliaLang/julia/v" version
+		 "/deps/patches/" name ".patch"))
+
+(define (julia-patch name sha)
+  (let ((version "1.3.1"))
+    (origin (method url-fetch)
+	    (uri (julia-patch-url version name))
+	    (sha256 (base32 sha))
+	    (file-name name))))
+
 (define libuv-julia
-  (let ((commit "26dbe5672c33fc885462c509fe2a9b36f35866fd")
-        (revision "6"))
+  (let ((commit "35b1504507a7a4168caae3d78db54d1121b121e1"))
     ;; When upgrading Julia, also upgrade this.
-    ;; Get the commit from https://github.com/JuliaLang/julia/blob/v1.1.1/deps/libuv.version
+    ;; Get the commit from
+    ;; https://github.com/JuliaLang/julia/blob/v1.3.1/deps/libuv.version
     (package
       (inherit libuv)
       (name "libuv-julia")
-      (version (string-append "1.9.0-" revision "." (string-take commit 8)))
+      (version (string-append "2.0.0" "." (string-take commit 8)))
       (source (origin
                 (method git-fetch)
                 (uri (git-reference
@@ -63,7 +77,7 @@
                 (file-name (string-append name "-" version "-checkout"))
                 (sha256
                  (base32
-                  "17pn2xmqaramilx897s9grs966i5246gi6sric5alch4g9j4685n"))))
+                  "0dn3v6fdp1z382pqg3nhjzk60l61ky9b65mfgaj29fv2da95rwjs"))))
       (build-system gnu-build-system)
       (arguments
        (substitute-keyword-arguments (package-arguments libuv)
@@ -72,16 +86,27 @@
              (delete 'autogen)))))
       (home-page "https://github.com/JuliaLang/libuv"))))
 
-(define (llvm-patch-url version name)
-  (string-append "https://raw.githubusercontent.com/JuliaLang/julia/v" version
-		 "/deps/patches/" name))
-
-(define (llvm-patch name sha)
-  (let ((version "1.1.1"))
-    (origin (method url-fetch)
-	    (uri (llvm-patch-url version name))
-	    (sha256 (base32 sha))
-	    (file-name name))))
+(define libunwind-julia
+  ;; julia requires a patched version
+  (package
+    (inherit libunwind)
+    (name "libunwind-julia")
+    (version "1.3.1")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "mirror://savannah/libunwind/libunwind-"
+                                  version ".tar.gz"))
+              (sha256
+               (base32
+                "1y0l08k6ak1mqbfj6accf9s5686kljwgsl4vcqpxzk5n74wpm6a3"))
+              (patches
+               (list
+	        (julia-patch "libunwind-prefer-extbl"
+                             "0lr4dafw8qyfh8sw8hhbwkql1dlhqv8px7k81y2l20hhxfgnh2m1")
+                (julia-patch "libunwind-static-arm"
+                             "1jk3bmiw61ypcchqkk1fyg5wh8wpggk574wxyfyaic870zh3lhgq")))))
+    (build-system gnu-build-system)
+    (home-page "https://github.com/JuliaLang/libuv")))
 
 (define llvm-julia
   (package
@@ -99,39 +124,39 @@
               ;; Discussion: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919628
               (patches
                (list
-	        (llvm-patch "llvm-6.0-D44650.patch"
+	        (julia-patch "llvm-6.0-D44650"
 			    "1336q4vqayr94wdcnlmcxh90mjdh34dzw9x2cbiqjnx9b1j8fxyb")
-	        (llvm-patch "llvm-6.0-DISABLE_ABI_CHECKS.patch"
+	        (julia-patch "llvm-6.0-DISABLE_ABI_CHECKS"
 			    "014fawd1ba7yckalypfld22zgic87x9nx3cim42zrwygywd36pyg")
-	        (llvm-patch "llvm-6.0-NVPTX-addrspaces.patch"
+	        (julia-patch "llvm-6.0-NVPTX-addrspaces"
 			    "1qdi2zmrjsrj0h84zv2vyly2hjcn4f67mfy0s1q353g4v4jkscqc")
-	        (llvm-patch "llvm-6.0.0_D27296-libssp.patch"
+	        (julia-patch "llvm-6.0.0_D27296-libssp"
 			    "0s5hi2r1j63i8m6ig1346crx2aiv9f7rgb3mg80kw1wx5y7pdpfh")
-	        (llvm-patch "llvm-D27629-AArch64-large_model_6.0.1.patch"
+	        (julia-patch "llvm-D27629-AArch64-large_model_6.0.1"
 			    "1qrshmlqvnasdyc158vfn3hnbigqph3lsq7acb9w8lwkpnnm2j4z")
-	        (llvm-patch "llvm-D34078-vectorize-fdiv.patch"
+	        (julia-patch "llvm-D34078-vectorize-fdiv"
 			    "1696hg84a0jxcnggvqsc2cdp271hf9a44p4qsd078qm1mfawkaay")
-	        (llvm-patch "llvm-D42262-jumpthreading-not-i1.patch"
+	        (julia-patch "llvm-D42262-jumpthreading-not-i1"
 			    "1c8w210gwidbnkkw8anp17dk5pnxws2fl3mb2qxh7y9wzfpixgaq")
-	        (llvm-patch "llvm-D44892-Perf-integration.patch"
+	        (julia-patch "llvm-D44892-Perf-integration"
 			    "0r37jd0ssh2k1pndkfd5blgpg9z90im4vlzprhb0n0wwz45g4b05")
-	        (llvm-patch "llvm-D46460.patch"
+	        (julia-patch "llvm-D46460"
 			    "1miqgswdc0qvbaf4571c2xkxyp9ais06b1bcpa83sq22vr4hbsfb")
-	        (llvm-patch "llvm-D49832-SCEVPred.patch"
+	        (julia-patch "llvm-D49832-SCEVPred"
 			    "0v5c88hgqj6dymv3j86ca5mhpqab5fbnrvjiw1nvnrnya9l4dlbn")
-	        (llvm-patch "llvm-D50010-VNCoercion-ni.patch"
+	        (julia-patch "llvm-D50010-VNCoercion-ni"
 			    "0iblb3q1xixwrb12jpb89h3ywmqmzdp6aqp416j4ncwakyjhhfkp")
-	        (llvm-patch "llvm-D50167-scev-umin.patch"
+	        (julia-patch "llvm-D50167-scev-umin"
 			    "1f2rakcnnyhr7w10k7gqg0k0491pyvx5ijplivw557f714ys3q6v")
-	        (llvm-patch "llvm-OProfile-line-num.patch"
+	        (julia-patch "llvm-OProfile-line-num"
 			    "1jvbbmwyags0xfwamb13qrf3rgcz9i1r03m9lava7swag8xb78c7")
-	        (llvm-patch "llvm-PPC-addrspaces.patch"
+	        (julia-patch "llvm-PPC-addrspaces"
 			    "1f23nhsxh2s3jskbgs7da9nwg3s1hrkbk5aahl08x41wi3mny01p")
-	        (llvm-patch "llvm-rL323946-LSRTy.patch"
+	        (julia-patch "llvm-rL323946-LSRTy"
 			    "10cz3vy1yw0w643z7xx021wa4kymx9fcm3bjg61s6vzdqd6d9fns")
-	        (llvm-patch "llvm-rL326967-aligned-load.patch"
+	        (julia-patch "llvm-rL326967-aligned-load"
 			    "04jxnv32yj5x17hqhi8g2p8rhgp38gmjzr871w7z8s44pq10v9v4")
-	        (llvm-patch "llvm-rL327898.patch"
+	        (julia-patch "llvm-rL327898"
 			    "15ah49gbsll23z28kpyahi5vl0fh3fkxcgd1zmxxdcl96s3x8bnq")))))
     (arguments
      (substitute-keyword-arguments (package-arguments llvm-6)
@@ -158,7 +183,7 @@
 (define-public julia
   (package
     (name "julia")
-    (version "1.1.1")
+    (version "1.3.1")
     (source (origin
               (method url-fetch)
               (uri (string-append
@@ -166,7 +191,7 @@
                     version "/julia-" version ".tar.gz"))
               (sha256
                (base32
-                "0hk983mywimclgnjc41zmlppm5kfdz2aj85ky07p49ilcqxi998f"))))
+                "1nwkmr9j55g1zkxdchnid1h022s0is52vx23niksshgvh793g41x"))))
     (build-system gnu-build-system)
     (arguments
      `(#:test-target "test"
@@ -188,30 +213,24 @@
          (add-after 'unpack 'prepare-deps
            (lambda* (#:key inputs #:allow-other-keys)
              (mkdir "deps/srccache")
-             (copy-file (assoc-ref inputs "dsfmt")
-                        "deps/srccache/dsfmt-2.2.3.tar.gz")
+             ;; no USE_SYSTEM_{OBJCONV/LIBWHICH}
              (copy-file (assoc-ref inputs "objconv")
                         "deps/srccache/objconv.zip")
-             (copy-file (assoc-ref inputs "suitesparse")
-                        "deps/srccache/SuiteSparse-4.4.5.tar.gz")
-             (copy-file (string-append (assoc-ref inputs "virtualenv")
-                                       "/bin/virtualenv")
-                        "julia-env")
              (copy-file (assoc-ref inputs "libwhich")
                         (string-append "deps/srccache/libwhich-"
                                        "81e9723c0273d78493dc8c8ed570f68d9ce7e89e"
                                        ".tar.gz"))
-             (copy-file (assoc-ref inputs "rmath")
-                        "deps/srccache/Rmath-julia-0.1.tar.gz")
-
 	     ;; needed by libwhich
 	     (setenv "LD_LIBRARY_PATH"
 		     (string-join (map (lambda (pkg)
                                          (string-append (assoc-ref inputs pkg)
                                                         "/lib"))
-                                       '("arpack-ng" "fftw" "gmp" "lapack"
-			                 "libgit2" "mpfr" "openblas" "openlibm"
-			                 "openspecfun" "pcre2"))
+                                       '("arpack-ng" "curl" "dsfmt"
+                                         "gmp" "lapack"
+                                         "libssh2" "libgit2"
+                                         "mbedtls" "mpfr"
+                                         "openblas" "openlibm" "pcre2"
+                                         "suitesparse"))
                                   ":"))
 	     #t))
          ;; FIXME: Building the documentation requires Julia packages that
@@ -233,44 +252,12 @@
           ;; using the output of "/sbin/ldconfig -p".  Since ldconfig is not
           ;; used in Guix, we patch runtime_ccall.cpp to contain a static map.
           (lambda* (#:key inputs #:allow-other-keys)
-            (use-modules (ice-9 match))
-            (substitute* "src/runtime_ccall.cpp"
-              ;; Patch out invocations of '/sbin/ldconfig' to avoid getting
-              ;; error messages about missing '/sbin/ldconfig' on Guix System.
-              (("popen\\(.*ldconfig.*\\);")
-               "NULL;\n")
-
-              ;; Populate 'sonameMap'.
-              (("jl_read_sonames.*;")
-               (string-join
-                (map (match-lambda
-                       ((input libname soname)
-                        (string-append
-                         "sonameMap[\"" libname "\"] = "
-                         "\"" (assoc-ref inputs input) "/lib/" soname "\";")))
-                     '(("libc"        "libc"           "libc.so.6")
-                       ("pcre2"       "libpcre2-8"     "libpcre2-8.so")
-                       ("mpfr"        "libmpfr"        "libmpfr.so")
-                       ("openblas"    "libblas"        "libopenblas.so")
-                       ("arpack-ng"   "libarpack"      "libarpack.so")
-                       ("lapack"      "liblapack"      "liblapack.so")
-                       ("libgit2"     "libgit2"        "libgit2.so")
-                       ("gmp"         "libgmp"         "libgmp.so")
-                       ("openspecfun" "libopenspecfun" "libopenspecfun.so")
-                       ("fftw"        "libfftw3"       "libfftw3_threads.so")
-                       ("fftwf"       "libfftw3f"      "libfftw3f_threads.so"))))))
             (substitute* "base/math.jl"
               (("const libm = Base.libm_name")
                (string-append "const libm = \""
                               (assoc-ref inputs "openlibm")
                               "/lib/libopenlibm.so"
-                              "\""))
-              (("const openspecfun = \"libopenspecfun\"")
-               (string-append "const openspecfun = \""
-                              (assoc-ref inputs "openspecfun")
-                              "/lib/libopenspecfun.so"
-                              "\"")))
-            #t))
+                              "\"")))))
          (add-before 'build 'fix-include-and-link-paths
           (lambda* (#:key inputs #:allow-other-keys)
             ;; LIBUTF8PROC is a linker flag, not a build target.  It is
@@ -278,10 +265,10 @@
             ;; collection of build targets and a list of libraries to link
             ;; against.
             (substitute* "src/flisp/Makefile"
-              (("\\$\\(BUILDDIR\\)/\\$\\(EXENAME\\): \\$\\(OBJS\\) \\$\\(LIBFILES_release\\)")
-               "$(BUILDDIR)/$(EXENAME): $(OBJS) $(LLT_release)")
-              (("\\$\\(BUILDDIR\\)/\\$\\(EXENAME\\)-debug: \\$\\(DOBJS\\) \\$\\(LIBFILES_debug\\)")
-               "$(BUILDDIR)/$(EXENAME)-debug: $(DOBJS) $(LLT_debug)"))
+               (("\\$\\(BUILDDIR\\)/\\$\\(EXENAME\\)\\$\\(EXE\\): \\$\\(OBJS\\) \\$\\(LIBFILES_release\\)")
+                "$(BUILDDIR)/$(EXENAME)$(EXE): $(OBJS) $(LLT_release)")
+               (("\\$\\(BUILDDIR\\)/\\$\\(EXENAME\\)-debug$(EXE): \\$\\(DOBJS\\) \\$\\(LIBFILES_debug\\)")
+                "$(BUILDDIR)/$(EXENAME)-debug\\$\\(EXE\\): $(DOBJS) $(LLT_debug)"))
 
             ;; The REPL must be linked with libuv.
             (substitute* "ui/Makefile"
@@ -296,43 +283,34 @@
                               "/include/uv/errno.h")))
             #t))
          (add-before 'build 'replace-default-shell
-          (lambda _
-            (substitute* "base/client.jl"
-              (("/bin/sh") (which "sh")))
-            #t))
-         (add-after 'unpack 'hardcode-paths
            (lambda _
-             (substitute* "stdlib/InteractiveUtils/src/InteractiveUtils.jl"
-               (("`which") (string-append "`" (which "which")))
-               (("`wget")  (string-append "`" (which "wget"))))
+             (substitute* "base/client.jl"
+               (("/bin/sh") (which "sh")))
              #t))
-         (add-before 'check 'disable-broken-tests
+         (add-before 'check 'fix-broken-tests
            (lambda _
-             (define (touch file-name)
-               (call-with-output-file file-name (const #t)))
-	     ;; FIXME: All git tests works except this one. But *THIS* "fix"
-             ;; is not working, so right now I'm disabling all libgit2.jl tests
-	     ;; (substitute* "stdlib/LibGit2/test/libgit2.jl"
-	     ;; (("!LibGit2.use_http_path(cfg, github_cred)") "true")
-	     ;; (("LibGit2.use_http_path(cfg, mygit_cred)") "true"))
-             (map (lambda (test)
-                    (delete-file test)
-                    (touch test))
-	          '("stdlib/Sockets/test/runtests.jl"
-		    "stdlib/Distributed/test/runtests.jl"
-                    ;; FIXME: see above
-		    "stdlib/LibGit2/test/libgit2.jl"))
-	     (substitute* "test/choosetests.jl"
-	       ;; These tests fail, probably because some of the input
-	       ;; binaries have been stripped and thus backtraces don't look
-	       ;; as expected.
-	       (("\"backtrace\",") "")
-	       (("\"cmdlineargs\",") ""))
+             (substitute* "test/choosetests.jl"
+               (("tests = testnames")
+                ;; Those failings are not deterministic.  They depends on the
+                ;; running order.  I think it depends on the number of
+                ;; runners, disabling it for now
+                ;; https://github.com/JuliaLang/julia/issues/34330
+                "tests = filter(e->!in(e,[\"backtrace\",\"exceptions\",\"client\",
+                                          \"stacktraces\",\"stress\",\"precompile\"]),
+                                testnames)"))
+             ;; When HOME is not set, julia calls uv_os_homedir, which in
+             ;; turns call getpwuid_r. Add the HOME env variable to the
+             ;; external julia call to fix this
+             (substitute* "test/cmdlineargs.jl"
+               (("\"JULIA_PROJECT\"") "\"HOME\"=>\"/tmp\", \"JULIA_PROJECT\""))
+             ;; Marking the test as broken as it's a known bug:
+             ;; https://github.com/JuliaLang/julia/issues/32377
+             (substitute* "stdlib/REPL/test/replcompletions.jl"
+               (("@test count") "@test_broken count"))
 	     #t)))
        #:make-flags
        (list
         (string-append "prefix=" (assoc-ref %outputs "out"))
-        (string-append "PREFIX=" (assoc-ref %outputs "out"))
 
         ;; Passing the MARCH flag is necessary to build binary substitutes for
         ;; the supported architectures.
@@ -345,26 +323,21 @@
            ;; platforms, e.g. when running "guix package --search="
            (_ "MARCH=UNSUPPORTED"))
 
-        "CONFIG_SHELL=bash"     ;needed to build bundled libraries
-        "USE_SYSTEM_DSFMT=0"    ;not packaged for Guix and upstream has no
-                                ;build system for a shared library.
+        "CONFIG_SHELL=bash"             ;needed to build bundled libraries
+        ;; list of "USE_SYSTEM_*" is here:
+        ;; https://github.com/JuliaLang/julia/blob/v1.3.1/Make.inc
+        "USE_SYSTEM_DSFMT=1"
+        "USE_SYSTEM_P7ZIP=1"
         "USE_SYSTEM_LAPACK=1"
         "USE_SYSTEM_BLAS=1"
         "USE_BLAS64=0"          ;needed when USE_SYSTEM_BLAS=1
         "LIBBLAS=-lopenblas"
         "LIBBLASNAME=libopenblas"
 
-        "USE_SYSTEM_FFTW=1"
-        "LIBFFTWNAME=libfftw3"
-        "LIBFFTWFNAME=libfftw3f"
-
-        ;; TODO: Suitesparse does not install shared libraries, so we cannot
-        ;; use the suitesparse package.
-        ;; "USE_SYSTEM_SUITESPARSE=1"
-        ;; (string-append "SUITESPARSE_INC=-I "
-        ;;                (assoc-ref %build-inputs "suitesparse")
-        ;;                "/include")
-
+        "USE_SYSTEM_SUITESPARSE=1"
+        (string-append "SUITESPARSE_INC=-I "
+                       (assoc-ref %build-inputs "suitesparse")
+                       "/include")
         "USE_GPL_LIBS=1"        ;proudly
         "USE_SYSTEM_UTF8PROC=1"
         (string-append "UTF8PROC_INC="
@@ -385,16 +358,19 @@
 	"USE_SYSTEM_PATCHELF=1"
 	"USE_SYSTEM_PCRE=1"
 	"USE_SYSTEM_OPENLIBM=1"
-
+        "USE_SYSTEM_LIBM=0"
+        "USE_SYSTEM_LIBSSH2=1"
+        "USE_SYSTEM_CURL=1"
 	"USE_SYSTEM_GMP=1"
 	"USE_SYSTEM_MPFR=1"
+	"USE_SYSTEM_MBEDTLS=1"
 	"USE_SYSTEM_ARPACK=1"
 	"USE_SYSTEM_LIBGIT2=1"
-	"USE_SYSTEM_ZLIB=1"
-	"USE_SYSTEM_OPENSPECFUN=1")))
+	"USE_SYSTEM_ZLIB=1")))
     (inputs
      `(("llvm" ,llvm-julia)
 
+       ("p7zip" ,p7zip)
        ;; The bundled version is 3.3.0 so stick to that version.  With other
        ;; versions, we get test failures in 'linalg/arnoldi' as described in
        ;; <https://bugs.gnu.org/30282>.
@@ -403,12 +379,12 @@
        ("coreutils" ,coreutils) ;for bindings to "mkdir" and the like
        ("lapack" ,lapack)
        ("openblas" ,openblas) ;Julia does not build with Atlas
-       ("libunwind" ,libunwind)
+       ("libunwind" ,libunwind-julia)
        ("openlibm" ,openlibm)
-       ("openspecfun" ,openspecfun)
+       ("mbedtls" ,mbedtls-apache)
+       ("curl" ,curl)
        ("libgit2" ,libgit2)
-       ("fftw" ,fftw)
-       ("fftwf" ,fftwf)
+       ("libssh2" ,libssh2)
        ("fortran" ,gfortran)
        ("libuv" ,libuv-julia)
        ("pcre2" ,pcre2)
@@ -418,31 +394,12 @@
        ("which" ,which)
        ("zlib" ,zlib)
        ("gmp" ,gmp)
-       ("virtualenv" ,python2-virtualenv)
+       ("suitesparse" ,suitesparse)
        ;; FIXME: The following inputs are downloaded from upstream to allow us
        ;; to use the lightweight Julia release tarball.  Ideally, these inputs
        ;; would eventually be replaced with proper Guix packages.
-
-       ;; TODO: run "make -f contrib/repackage_system_suitesparse4.make" to copy static lib
-       ;; Find dependency versions here:
-       ;; https://raw.githubusercontent.com/JuliaLang/julia/77a2c1e245c85812dc1c7687540beedecc52758f/deps/Versions.make
-       ("rmath"
-	,(origin
-	   (method git-fetch)
-	   (uri (git-reference
-                 (url "https://github.com/JuliaLang/Rmath-julia")
-                 (commit "v0.1")))
-           (file-name "rmath-julia-0.1-checkout")
-	   (sha256
-	    (base32
-	     "1zkpy0cg5zivq40zbhbdgj9128fqzs2j94wkwih8nc6xaj3gp9p6"))))
-       ("suitesparse"
-        ,(origin
-           (method url-fetch)
-           (uri "http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-4.4.5.tar.gz")
-           (sha256
-            (base32
-             "1jcbxb8jx5wlcixzf6n5dca2rcfx6mlcms1k2rl5gp67ay3bix43"))))
+       ;; Find dependencies versions here:
+       ;; https://raw.githubusercontent.com/JuliaLang/julia/v1.3.0/deps/Versions.make
        ("objconv"
         ,(origin
            (method url-fetch)
@@ -467,15 +424,7 @@
 	     (sha256
 	      (base32
 	       "1p7zg31kpmpbmh1znrk1xrbd074agx13b9q4dcw8n2zrwwdlbz3b")))))
-       ("dsfmt"
-        ,(origin
-           (method url-fetch)
-           (uri (string-append
-                 "http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/"
-                 "SFMT/dSFMT-src-2.2.3.tar.gz"))
-           (sha256
-            (base32
-             "03kaqbjbi6viz0n33dk5jlf6ayxqlsq4804n7kwkndiga9s4hd42"))))))
+       ("dsfmt" ,dsfmt)))
     (native-inputs
      `(("openssl" ,openssl)
        ("perl" ,perl)
-- 
2.25.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0001-gnu-julia-use-SOURCE_DATE_EPOCH-for-precompilation-t.patch --]
[-- Type: text/x-patch, Size: 3061 bytes --]

From f33153f5f185562d01cfbfbfed43732483132f84 Mon Sep 17 00:00:00 2001
From: nixo <nicolo@nixo.xyz>
Date: Sat, 18 Jan 2020 13:58:37 +0100
Subject: [PATCH] gnu: julia: use SOURCE_DATE_EPOCH for precompilation
 timestamp

* gnu/packages/patches/julia-fake-mtime-with-SOURCE_DATE_EPOCH.patch: new file
---
 gnu/local.mk                                  |  1 +
 gnu/packages/julia.scm                        |  4 ++-
 .../julia-SOURCE_DATE_EPOCH-mtime.patch       | 29 +++++++++++++++++++
 3 files changed, 33 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/julia-SOURCE_DATE_EPOCH-mtime.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 00ff3b8d61..fe8634b44d 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1010,6 +1010,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/java-xerces-bootclasspath.patch	\
   %D%/packages/patches/java-xerces-build_dont_unzip.patch	\
   %D%/packages/patches/java-xerces-xjavac_taskdef.patch	\
+  %D%/packages/patches/julia-SOURCE_DATE_EPOCH-mtime.patch      \
   %D%/packages/patches/jbig2dec-ignore-testtest.patch		\
   %D%/packages/patches/kdbusaddons-kinit-file-name.patch	\
   %D%/packages/patches/libvirt-create-machine-cgroup.patch	\
diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index 3455dc6239..794980c12f 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -191,7 +191,9 @@
                     version "/julia-" version ".tar.gz"))
               (sha256
                (base32
-                "1nwkmr9j55g1zkxdchnid1h022s0is52vx23niksshgvh793g41x"))))
+                "1nwkmr9j55g1zkxdchnid1h022s0is52vx23niksshgvh793g41x"))
+              (patches
+               (search-patches "julia-SOURCE_DATE_EPOCH-mtime.patch"))))
     (build-system gnu-build-system)
     (arguments
      `(#:test-target "test"
diff --git a/gnu/packages/patches/julia-SOURCE_DATE_EPOCH-mtime.patch b/gnu/packages/patches/julia-SOURCE_DATE_EPOCH-mtime.patch
new file mode 100644
index 0000000000..467e6d68d1
--- /dev/null
+++ b/gnu/packages/patches/julia-SOURCE_DATE_EPOCH-mtime.patch
@@ -0,0 +1,29 @@
+From e4dc28db1d70819505fd1a68fd9d2bfc4fb61a7d Mon Sep 17 00:00:00 2001
+From: nixo <nicolo@nixo.xyz>
+Date: Fri, 17 Jan 2020 11:28:30 +0100
+Subject: [PATCH] base: loading: support fake mtime with SOURCE_DATE_EPCOCH
+
+Do this when loading libraries
+---
+ base/loading.jl | 5 ++++-
+ 1 file changed, 4 insertions(+), 1 deletion(-)
+
+diff --git a/base/loading.jl b/base/loading.jl
+index 7f11a2d4fc..1b4686d1dc 100644
+--- a/base/loading.jl
++++ b/base/loading.jl
+@@ -807,7 +807,10 @@ function _include_dependency(mod::Module, _path::AbstractString)
+         path = normpath(joinpath(dirname(prev), _path))
+     end
+     if _track_dependencies[]
+-        push!(_require_dependencies, (mod, path, mtime(path)))
++        push!(_require_dependencies,
++              (mod, path,
++               haskey(ENV, "SOURCE_DATE_EPOCH") ?
++               parse(Float64, ENV["SOURCE_DATE_EPOCH"]) : mtime(path)))
+     end
+     return path, prev
+ end
+-- 
+2.24.1
+
-- 
2.25.0


[-- Attachment #5: Type: text/plain, Size: 971 bytes --]



zimoun <zimon.toutoune@gmail.com> writes:

> Hi Nicolò,
>
> Thank you for working on this.
> Julia rocks! ;-)
>
>
> However, I am lost with your patch set.
>
> I have tried only the first patch you attached. I hit this error:
>
> --8<---------------cut here---------------start------------->8---
> error: dsfmt: unbound variable
> --8<---------------cut here---------------end--------------->8---
>
> So something is missing. :-)
>
> Then I notice that you change the list of 'patches' and you "forgot"
> to send them too. :-)
>
>
> Almost related, does this update fix the reproducibility bug [1]? For example,
>
> --8<---------------cut here---------------start------------->8---
> ./pre-inst-env guix build julia
> ./pre-inst-env guix build julia --check --no-grafts -K
> --8<---------------cut here---------------end--------------->8---
>
>
> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22304
>
>
> All the best,
> simon