[bug#36856] build: Add julia-build-system

["Nicolò Balzarotti" <anothersms@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 3069 bytes --]

Yeah I forgot to attach the patches. Here we go

Il giorno sab 3 ago 2019 alle ore 10:03 Nicolò Balzarotti <
anothersms@gmail.com> ha scritto:

> Hi, I'm back.
>
> I fixed the problem I had (I forgot the ./pre-inst-env -.-")
>
> Following this discussion, I'm adding ``native-search-paths'' to the julia
> package. Now, without any other hack (like the setup.jl one), it can find
> the packages but only when installed with guix package -i . It cannot find
> them when in an environment (guix environment --ad-hoc julia-package).
> Can you help?
>
> Except from this, I cleaned everything a bit, and added a draft of the
> documentation. I'm attaching the new patches here (without the julia-xyz
> part one, to which I'll add all package synopsis and description, and I'll
> split in multiple patches before re-submitting).
>
> Thanks, Nicolò
>
> Il giorno mar 30 lug 2019 alle ore 16:02 Nicolò Balzarotti <
> anothersms@gmail.com> ha scritto:
>
>> Thanks for the quick response!
>>
>> Well, the environment variable JULIA_LOAD_PATH (that inside julia is just
>> LOAD_PATH) works exactly like that (is a column-concatenated path list). It
>> just needs the special ":@stdlib" path to let julia find its standard
>> libraries.
>> Example:
>> > JULIA_LOAD_PATH=/my/new/path/:/profile/path/:@stdlib julia --startup=no
>> -E 'LOAD_PATH'
>> ["/my/new/path/", "/profile/path/", "@stdlib"]
>>
>> Is setting this variable from guix fine, or we need a special
>> GUIX_SOMETHING variable?
>> Where exactly should I set this?
>>
>> Thanks
>>
>> Il giorno mar 30 lug 2019 alle ore 14:33 Ricardo Wurmus <
>> rekado@elephly.net> ha scritto:
>>
>>>
>>> Julien Lepiller <julien@lepiller.eu> writes:
>>>
>>> > Le 30 juillet 2019 16:11:19 GMT+02:00, Ricardo Wurmus <
>>> rekado@elephly.net> a écrit :
>>> >>
>>> >>Nicolò Balzarotti <anothersms@gmail.com> writes:
>>> >>
>>> >>> 3. By adding a simple startup.jl script, inspired by what emacs does.
>>> >>This
>>> >>> is simplified so I'm not sure it works for every corner case.
>>> >>> #+begin_src julia
>>> >>> let paths = [expanduser("~/.guix-profile"),
>>> >>"/run/current-system/profile"]
>>> >>>     ("GUIX_ENVIRONMENT" in keys(ENV)) && push!(paths,
>>> >>> ENV["GUIX_ENVIRONMENT"])
>>> >>>     empty!(LOAD_PATH)
>>> >>>     push!.(Ref(LOAD_PATH), joinpath.(paths, "share/julia/packages/"))
>>> >>>     push!(LOAD_PATH, "@stdlib")
>>> >>>     push!.(Ref(DEPOT_PATH), joinpath.(paths, "share/julia/"))
>>> >>>     nothing
>>> >>> end
>>> >>> #+end_src
>>> >>
>>> >>Could this perhaps be handled by a profile hook that is included only
>>> >>when the profile manifest contains a julia package?
>>> >
>>> > Or simply with an environment variable? GUIX_JULIA_PATH or something?
>>>
>>> If this is the route you go down, please ensure that it can be used as a
>>> search path with more than one directory.  This would make it possible
>>> to extend a Julia environment with the contents of more than one
>>> profile.
>>>
>>> --
>>> Ricardo
>>>
>>>

[-- Attachment #1.2: Type: text/html, Size: 4602 bytes --]

[-- Attachment #2: 0001-build-Add-julia-build-system.patch --]
[-- Type: text/x-patch, Size: 13390 bytes --]

From 5538e759e6a50041a40c350cf8c08c0b4c0f647e Mon Sep 17 00:00:00 2001
From: nixo <nicolo@nixo.xyz>
Date: Mon, 29 Jul 2019 18:45:26 +0200
Subject: [PATCH 1/2] build: Add julia-build-system.

* guix/build/julia-build-system.scm: New file.
* guix/build-system/julia.scm: New file.
* Makefile.am: Added new files.
---
 Makefile.am                       |   2 +
 doc/guix.texi                     |  23 +++++
 guix/build-system/julia.scm       | 132 +++++++++++++++++++++++++++++
 guix/build/julia-build-system.scm | 135 ++++++++++++++++++++++++++++++
 4 files changed, 292 insertions(+)
 create mode 100644 guix/build-system/julia.scm
 create mode 100644 guix/build/julia-build-system.scm

diff --git a/Makefile.am b/Makefile.am
index 0bd85e8fcf..d14487045a 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -125,6 +125,7 @@ MODULES =					\
   guix/build-system/gnu.scm			\
   guix/build-system/guile.scm			\
   guix/build-system/haskell.scm			\
+  guix/build-system/julia.scm			\
   guix/build-system/linux-module.scm		\
   guix/build-system/node.scm			\
   guix/build-system/perl.scm			\
@@ -183,6 +184,7 @@ MODULES =					\
   guix/build/texlive-build-system.scm		\
   guix/build/waf-build-system.scm		\
   guix/build/haskell-build-system.scm		\
+  guix/build/julia-build-system.scm		\
   guix/build/linux-module-build-system.scm	\
   guix/build/store-copy.scm			\
   guix/build/json.scm				\
diff --git a/doc/guix.texi b/doc/guix.texi
index ccc36a8a97..e8aa4bc906 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6032,6 +6032,29 @@ Packages built with @code{guile-build-system} must provide a Guile package in
 their @code{native-inputs} field.
 @end defvr
 
+@defvr {Scheme Variable} julia-build-system
+This variable is exported by @code{(guix build-system julia)}.  It implements
+the build procedure used by @uref{https://julialang.org/, julia} packages,
+which essentially is similar to running @command{julia -e 'using Pkg;
+Pkg.add(package)'} in an environment where @code{JULIA_LOAD_PATH} contains the
+paths to all Julia package inputs.  Tests are run not run.
+
+Julia packages require the source @code{file-name} to be the real name of the
+package, correctly capitalized.
+
+For packages requiring shared library dependencies, you may need to write the
+@file{/deps/deps.jl} file manually. It's usually a line of @code{const
+variable = /gnu/store/libary.so} for each dependency, plus a void function
+@code{check_deps() = nothing}.
+
+Some older packages that aren't using @file{Package.toml} yet, will require
+this file to be created, too. The function @code{julia-create-package-toml}
+helps creating the file. You need to pass the outputs and the source of the
+package, it's name (the same as the @code{file-name} parameter), the package
+uuid, the package version, and a list of dependencies specified by their name
+and their uuid.
+@end defvr
+
 @defvr {Scheme Variable} minify-build-system
 This variable is exported by @code{(guix build-system minify)}.  It
 implements a minification procedure for simple JavaScript packages.
diff --git a/guix/build-system/julia.scm b/guix/build-system/julia.scm
new file mode 100644
index 0000000000..4e0d6ccddf
--- /dev/null
+++ b/guix/build-system/julia.scm
@@ -0,0 +1,132 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2019 Nicolò Balzarotti <nicolo@nixo.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build-system julia)
+  #:use-module ((guix build julia-build-system))
+  #:use-module (gnu packages julia)
+  #:use-module (guix store)
+  #:use-module (guix utils)
+  #:use-module (guix packages)
+  #:use-module (guix derivations)
+  #:use-module (guix search-paths)
+  #:use-module (guix build-system)
+  #:use-module (guix build-system gnu)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-26)
+  #:export (%julia-build-system-modules
+            julia-build
+            julia-build-system))
+
+;; Commentary:
+;;
+;; Standard build procedure for Julia packages.
+;;
+;; Code:
+
+(define %julia-build-system-modules
+  ;; Build-side modules imported by default.
+  `((guix build julia-build-system)
+    ,@%gnu-build-system-modules))
+
+(define (default-julia)
+  "Return the default Julia package."
+  ;; Lazily resolve the binding to avoid a circular dependency.
+  (let ((julia-mod (resolve-interface '(gnu packages julia))))
+    (module-ref julia-mod 'julia)))
+
+(define* (lower name
+                #:key source inputs native-inputs outputs system target
+                (julia julia)
+                #:allow-other-keys
+                #:rest arguments)
+  "Return a bag for NAME."
+  (define private-keywords
+    '(#:target #:julia #:inputs #:native-inputs))
+
+  (and (not target)			;XXX: no cross-compilation
+       (bag
+	 (name name)
+	 (system system)
+	 (host-inputs `(,@(if source
+			      `(("source" ,source))
+			      '())
+		        ,@inputs
+                        
+                        ;; Keep the standard inputs of 'gnu-build-system'.
+		        ,@(standard-packages)))
+	 (build-inputs `(("julia" ,julia)
+			 ,@native-inputs))
+	 (outputs outputs)
+	 (build julia-build)
+	 (arguments (strip-keyword-arguments private-keywords arguments)))))
+
+(define* (julia-build store name inputs
+                      #:key source
+                      (tests? #f)
+                      (phases '(@ (guix build julia-build-system)
+                                  %standard-phases))
+                      (outputs '("out"))
+                      (search-paths '())
+                      (system (%current-system))
+                      (guile #f)
+                      (imported-modules %julia-build-system-modules)
+                      (modules '((guix build julia-build-system)
+                                 (guix build utils))))
+  "Build SOURCE using Julia, and with INPUTS."
+  (define builder
+    `(begin
+       (use-modules ,@modules)
+       (julia-build #:name ,name
+                    #:source ,(match (assoc-ref inputs "source")
+                                (((? derivation? source))
+                                 (derivation->output-path source))
+                                ((source)
+                                 source)
+                                (source
+                                 source))
+                    #:system ,system
+                    #:tests? ,tests?
+                    #:phases ,phases
+                    #:outputs %outputs
+                    #:search-paths ',(map search-path-specification->sexp
+                                          search-paths)
+                    #:inputs %build-inputs)))
+
+  (define guile-for-build
+    (match guile
+      ((? package?)
+       (package-derivation store guile system #:graft? #f))
+      (#f                                         ; the default
+       (let* ((distro (resolve-interface '(gnu packages commencement)))
+              (guile  (module-ref distro 'guile-final)))
+         (package-derivation store guile system #:graft? #f)))))
+
+  (build-expression->derivation store name builder
+                                #:inputs inputs
+                                #:system system
+                                #:modules imported-modules
+                                #:outputs outputs
+                                #:guile-for-build guile-for-build))
+
+(define julia-build-system
+  (build-system
+   (name 'julia)
+   (description "The build system for Julia packages")
+   (lower lower)))
+
+;;; julia.scm ends here
diff --git a/guix/build/julia-build-system.scm b/guix/build/julia-build-system.scm
new file mode 100644
index 0000000000..728bb65893
--- /dev/null
+++ b/guix/build/julia-build-system.scm
@@ -0,0 +1,135 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2019 Nicolò Balzarotti <nicolo@nixo.xyz>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+
+(define-module (guix build julia-build-system)
+  #:use-module ((guix build gnu-build-system) #:prefix gnu:)
+  #:use-module (guix build utils)
+  #:use-module (ice-9 match)
+  #:export (%standard-phases
+	    julia-create-package-toml
+            julia-build))
+
+;; Commentary:
+;;
+;; Builder-side code of the standard build procedure for Julia packages.
+;;
+;; Code:
+
+(define (invoke-julia code)
+  (invoke "julia" "-e" code))
+
+;; subpath where we store the package content
+(define %package-path "/share/julia/packages/")
+
+(define (generate-load-path inputs outputs)
+  (string-append
+   (string-join (map (match-lambda
+                       ((_ . path)
+                        (string-append path %package-path)))
+                     ;; Restrict to inputs beginning with "julia-".
+                     (filter (match-lambda
+                               ((name . _)
+                                (string-prefix? "julia-" name)))
+                             inputs))
+                ":")
+   (string-append ":" (assoc-ref outputs "out") %package-path)
+   ;; stdlib is always required to find Julia's standard libraries.
+   ;; usually there are other two paths in this variable:
+   ;; "@" and "@v#.#"
+   ":@stdlib"))
+
+(define* (install #:key source inputs outputs #:allow-other-keys)
+  (let* ((out (assoc-ref outputs "out"))
+	 (package-dir (string-append out %package-path
+				     (string-append
+				      (strip-store-file-name source)))))
+    (setenv "JULIA_LOAD_PATH" (generate-load-path inputs outputs))
+    (mkdir-p package-dir)
+    (copy-recursively source package-dir))
+  #t)
+
+;; TODO: Precompilation is working, but I don't know how to tell
+;; julia to use use it. If (on rantime) we set HOME to
+;; store path, julia tries to write files there (failing)
+(define* (precompile #:key source inputs outputs #:allow-other-keys)
+  (let* ((out (assoc-ref outputs "out"))
+	 (builddir (string-append out "/share/julia/"))
+	 (package (strip-store-file-name source)))
+    (mkdir-p builddir)
+    (setenv "JULIA_DEPOT_PATH" builddir)
+    (setenv "JULIA_LOAD_PATH" (generate-load-path inputs outputs))
+    ;; Actual precompilation
+    (invoke-julia (string-append "using " package)))
+  #t)
+
+(define* (check #:key source inputs outputs #:allow-other-keys)
+  (let* ((out (assoc-ref outputs "out"))
+	 (package (strip-store-file-name source))
+	 (builddir (string-append out "/share/julia/")))
+    (setenv "JULIA_DEPOT_PATH" builddir)
+    (setenv "JULIA_LOAD_PATH" (generate-load-path inputs outputs))
+    (invoke-julia (string-append "using Pkg;Pkg.test(\"" package "\")")))
+  #t)
+
+(define (julia-create-package-toml outputs source
+				   name uuid version
+				   deps)
+  "Some packages are not using the new Package.toml dependency specifications.
+Write this file manually, so that Julia can find its dependencies."
+  (let ((f (open-file
+	    (string-append
+	     (assoc-ref outputs "out")
+	     %package-path
+	     (string-append
+	      name "/Project.toml"))
+	    "w")))
+    (display (string-append
+	      "
+name = \"" name "\"
+uuid = \"" uuid "\"
+version = \"" version "\"
+") f)
+    (when (not (null? deps))
+      (display "[deps]\n" f)
+      (for-each (lambda dep
+		  (display (string-append (car (car dep)) " = \"" (cdr (car dep)) "\"\n")
+			   f))
+		deps))
+    (close-port f))
+  #t)
+
+(define %standard-phases
+  (modify-phases gnu:%standard-phases
+    (delete 'check) ; tests must be run after installation
+    (replace 'install install)
+    (add-after 'install 'precompile precompile)
+    ;; (add-after 'install 'check check)
+    ;; TODO: In the future we could add a "system-image-generation" phase
+    ;; where we use PackageCompiler.jl to speed up package loading times
+    (delete 'configure)
+    (delete 'bootstrap)
+    (delete 'patch-usr-bin-file)
+    (delete 'build)))
+
+(define* (julia-build #:key inputs (phases %standard-phases)
+                      #:allow-other-keys #:rest args)
+  "Build the given Julia package, applying all of PHASES in order."
+  (apply gnu:gnu-build
+         #:inputs inputs #:phases phases
+         args))
-- 
2.22.0


[-- Attachment #3: 0002-packages-julia-set-JULIA_LOAD_PATH-to-be-able-to-fin.patch --]
[-- Type: text/x-patch, Size: 934 bytes --]

From a0b0bf1350023404b782656c2a093ca0bfd6abb8 Mon Sep 17 00:00:00 2001
From: nixo <nicolo@nixo.xyz>
Date: Sat, 3 Aug 2019 11:19:11 +0200
Subject: [PATCH 2/2] packages: julia: set JULIA_LOAD_PATH to be able to find
 packages

---
 gnu/packages/julia.scm | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index 43254f25fd..65a5e42beb 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -482,6 +482,10 @@
        ("patchelf" ,patchelf)
        ("pkg-config" ,pkg-config)
        ("python" ,python-2)))
+    (native-search-paths
+     (list (search-path-specification
+            (variable "JULIA_LOAD_PATH")
+            (files (list "share/julia/packages/")))))
     ;; Julia is not officially released for ARM and MIPS.
     ;; See https://github.com/JuliaLang/julia/issues/10639
     (supported-systems '("i686-linux" "x86_64-linux" "aarch64-linux"))
-- 
2.22.0