[bug#32483] [PATCH] import: stackage: Support recursive importing.

[bug#32483] [PATCH] import: stackage: Support recursive importing.

[Ricardo Wurmus]

* guix/import/hackage.scm (hackage-name->package-name): Export procedure.
* guix/import/stackage.scm (lts-info-packages-lts-info): Fix match expression.
(stackage-recursive-import): New procedure.
(stackage->guix-package): Memoize results.
* guix/scripts/import/stackage.scm (show-help, %options,
guix-import-stackage): Support recursive importing.
* doc/guix.texi (Invoking guix import): Document option.
---
 doc/guix.texi                    |  7 ++++-
 guix/import/hackage.scm          |  1 +
 guix/import/stackage.scm         | 45 +++++++++++++++++++------------
 guix/scripts/import/stackage.scm | 46 ++++++++++++++++++++++++--------
 4 files changed, 70 insertions(+), 29 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 0b72e5d8c..54f8cf5ea 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6699,9 +6699,14 @@ Specific command-line options are:
 @itemx -t
 Do not include dependencies required only by the test suites.
 @item --lts-version=@var{version}
-@itemx -r @var{version}
+@itemx -l @var{version}
 @var{version} is the desired LTS release version.  If omitted the latest
 release is used.
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
 @end table
 
 The command below imports metadata for the @code{HTTP} Haskell package
diff --git a/guix/import/hackage.scm b/guix/import/hackage.scm
index 3c00f680b..54301de2e 100644
--- a/guix/import/hackage.scm
+++ b/guix/import/hackage.scm
@@ -44,6 +44,7 @@
             %hackage-updater
 
             guix-package->hackage-name
+            hackage-name->package-name
             hackage-fetch
             hackage-source-url
             hackage-cabal-url
diff --git a/guix/import/stackage.scm b/guix/import/stackage.scm
index ec93fbced..afd5d997a 100644
--- a/guix/import/stackage.scm
+++ b/guix/import/stackage.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017 Federico Beffa <beffa@fbengineering.ch>
+;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -25,10 +26,12 @@
   #:use-module (srfi srfi-35)
   #:use-module (guix import json)
   #:use-module (guix import hackage)
+  #:use-module (guix import utils)
   #:use-module (guix memoization)
   #:use-module (guix packages)
   #:use-module (guix upstream)
   #:export (stackage->guix-package
+            stackage-recursive-import
             %stackage-updater))
 
 \f
@@ -45,9 +48,9 @@
     (_ #f)))
 
 (define (lts-info-packages lts-info)
-  "Retruns the alist of packages contained in LTS-INFO."
+  "Returns the alist of packages contained in LTS-INFO."
   (match lts-info
-    ((_ ("packages" pkg ...)) pkg)
+    ((("packages" pkg ...) . _) pkg)
     (_ '())))
 
 (define (leave-with-message fmt . args)
@@ -85,25 +88,33 @@
 (define (hackage-name-version name version)
   (and version (string-append  name "@" version)))
 
-(define* (stackage->guix-package package-name ; upstream name
-                                 #:key
-                                 (include-test-dependencies? #t)
-                                 (lts-version "")
-                                 (packages-info
-                                  (lts-info-packages
-                                   (stackage-lts-info-fetch lts-version))))
-  "Fetch Cabal file for PACKAGE-NAME from hackage.haskell.org.  The retrieved
+(define stackage->guix-package
+  (memoize
+   (lambda* (package-name ; upstream name
+             #:key
+             (include-test-dependencies? #t)
+             (lts-version "")
+             (packages-info
+              (lts-info-packages
+               (stackage-lts-info-fetch lts-version))))
+     "Fetch Cabal file for PACKAGE-NAME from hackage.haskell.org.  The retrieved
 vesion corresponds to the version of PACKAGE-NAME specified in the LTS-VERSION
 release at stackage.org.  Return the `package' S-expression corresponding to
 that package, or #f on failure.  PACKAGES-INFO is the alist with the packages
 included in the Stackage LTS release."
-  (let* ((version (lts-package-version packages-info package-name))
-         (name-version (hackage-name-version package-name version)))
-    (if name-version
-        (hackage->guix-package name-version
-                               #:include-test-dependencies?
-                               include-test-dependencies?)
-        (leave-with-message "~a: Stackage package not found" package-name))))
+     (let* ((version (lts-package-version packages-info package-name))
+            (name-version (hackage-name-version package-name version)))
+       (if name-version
+           (hackage->guix-package name-version
+                                  #:include-test-dependencies?
+                                  include-test-dependencies?)
+           (leave-with-message "~a: Stackage package not found" package-name))))))
+
+(define (stackage-recursive-import package-name . args)
+  (recursive-import package-name #f
+                    #:repo->guix-package (lambda (name repo)
+                                           (apply stackage->guix-package (cons name args)))
+                    #:guix-name hackage-name->package-name))
 
 \f
 ;;;
diff --git a/guix/scripts/import/stackage.scm b/guix/scripts/import/stackage.scm
index e6676e93e..b4b12581b 100644
--- a/guix/scripts/import/stackage.scm
+++ b/guix/scripts/import/stackage.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017 Federico Beffa <beffa@fbengineering.ch>
+;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -26,6 +27,7 @@
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-41)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
   #:export (guix-import-stackage))
@@ -43,11 +45,13 @@
   (display (G_ "Usage: guix import stackage PACKAGE-NAME
 Import and convert the LTS Stackage package for PACKAGE-NAME.\n"))
   (display (G_ "
-  -r VERSION, --lts-version=VERSION
+  -l VERSION, --lts-version=VERSION
                                specify the LTS version to use"))
   (display (G_ "
   -h, --help                   display this help and exit"))
   (display (G_ "
+  -r, --recursive              import packages recursively"))
+  (display (G_ "
   -t, --no-test-dependencies   don't include test-only dependencies"))
   (display (G_ "
   -V, --version                display version information and exit"))
@@ -68,11 +72,14 @@ Import and convert the LTS Stackage package for PACKAGE-NAME.\n"))
                    (alist-cons 'include-test-dependencies? #f
                                (alist-delete 'include-test-dependencies?
                                              result))))
-         (option '(#\r "lts-version") #t #f
+         (option '(#\l "lts-version") #t #f
                  (lambda (opt name arg result)
                    (alist-cons 'lts-version arg
                                (alist-delete 'lts-version
                                              result))))
+         (option '(#\r "recursive") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'recursive #t result)))
          %standard-import-options))
 
 \f
@@ -90,6 +97,27 @@ Import and convert the LTS Stackage package for PACKAGE-NAME.\n"))
                   (alist-cons 'argument arg result))
                 %default-options))
 
+  (define (run-importer package-name opts error-fn)
+    (let* ((arguments (list
+                       package-name
+                       #:include-test-dependencies?
+                       (assoc-ref opts 'include-test-dependencies?)
+                       #:lts-version (assoc-ref opts 'lts-version)))
+           (sexp (if (assoc-ref opts 'recursive)
+                     ;; Recursive import
+                     (map (match-lambda
+                            ((and ('package ('name name) . rest) pkg)
+                             `(define-public ,(string->symbol name)
+                                ,pkg))
+                            (_ #f))
+                          (reverse
+                           (stream->list
+                            (apply stackage-recursive-import arguments))))
+                     ;; Single import
+                     (apply stackage->guix-package arguments))))
+      (unless sexp (error-fn))
+      sexp))
+
   (let* ((opts (parse-options))
          (args (filter-map (match-lambda
                             (('argument . value)
@@ -99,15 +127,11 @@ Import and convert the LTS Stackage package for PACKAGE-NAME.\n"))
     (match args
       ((package-name)
        (with-error-handling
-        (let ((sexp (stackage->guix-package
-                     package-name
-                     #:include-test-dependencies?
-                     (assoc-ref opts 'include-test-dependencies?)
-                     #:lts-version (assoc-ref opts 'lts-version))))
-          (unless sexp
-            (leave (G_ "failed to download cabal file for package '~a'~%")
-                   package-name))
-          sexp)))
+         (run-importer package-name opts
+                       (lambda ()
+                         (leave (G_ "failed to download cabal file \
+for package '~a'~%")
+                                package-name)))))
       (()
        (leave (G_ "too few arguments~%")))
       ((many ...)
-- 
2.18.0

[bug#32483] [PATCH 1/4] gnu: Add python-llvmlite.

[Ricardo Wurmus]

* gnu/packages/llvm.scm (python-llvmlite): New variable.
---
 gnu/packages/llvm.scm | 53 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 52 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index 98592ad09..e9b7065ab 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -3,7 +3,7 @@
 ;;; Copyright © 2015 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2015, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2016 Dennis Mungai <dmngaie@gmail.com>
-;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2016, 2018 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2017 Roel Janssen <roel@gnu.org>
 ;;; Copyright © 2018 Marius Bakke <mbakke@fastmail.com>
 ;;;
@@ -29,6 +29,7 @@
   #:use-module (guix utils)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system cmake)
+  #:use-module (guix build-system python)
   #:use-module (gnu packages)
   #:use-module (gnu packages gcc)
   #:use-module (gnu packages bootstrap)           ;glibc-dynamic-linker
@@ -379,3 +380,53 @@ code analysis tools.")
        (patches (list (search-patch "llvm-for-extempore.patch")))))
     ;; Extempore refuses to build on architectures other than x86_64
     (supported-systems '("x86_64-linux"))))
+
+(define-public python-llvmlite
+  (package
+    (name "python-llvmlite")
+    (version "0.24.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "llvmlite" version))
+       (sha256
+        (base32
+         "01zwjlc3c5mhrwmv4b73zgbskwqps9ly0nrh54bbj1f1l72f839j"))))
+    (build-system python-build-system)
+    (inputs
+     `(("llvm"
+        ,(package
+           (inherit llvm)
+           (source (origin
+                     (inherit (package-source llvm))
+                     (patches
+                      (list
+                       (origin
+                         (method url-fetch)
+                         (uri (string-append "https://raw.githubusercontent.com/numba/"
+                                             "llvmlite/v" version "/conda-recipes/"
+                                             "D47188-svml.patch"))
+                         (sha256
+                          (base32
+                           "0mrj24jvkv3hjcmyg98zmvmyl1znlh2j63rdr69f6g7s96d2pfv1")))
+                       (origin
+                         (method url-fetch)
+                         (uri (string-append "https://raw.githubusercontent.com/numba/"
+                                             "llvmlite/v" version "/conda-recipes/"
+                                             "twine_cfg_undefined_behavior.patch"))
+                         (sha256
+                          (base32
+                           "07h71n2m1mn9zcfgw04zglffknplb233zqbcd6pckq0wygkrxflp")))
+                       (origin
+                         (method url-fetch)
+                         (uri (string-append "https://raw.githubusercontent.com/numba/"
+                                             "llvmlite/v" version "/conda-recipes/"
+                                             "0001-Transforms-Add-missing-header-for-InstructionCombini.patch"))
+                         (sha256
+                          (base32
+                           "1pp0z9696l6j4dwz7ypjrm4vvkj0d3mlf1g8zmiyk08akw5lz0cb")))))))))))
+    (home-page "http://llvmlite.pydata.org")
+    (synopsis "Wrapper around basic LLVM functionality")
+    (description
+     "This package provides a Python binding to LLVM for use in Numba.")
+    (license license:bsd-3)))
-- 
2.18.0

[bug#32483] [PATCH 2/4] gnu: Add python-numba.

[Ricardo Wurmus]

* gnu/packages/python.scm (python-numba): New variable.
---
 gnu/packages/python.scm | 84 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 3db29e23f..7f343bdab 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -99,6 +99,7 @@
   #:use-module (gnu packages libevent)
   #:use-module (gnu packages libffi)
   #:use-module (gnu packages linux)
+  #:use-module (gnu packages llvm)
   #:use-module (gnu packages machine-learning)
   #:use-module (gnu packages man)
   #:use-module (gnu packages maths)
@@ -13889,3 +13890,86 @@ Let's Encrypt.")
 development that supports command line argument parsing, command string
 validation testing and application logic.")
     (license license:expat)))
+
+;; Make sure to upgrade python-llvmlite in (gnu packages llvm) together with
+;; python-numba.  They have a very unflexible relationship.
+(define-public python-numba
+  (package
+    (name "python-numba")
+    (version "0.39.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "numba" version))
+       (sha256
+        (base32
+         "1bibvkwga1v8293i9ivl469d8bzgabn3vgr2ig7c1i68v8frsx07"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:modules ((guix build utils)
+                  (guix build python-build-system)
+                  (ice-9 ftw)
+                  (srfi srfi-1)
+                  (srfi srfi-26))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'disable-proprietary-features
+           (lambda _
+             (setenv "NUMBA_DISABLE_HSA" "1")
+             (setenv "NUMBA_DISABLE_CUDA" "1")
+             #t))
+         (add-after 'unpack 'remove-failing-tests
+           (lambda _
+             ;; FIXME: these tests fail for unknown reasons:
+             ;; test_non_writable_pycache, test_non_creatable_pycache, and
+             ;; test_frozen (all in numba.tests.test_dispatcher.TestCache).
+             (substitute* "numba/tests/test_dispatcher.py"
+               (("def test(_non_writable_pycache)" _ m)
+                (string-append "def guix_skip" m))
+               (("def test(_non_creatable_pycache)" _ m)
+                (string-append "def guix_skip" m))
+               (("def test(_frozen)" _ m)
+                (string-append "def guix_skip" m)))
+
+             ;; These tests fail because we don't run the tests from the build
+             ;; directory: test_setup_py_distutils, test_setup_py_setuptools
+             ;; They ar in numba.tests.test_pycc.TestDistutilsSupport.
+             (substitute* "numba/tests/test_pycc.py"
+               (("def test(_setup_py_distutils|_setup_py_setuptools)" _ m)
+                (string-append "def guix_skip" m)))
+             #t))
+         (replace 'check
+           (lambda _
+             (let ((cwd (getcwd)))
+               (setenv "PYTHONPATH"
+                       (string-append cwd "/build/"
+                                      (find (cut string-prefix? "lib" <>)
+                                            (scandir (string-append cwd "/build")))
+                                      ":"
+                                      (getenv "PYTHONPATH")))
+               ;; Something is wrong with the PYTHONPATH when running the
+               ;; tests from the build directory, as it complains about not being
+               ;; able to import certain modules.
+               (with-directory-excursion "/tmp"
+                 (invoke "python3" "-m" "numba.runtests" "-v" "-m")))
+             #t)))))
+    (propagated-inputs
+     `(("python-llvmlite" ,python-llvmlite)
+       ("python-numpy" ,python-numpy)
+       ("python-singledispatch" ,python-singledispatch)))
+    ;; Needed for tests.
+    (inputs
+     `(("python-jinja2" ,python-jinja2)
+       ("python-pygments" ,python-pygments)))
+    (home-page "https://numba.pydata.org")
+    (synopsis "Compile Python code using LLVM")
+    (description "Numba gives you the power to speed up your applications with
+high performance functions written directly in Python.  With a few
+annotations, array-oriented and math-heavy Python code can be just-in-time
+compiled to native machine instructions, similar in performance to C, C++ and
+Fortran, without having to switch languages or Python interpreters.
+
+Numba works by generating optimized machine code using the LLVM compiler
+infrastructure at import time, runtime, or statically (using the included pycc
+tool).")
+    (license license:bsd-3)))
-- 
2.18.0

[bug#32483] [PATCH 3/4] gnu: Add python-anndata.

[Ricardo Wurmus]

* gnu/packages/python.scm (python-anndata): New variable.
---
 gnu/packages/python.scm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 7f343bdab..2838d2ae5 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -13973,3 +13973,29 @@ Numba works by generating optimized machine code using the LLVM compiler
 infrastructure at import time, runtime, or statically (using the included pycc
 tool).")
     (license license:bsd-3)))
+
+(define-public python-anndata
+  (package
+    (name "python-anndata")
+    (version "0.6.9")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "anndata" version))
+       (sha256
+        (base32
+         "1fh461xyyc7pcrjfgd013bdc2alf53r46ss3gfw3431mbb1gappi"))))
+    (build-system python-build-system)
+    (propagated-inputs
+     `(("python-h5py" ,python-h5py)
+       ("python-natsort" ,python-natsort)
+       ("python-pandas" ,python-pandas)
+       ("python-scipy" ,python-scipy)))
+    (home-page "https://github.com/theislab/anndata")
+    (synopsis "Annotated data for data analysis pipelines")
+    (description "Anndata is a package for simple (functional) high-level APIs
+for data analysis pipelines.  In this context, it provides an efficient,
+scalable way of keeping track of data together with learned annotations and
+reduces the code overhead typically encountered when using a mostly
+object-oriented library such as @code{scikit-learn}.")
+    (license license:bsd-3)))
-- 
2.18.0

[bug#32483] [PATCH 4/4] gnu: Add python-scanpy.

[Ricardo Wurmus]

* gnu/packages/bioinformatics.scm (python-scanpy): New variable.
---
 gnu/packages/bioinformatics.scm | 36 +++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index dce8ab62f..ef9d81435 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -13406,3 +13406,39 @@ conversions, region filtering, FASTA sequence extraction and more.")
       ;; gffread is under Expat, but gclib is under Artistic 2.0
       (license (list license:expat
                      license:artistic2.0)))))
+
+(define-public python-scanpy
+  (package
+    (name "python-scanpy")
+    (version "1.2.2")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "scanpy" version))
+       (sha256
+        (base32
+         "1ak7bxms5a0yvf65prppq2g38clkv7c7jnjbnfpkh3xxv7q512jz"))))
+    (build-system python-build-system)
+    (propagated-inputs
+     `(("python-anndata" ,python-anndata)
+       ("python-igraph" ,python-igraph)
+       ("python-numba" ,python-numba)
+       ("python-joblib" ,python-joblib)
+       ("python-natsort" ,python-natsort)
+       ("python-networkx" ,python-networkx)
+       ("python-statsmodels" ,python-statsmodels)
+       ("python-scikit-learn" ,python-scikit-learn)
+       ("python-matplotlib" ,python-matplotlib)
+       ("python-pandas" ,python-pandas)
+       ("python-scipy" ,python-scipy)
+       ("python-seaborn" ,python-seaborn)
+       ("python-h5py" ,python-h5py)
+       ("python-tables" ,python-tables)))
+    (home-page "http://github.com/theislab/scanpy")
+    (synopsis "Single-Cell Analysis in Python.")
+    (description "Scanpy is a scalable toolkit for analyzing single-cell gene
+expression data.  It includes preprocessing, visualization, clustering,
+pseudotime and trajectory inference and differential expression testing.  The
+Python-based implementation efficiently deals with datasets of more than one
+million cells.")
+    (license license:bsd-3)))
-- 
2.18.0

[bug#32483] [PATCH] import: stackage: Support recursive importing.

[Ricardo Wurmus]

My apologies for sending a bunch of unrelated patches here!  They were
supposed to go to a separate bug instead.

-- 
Ricardo

[bug#32483] [PATCH] import: stackage: Support recursive importing.

[Ludovic Courtès]

Hello!

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> * guix/import/hackage.scm (hackage-name->package-name): Export procedure.
> * guix/import/stackage.scm (lts-info-packages-lts-info): Fix match expression.
> (stackage-recursive-import): New procedure.
> (stackage->guix-package): Memoize results.
> * guix/scripts/import/stackage.scm (show-help, %options,
> guix-import-stackage): Support recursive importing.
> * doc/guix.texi (Invoking guix import): Document option.

Could you add a test for recursive imports, similar to what Oleg did in
tests/gem.scm?  Or perhaps that’d be redundant since it does little more
than call ‘recursive-import’ from (guix import utils)?  (And I realize
tests/elpa.scm doesn’t have recursive import tests.)

Anyway, apart from that it LGTM.  Thanks!

Ludo’.

bug#32483: [PATCH] import: stackage: Support recursive importing.

[Ricardo Wurmus]

Thanks for the review.

I chose not to add a test for the recursive option because that
behaviour is implemented independent of the stackage importer.

Pushed to “master” branch with commit
a3ece51a29241c7060323cbbfc602c83200ffe4a.

--
Ricardo