[bug#31399] [PATCH] import: elpa: Implement recursive import.

[bug#31399] [PATCH] import: elpa: Implement recursive import.

[Oleg Pykhalov]

[-- Attachment #1.1: Type: text/plain, Size: 245 bytes --]

Hello Guix,

I stole recursive importer from ‘cran’ and made it for ‘elpa’, the patch
is attached.  I don't like it and want to share the code with ‘cran’
importer, but I don't know how to do it without increasing complexity.


[-- Attachment #1.2: [PATCH] emacs: implement recursive-import. --]
[-- Type: text/x-patch, Size: 8220 bytes --]

From 5c5023c5f343d673e96517b822b5e884ff20f714 Mon Sep 17 00:00:00 2001
From: Oleg Pykhalov <go.wigust@gmail.com>
Date: Thu, 10 May 2018 10:07:49 +0300
Subject: [PATCH] import: elpa: Implement recursive import.

* guix/import/elpa.scm (guix-name, recursive-import): New procedures.
* guix/scripts/import/elpa.scm (%options): Add 'recursive-import'.
(guix-import-elpa): Add this.
(show-help): Document this.
* doc/guix.texi (Invoking guix import): Document this.
---
 doc/guix.texi                |  6 +++
 guix/import/elpa.scm         | 81 +++++++++++++++++++++++++++++++++++-
 guix/scripts/import/elpa.scm | 22 ++++++++--
 3 files changed, 104 insertions(+), 5 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 8b9f8721b..40d3a4c73 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6481,6 +6481,12 @@ signatures,, emacs, The GNU Emacs Manual}).
 @uref{http://melpa.org/packages, MELPA}, selected by the @code{melpa}
 identifier.
 @end itemize
+
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
 @end table
 
 @item crate
diff --git a/guix/import/elpa.scm b/guix/import/elpa.scm
index 43e9eb60c..78a69dcc0 100644
--- a/guix/import/elpa.scm
+++ b/guix/import/elpa.scm
@@ -26,6 +26,8 @@
   #:use-module (srfi srfi-9 gnu)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-41)
+  #:use-module (gnu packages)
   #:use-module ((guix download) #:select (download-to-store))
   #:use-module (guix import utils)
   #:use-module (guix http-client)
@@ -37,7 +39,8 @@
   #:use-module (guix packages)
   #:use-module ((guix utils) #:select (call-with-temporary-output-file))
   #:export (elpa->guix-package
-            %elpa-updater))
+            %elpa-updater
+            recursive-import))
 
 (define (elpa-dependencies->names deps)
   "Convert DEPS, a list of symbol/version pairs à la ELPA, to a list of
@@ -289,4 +292,80 @@ type '<elpa-package>'."
    (pred package-from-gnu.org?)
    (latest latest-release)))
 
+(define (guix-name name)
+  "Return a Guix package name for a given Emacs package name."
+  (string-append "emacs-" (string-map (match-lambda
+                                        (#\_ #\-)
+                                        (#\. #\-)
+                                        (chr (char-downcase chr)))
+                                      name)))
+
+(define* (recursive-import package-name #:optional (repo 'gnu))
+  "Generate a stream of package expressions for PACKAGE-NAME and all its
+dependencies."
+  (define (propagated-inputs package)
+    "Return a list of package names in propagated inputs from PACKAGE."
+    (and=> (match package
+             ((package fields ...) (assq 'propagated-inputs fields))
+             (#f #f))
+           (match-lambda
+               ((propagated-inputs (qp ((package-name package) ...)))
+                (map (cut string-drop <> (string-length "emacs-"))
+                     package-name))
+             (#f #f))))
+  (let* ((package (elpa->guix-package package-name repo))
+         (dependencies (propagated-inputs package)))
+    (if (not package)
+        stream-null
+
+        ;; Generate a lazy stream of package expressions for all unknown
+        ;; dependencies in the graph.
+        (let* ((make-state (lambda (queue done)
+                             (cons queue done)))
+               (next       (match-lambda
+                             (((next . rest) . done) next)))
+               (imported   (match-lambda
+                             ((queue . done) done)))
+               (done?      (match-lambda
+                             ((queue . done)
+                              (zero? (length queue)))))
+               (unknown?   (lambda* (dependency #:optional (done '()))
+                             (and (not (member dependency
+                                               done))
+                                  (null? (find-packages-by-name
+                                          (guix-name dependency))))))
+               (update     (lambda (state new-queue)
+                             (match state
+                               (((head . tail) . done)
+                                (make-state (lset-difference
+                                             equal?
+                                             (lset-union equal? new-queue tail)
+                                             done)
+                                            (cons head done)))))))
+          (stream-cons
+           package
+           (stream-unfold
+            ;; map: produce a stream element
+            (lambda (state)
+              (elpa->guix-package (next state) repo))
+
+            ;; predicate
+            (negate done?)
+
+            ;; generator: update the queue
+            (lambda (state)
+              (let* ((package (elpa->guix-package (next state) repo))
+                     (dependencies (propagated-inputs package)))
+                (if package
+                    (update state (filter (cut unknown? <>
+                                               (cons (next state)
+                                                     (imported state)))
+                                          dependencies))
+                    ;; TODO: Try the other archives before giving up
+                    (update state (imported state)))))
+
+            ;; initial state
+            (make-state (filter unknown? dependencies)
+                        (list package-name))))))))
+
 ;;; elpa.scm ends here
diff --git a/guix/scripts/import/elpa.scm b/guix/scripts/import/elpa.scm
index 34eb16485..c49c3ac9e 100644
--- a/guix/scripts/import/elpa.scm
+++ b/guix/scripts/import/elpa.scm
@@ -25,6 +25,7 @@
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-41)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
   #:export (guix-import-elpa))
@@ -45,6 +46,8 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n"))
   (display (G_ "
   -h, --help                     display this help and exit"))
   (display (G_ "
+  -r, --recursive                generate package expressions for all Emacs packages that are not yet in Guix"))
+  (display (G_ "
   -V, --version                  display version information and exit"))
   (newline)
   (show-bug-report-information))
@@ -62,6 +65,9 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n"))
                  (lambda (opt name arg result)
                    (alist-cons 'repo (string->symbol arg)
                                (alist-delete 'repo result))))
+         (option '(#\r "recursive") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'recursive #t result)))
          %standard-import-options))
 
 \f
@@ -87,10 +93,18 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n"))
                            (reverse opts))))
     (match args
       ((package-name)
-       (let ((sexp (elpa->guix-package package-name (assoc-ref opts 'repo))))
-         (unless sexp
-           (leave (G_ "failed to download package '~a'~%") package-name))
-         sexp))
+       (if (assoc-ref opts 'recursive)
+           (map (match-lambda
+                  ((and ('package ('name name) . rest) pkg)
+                   `(define-public ,(string->symbol name)
+                      ,pkg))
+                  (_ #f))
+                (reverse (stream->list (recursive-import package-name
+                                                         (or (assoc-ref opts 'repo) 'cran)))))
+           (let ((sexp (elpa->guix-package package-name (assoc-ref opts 'repo))))
+             (unless sexp
+               (leave (G_ "failed to download package '~a'~%") package-name))
+             sexp)))
       (()
        (leave (G_ "too few arguments~%")))
       ((many ...)
-- 
2.17.0


[-- Attachment #1.3: Type: text/plain, Size: 7 bytes --]


Oleg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#31399] [PATCH] import: elpa: Implement recursive import.

[Ludovic Courtès]

Hi Oleg,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> I stole recursive importer from ‘cran’ and made it for ‘elpa’, the patch
> is attached.  I don't like it and want to share the code with ‘cran’
> importer, but I don't know how to do it without increasing complexity.

Indeed.  Ricardo and I discussed this in the past and Ricardo started
implementing something:

  http://lists.gnu.org/archive/html/guix-devel/2016-08/msg00381.html

It would be great to resume work on this, especially since we have other
importers that would immediately benefit from it (CRAN, Go, Crates).

The solution I proposed back then was to have importers return a SRFI-41
lazy stream of package definitions.

Perhaps there are other ways to achieve this, like simply letting the
top-level importer code check whether a package input is missing and
recurse.

Thoughts?

Thanks,
Ludo’.

[bug#31399] [PATCH] import: elpa: Implement recursive import.

[Ricardo Wurmus]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Oleg Pykhalov <go.wigust@gmail.com> skribis:
>
>> I stole recursive importer from ‘cran’ and made it for ‘elpa’, the patch
>> is attached.  I don't like it and want to share the code with ‘cran’
>> importer, but I don't know how to do it without increasing complexity.
>
> Indeed.  Ricardo and I discussed this in the past and Ricardo started
> implementing something:
>
>   http://lists.gnu.org/archive/html/guix-devel/2016-08/msg00381.html
>
> It would be great to resume work on this, especially since we have other
> importers that would immediately benefit from it (CRAN, Go, Crates).
>
> The solution I proposed back then was to have importers return a SRFI-41
> lazy stream of package definitions.

The CRAN importer supports recursion and it is implemented with SRFI-41
lazy streams and AIUI this is what this patch does for the elpa
importer.

We should move “recursive-import” to “(guix import utils)” and let it
take the package generator procedure (e.g. “cran->guix-package”) as an
argument.  It looks like this patch copied “recursive-import” and
changed little more than “cran->guix-package” to “elpa->guix-package”,
so this seems like a simple change.

--
Ricardo

[bug#31399] [PATCH] import: elpa: Implement recursive import.

[Ludovic Courtès]

Hello,

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Oleg Pykhalov <go.wigust@gmail.com> skribis:
>>
>>> I stole recursive importer from ‘cran’ and made it for ‘elpa’, the patch
>>> is attached.  I don't like it and want to share the code with ‘cran’
>>> importer, but I don't know how to do it without increasing complexity.
>>
>> Indeed.  Ricardo and I discussed this in the past and Ricardo started
>> implementing something:
>>
>>   http://lists.gnu.org/archive/html/guix-devel/2016-08/msg00381.html
>>
>> It would be great to resume work on this, especially since we have other
>> importers that would immediately benefit from it (CRAN, Go, Crates).
>>
>> The solution I proposed back then was to have importers return a SRFI-41
>> lazy stream of package definitions.
>
> The CRAN importer supports recursion and it is implemented with SRFI-41
> lazy streams and AIUI this is what this patch does for the elpa
> importer.
>
> We should move “recursive-import” to “(guix import utils)” and let it
> take the package generator procedure (e.g. “cran->guix-package”) as an
> argument.  It looks like this patch copied “recursive-import” and
> changed little more than “cran->guix-package” to “elpa->guix-package”,
> so this seems like a simple change.

Sounds like a plan.

Oleg, could you factorize what’s common between the two importers in
(guix import utils) like Ricardo suggests?

Thanks,
Ludo’.

[bug#31399] [PATCH] import: elpa: Implement recursive import.

[Oleg Pykhalov]

[-- Attachment #1.1: Type: text/plain, Size: 208 bytes --]

Hello,

ludo@gnu.org (Ludovic Courtès) writes:

[…]

> Oleg, could you factorize what’s common between the two importers in
> (guix import utils) like Ricardo suggests?

OK, here is a patch:


[-- Attachment #1.2: import: utils: Add recursive-import. --]
[-- Type: text/x-patch, Size: 19451 bytes --]

From 80015053776fb8b3aad6ae730c0e32f655536d9e Mon Sep 17 00:00:00 2001
From: Oleg Pykhalov <go.wigust@gmail.com>
Date: Wed, 30 May 2018 19:08:50 +0300
Subject: [PATCH] import: utils: Add recursive-import.

* doc/guix.texi (Invoking guix import): Document elpa recursive import.
* guix/import/cran.scm (cran-guix-name, cran-recursive-import): New
procedures.
(recursive-import): Remove procedure.
* guix/import/elpa.scm (elpa-package->sexp): Return package and
dependencies values.
(elpa-guix-name, elpa-recursive-import): New procedures.
* guix/import/utils.scm (guix-name, recursive-import): New procedures.
* guix/scripts/import/cran.scm (guix-import-cran): Use
'cran-recursive-import' procedure.
* guix/scripts/import/elpa.scm (show-help, %options): Add recursive
option.
(guix-import-elpa): Use 'elpa-recursive-import'.
---
 doc/guix.texi                |  6 +++
 guix/import/cran.scm         | 77 +++++-------------------------------
 guix/import/elpa.scm         | 63 ++++++++++++++++++-----------
 guix/import/utils.scm        | 77 +++++++++++++++++++++++++++++++++++-
 guix/scripts/import/cran.scm |  6 ++-
 guix/scripts/import/elpa.scm | 26 ++++++++++--
 6 files changed, 156 insertions(+), 99 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 5129b998b..9dd5b31f1 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6530,6 +6530,12 @@ signatures,, emacs, The GNU Emacs Manual}).
 @uref{http://melpa.org/packages, MELPA}, selected by the @code{melpa}
 identifier.
 @end itemize
+
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
 @end table
 
 @item crate
diff --git a/guix/import/cran.scm b/guix/import/cran.scm
index ec2b7e602..b7d3392cd 100644
--- a/guix/import/cran.scm
+++ b/guix/import/cran.scm
@@ -43,7 +43,7 @@
   #:use-module (gnu packages)
   #:export (cran->guix-package
             bioconductor->guix-package
-            recursive-import
+            cran-recursive-import
             %cran-updater
             %bioconductor-updater
 
@@ -231,13 +231,7 @@ empty list when the FIELD cannot be found."
         "translations"
         "utils"))
 
-(define (guix-name name)
-  "Return a Guix package name for a given R package name."
-  (string-append "r-" (string-map (match-lambda
-                                    (#\_ #\-)
-                                    (#\. #\-)
-                                    (chr (char-downcase chr)))
-                                  name)))
+(define cran-guix-name (cut guix-name "r-" <>))
 
 (define (needs-fortran? tarball)
   "Check if the TARBALL contains Fortran source files."
@@ -318,7 +312,7 @@ from the alist META, which was derived from the R package's DESCRIPTION file."
                                                  (listify meta "Depends"))))))
     (values
      `(package
-        (name ,(guix-name name))
+        (name ,(cran-guix-name name))
         (version ,version)
         (source (origin
                   (method url-fetch)
@@ -327,12 +321,12 @@ from the alist META, which was derived from the R package's DESCRIPTION file."
                    (base32
                     ,(bytevector->nix-base32-string (file-sha256 tarball))))))
         ,@(if (not (equal? (string-append "r-" name)
-                           (guix-name name)))
+                           (cran-guix-name name)))
               `((properties ,`(,'quasiquote ((,'upstream-name . ,name)))))
               '())
         (build-system r-build-system)
         ,@(maybe-inputs sysdepends)
-        ,@(maybe-inputs (map guix-name propagate) 'propagated-inputs)
+        ,@(maybe-inputs (map cran-guix-name propagate) 'propagated-inputs)
         ,@(maybe-inputs
            `(,@(if (needs-fortran? tarball)
                    '("gfortran") '())
@@ -356,63 +350,10 @@ s-expression corresponding to that package, or #f on failure."
      (and=> (fetch-description repo package-name)
             (cut description->package repo <>)))))
 
-(define* (recursive-import package-name #:optional (repo 'cran))
-  "Generate a stream of package expressions for PACKAGE-NAME and all its
-dependencies."
-  (receive (package . dependencies)
-      (cran->guix-package package-name repo)
-    (if (not package)
-        stream-null
-
-        ;; Generate a lazy stream of package expressions for all unknown
-        ;; dependencies in the graph.
-        (let* ((make-state (lambda (queue done)
-                             (cons queue done)))
-               (next       (match-lambda
-                             (((next . rest) . done) next)))
-               (imported   (match-lambda
-                             ((queue . done) done)))
-               (done?      (match-lambda
-                             ((queue . done)
-                              (zero? (length queue)))))
-               (unknown?   (lambda* (dependency #:optional (done '()))
-                             (and (not (member dependency
-                                               done))
-                                  (null? (find-packages-by-name
-                                          (guix-name dependency))))))
-               (update     (lambda (state new-queue)
-                             (match state
-                               (((head . tail) . done)
-                                (make-state (lset-difference
-                                             equal?
-                                             (lset-union equal? new-queue tail)
-                                             done)
-                                            (cons head done)))))))
-          (stream-cons
-           package
-           (stream-unfold
-            ;; map: produce a stream element
-            (lambda (state)
-              (cran->guix-package (next state) repo))
-
-            ;; predicate
-            (negate done?)
-
-            ;; generator: update the queue
-            (lambda (state)
-              (receive (package . dependencies)
-                  (cran->guix-package (next state) repo)
-                (if package
-                    (update state (filter (cut unknown? <>
-                                               (cons (next state)
-                                                     (imported state)))
-                                          (car dependencies)))
-                    ;; TODO: Try the other archives before giving up
-                    (update state (imported state)))))
-
-            ;; initial state
-            (make-state (filter unknown? (car dependencies))
-                        (list package-name))))))))
+(define* (cran-recursive-import package-name #:optional (repo 'gnu))
+  (recursive-import package-name repo
+                    #:repo->guix-package cran->guix-package
+                    #:guix-name cran-guix-name))
 
 \f
 ;;;
diff --git a/guix/import/elpa.scm b/guix/import/elpa.scm
index 43e9eb60c..24021bf2d 100644
--- a/guix/import/elpa.scm
+++ b/guix/import/elpa.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch>
 ;;; Copyright © 2015, 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -26,6 +27,8 @@
   #:use-module (srfi srfi-9 gnu)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-41)
+  #:use-module (gnu packages)
   #:use-module ((guix download) #:select (download-to-store))
   #:use-module (guix import utils)
   #:use-module (guix http-client)
@@ -37,7 +40,8 @@
   #:use-module (guix packages)
   #:use-module ((guix utils) #:select (call-with-temporary-output-file))
   #:export (elpa->guix-package
-            %elpa-updater))
+            %elpa-updater
+            elpa-recursive-import))
 
 (define (elpa-dependencies->names deps)
   "Convert DEPS, a list of symbol/version pairs à la ELPA, to a list of
@@ -200,13 +204,15 @@ type '<elpa-package>'."
 
   (define source-url (elpa-package-source-url pkg))
 
+  (define dependencies-names
+    (filter-dependencies (elpa-dependencies->names
+                          (elpa-package-inputs pkg))))
+
   (define dependencies
-    (let* ((deps (elpa-package-inputs pkg))
-           (names (filter-dependencies (elpa-dependencies->names deps))))
-      (map (lambda (n)
-             (let ((new-n (elpa-name->package-name n)))
-               (list new-n (list 'unquote (string->symbol new-n)))))
-           names)))
+    (map (lambda (n)
+           (let ((new-n (elpa-name->package-name n)))
+             (list new-n (list 'unquote (string->symbol new-n)))))
+         dependencies-names))
 
   (define (maybe-inputs input-type inputs)
     (match inputs
@@ -218,23 +224,25 @@ type '<elpa-package>'."
 
   (let ((tarball (with-store store
                    (download-to-store store source-url))))
-    `(package
-       (name ,(elpa-name->package-name name))
-       (version ,version)
-       (source (origin
-                 (method url-fetch)
-                 (uri (string-append ,@(factorize-uri source-url version)))
-                 (sha256
-                  (base32
-                   ,(if tarball
-                        (bytevector->nix-base32-string (file-sha256 tarball))
-                        "failed to download package")))))
-       (build-system emacs-build-system)
-       ,@(maybe-inputs 'propagated-inputs dependencies)
-       (home-page ,(elpa-package-home-page pkg))
-       (synopsis ,(elpa-package-synopsis pkg))
-       (description ,(elpa-package-description pkg))
-       (license ,license))))
+    (values
+     `(package
+        (name ,(elpa-name->package-name name))
+        (version ,version)
+        (source (origin
+                  (method url-fetch)
+                  (uri (string-append ,@(factorize-uri source-url version)))
+                  (sha256
+                   (base32
+                    ,(if tarball
+                         (bytevector->nix-base32-string (file-sha256 tarball))
+                         "failed to download package")))))
+        (build-system emacs-build-system)
+        ,@(maybe-inputs 'propagated-inputs dependencies)
+        (home-page ,(elpa-package-home-page pkg))
+        (synopsis ,(elpa-package-synopsis pkg))
+        (description ,(elpa-package-description pkg))
+        (license ,license))
+     dependencies-names)))
 
 (define* (elpa->guix-package name #:optional (repo 'gnu))
   "Fetch the package NAME from REPO and produce a Guix package S-expression."
@@ -289,4 +297,11 @@ type '<elpa-package>'."
    (pred package-from-gnu.org?)
    (latest latest-release)))
 
+(define elpa-guix-name (cut guix-name "emacs-" <>))
+
+(define* (elpa-recursive-import package-name #:optional (repo 'gnu))
+  (recursive-import package-name repo
+                    #:repo->guix-package elpa->guix-package
+                    #:guix-name elpa-guix-name))
+
 ;;; elpa.scm ends here
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index efc616907..df85904c6 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org>
 ;;; Copyright © 2016 David Craven <david@craven.ch>
 ;;; Copyright © 2017 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -39,6 +40,8 @@
   #:use-module (ice-9 regex)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-41)
   #:export (factorize-uri
 
             hash-table->alist
@@ -61,7 +64,11 @@
             alist->package
 
             read-lines
-            chunk-lines))
+            chunk-lines
+
+            guix-name
+
+            recursive-import))
 
 (define (factorize-uri uri version)
   "Factorize URI, a package tarball URI as a string, such that any occurrences
@@ -357,3 +364,71 @@ separated by PRED."
         (if (null? after)
             (reverse res)
             (loop (cdr after) res))))))
+
+(define (guix-name prefix name)
+  "Return a Guix package name for a given package name."
+  (string-append prefix (string-map (match-lambda
+                                      (#\_ #\-)
+                                      (#\. #\-)
+                                      (chr (char-downcase chr)))
+                                    name)))
+
+(define* (recursive-import package-name repo
+                           #:key repo->guix-package guix-name
+                           #:allow-other-keys)
+  "Generate a stream of package expressions for PACKAGE-NAME and all its
+dependencies."
+  (receive (package . dependencies)
+      (repo->guix-package package-name repo)
+    (if (not package)
+        stream-null
+
+        ;; Generate a lazy stream of package expressions for all unknown
+        ;; dependencies in the graph.
+        (let* ((make-state (lambda (queue done)
+                             (cons queue done)))
+               (next       (match-lambda
+                             (((next . rest) . done) next)))
+               (imported   (match-lambda
+                             ((queue . done) done)))
+               (done?      (match-lambda
+                             ((queue . done)
+                              (zero? (length queue)))))
+               (unknown?   (lambda* (dependency #:optional (done '()))
+                             (and (not (member dependency
+                                               done))
+                                  (null? (find-packages-by-name
+                                          (guix-name dependency))))))
+               (update     (lambda (state new-queue)
+                             (match state
+                               (((head . tail) . done)
+                                (make-state (lset-difference
+                                             equal?
+                                             (lset-union equal? new-queue tail)
+                                             done)
+                                            (cons head done)))))))
+          (stream-cons
+           package
+           (stream-unfold
+            ;; map: produce a stream element
+            (lambda (state)
+              (repo->guix-package (next state) repo))
+
+            ;; predicate
+            (negate done?)
+
+            ;; generator: update the queue
+            (lambda (state)
+              (receive (package . dependencies)
+                  (repo->guix-package package-name repo)
+                (if package
+                    (update state (filter (cut unknown? <>
+                                               (cons (next state)
+                                                     (imported state)))
+                                          (car dependencies)))
+                    ;; TODO: Try the other archives before giving up
+                    (update state (imported state)))))
+
+            ;; initial state
+            (make-state (filter unknown? (car dependencies))
+                        (list package-name))))))))
diff --git a/guix/scripts/import/cran.scm b/guix/scripts/import/cran.scm
index d65c644c0..30ae6d434 100644
--- a/guix/scripts/import/cran.scm
+++ b/guix/scripts/import/cran.scm
@@ -99,8 +99,10 @@ Import and convert the CRAN package for PACKAGE-NAME.\n"))
                    `(define-public ,(string->symbol name)
                       ,pkg))
                   (_ #f))
-                (reverse (stream->list (recursive-import package-name
-                                                         (or (assoc-ref opts 'repo) 'cran)))))
+                (reverse
+                 (stream->list
+                  (cran-recursive-import package-name
+                                         (or (assoc-ref opts 'repo) 'cran)))))
            ;; Single import
            (let ((sexp (cran->guix-package package-name
                                            (or (assoc-ref opts 'repo) 'cran))))
diff --git a/guix/scripts/import/elpa.scm b/guix/scripts/import/elpa.scm
index 34eb16485..f1ed5016b 100644
--- a/guix/scripts/import/elpa.scm
+++ b/guix/scripts/import/elpa.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch>
+;;; Copyright © 2018 Oleg Pykhalov <go.wigust@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -21,10 +22,12 @@
   #:use-module (guix utils)
   #:use-module (guix scripts)
   #:use-module (guix import elpa)
+  #:use-module (guix import utils)
   #:use-module (guix scripts import)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-41)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
   #:export (guix-import-elpa))
@@ -45,6 +48,8 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n"))
   (display (G_ "
   -h, --help                     display this help and exit"))
   (display (G_ "
+  -r, --recursive                generate package expressions for all Emacs packages that are not yet in Guix"))
+  (display (G_ "
   -V, --version                  display version information and exit"))
   (newline)
   (show-bug-report-information))
@@ -62,6 +67,9 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n"))
                  (lambda (opt name arg result)
                    (alist-cons 'repo (string->symbol arg)
                                (alist-delete 'repo result))))
+         (option '(#\r "recursive") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'recursive #t result)))
          %standard-import-options))
 
 \f
@@ -87,10 +95,20 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n"))
                            (reverse opts))))
     (match args
       ((package-name)
-       (let ((sexp (elpa->guix-package package-name (assoc-ref opts 'repo))))
-         (unless sexp
-           (leave (G_ "failed to download package '~a'~%") package-name))
-         sexp))
+       (if (assoc-ref opts 'recursive)
+           (map (match-lambda
+                  ((and ('package ('name name) . rest) pkg)
+                   `(define-public ,(string->symbol name)
+                      ,pkg))
+                  (_ #f))
+                (reverse
+                 (stream->list
+                  (elpa-recursive-import package-name
+                                         (or (assoc-ref opts 'repo) 'gnu)))))
+           (let ((sexp (elpa->guix-package package-name (assoc-ref opts 'repo))))
+             (unless sexp
+               (leave (G_ "failed to download package '~a'~%") package-name))
+             sexp)))
       (()
        (leave (G_ "too few arguments~%")))
       ((many ...)
-- 
2.17.0


[-- Attachment #1.3: Type: text/plain, Size: 7 bytes --]


Oleg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#31399] [PATCH] import: elpa: Implement recursive import.

[Ludovic Courtès]

Hello Oleg!

Oleg Pykhalov <go.wigust@gmail.com> skribis:

>> Oleg, could you factorize what’s common between the two importers in
>> (guix import utils) like Ricardo suggests?
>
> OK, here is a patch:

That was fast.  :-)

> From 80015053776fb8b3aad6ae730c0e32f655536d9e Mon Sep 17 00:00:00 2001
> From: Oleg Pykhalov <go.wigust@gmail.com>
> Date: Wed, 30 May 2018 19:08:50 +0300
> Subject: [PATCH] import: utils: Add recursive-import.
>
> * doc/guix.texi (Invoking guix import): Document elpa recursive import.
> * guix/import/cran.scm (cran-guix-name, cran-recursive-import): New
> procedures.
> (recursive-import): Remove procedure.
> * guix/import/elpa.scm (elpa-package->sexp): Return package and
> dependencies values.
> (elpa-guix-name, elpa-recursive-import): New procedures.
> * guix/import/utils.scm (guix-name, recursive-import): New procedures.
> * guix/scripts/import/cran.scm (guix-import-cran): Use
> 'cran-recursive-import' procedure.
> * guix/scripts/import/elpa.scm (show-help, %options): Add recursive
> option.
> (guix-import-elpa): Use 'elpa-recursive-import'.

Nice!

I’d find it clearer to have a first patch that moves code from cran.scm
to utils.scm, and a second patch containing the ELPA changes.  No big
deal though.

Please double-check that tests/{elpa,cran}.scm still pass, but if they
do, I think you can go ahead and push.

Thank you!

Ludo’.

[bug#31399] [PATCH] import: elpa: Implement recursive import.

[Oleg Pykhalov]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

Hello Ludovic, Guix,

Apologies for a delay.

ludo@gnu.org (Ludovic Courtès) writes:

[…]

> I’d find it clearer to have a first patch that moves code from cran.scm
> to utils.scm, and a second patch containing the ELPA changes.  No big
> deal though.

OK, splitted the patch.  Also removed unused srfi-41.

> Please double-check that tests/{elpa,cran}.scm still pass, but if they
> do, I think you can go ahead and push.

I tested both commits separately.  Pushed as:

- 74032da3a2ef3e99e89dd58701414004f5a6c061
- ae9e5d6602544390fa5da0a87450405ebba012fd

While I've tested, ‘test-tmp/db/’ directory was missing in my Guix Git
repository (‘test-tmp’ directory was present), tests failed until I
manually created by invoking ‘mkdir test-tmp/db’.  I've tried to remove
‘test-tmp’, invoke ‘./configure --localstatedir=/var --prefix=’ and
‘make’, but it produced only ‘test-tmp’ directory.

[…]

For the record.

I did some manually tests by commenting package and dependency package
recipes in Guix package collection, then invoked import:

- ‘r-circlize’ which depends ‘r-shape’
--8<---------------cut here---------------start------------->8---
./pre-inst-env env GUIX_PACKAGE_PATH= guix import cran -r circlize
--8<---------------cut here---------------end--------------->8---

- ‘emacs-xelb’ which depends on ‘emacs-exwm’
--8<---------------cut here---------------start------------->8---
./pre-inst-env env GUIX_PACKAGE_PATH= guix import elpa -r exwm
--8<---------------cut here---------------end--------------->8---

- ‘emacs-ace-window’ which depends on ‘emacs-avy’
--8<---------------cut here---------------start------------->8---
./pre-inst-env env GUIX_PACKAGE_PATH= guix import elpa -a melpa -r ace-window
--8<---------------cut here---------------end--------------->8---

Thanks,
Oleg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#31399] [PATCH] import: elpa: Implement recursive import.

[Ricardo Wurmus]

Hi Oleg,

>> I’d find it clearer to have a first patch that moves code from cran.scm
>> to utils.scm, and a second patch containing the ELPA changes.  No big
>> deal though.
>
> OK, splitted the patch.  Also removed unused srfi-41.
>
>> Please double-check that tests/{elpa,cran}.scm still pass, but if they
>> do, I think you can go ahead and push.
>
> I tested both commits separately.  Pushed as:
>
> - 74032da3a2ef3e99e89dd58701414004f5a6c061
> - ae9e5d6602544390fa5da0a87450405ebba012fd

Thank you!

> While I've tested, ‘test-tmp/db/’ directory was missing in my Guix Git
> repository (‘test-tmp’ directory was present), tests failed until I
> manually created by invoking ‘mkdir test-tmp/db’.  I've tried to remove
> ‘test-tmp’, invoke ‘./configure --localstatedir=/var --prefix=’ and
> ‘make’, but it produced only ‘test-tmp’ directory.

This seems to be a problem that was introduced with commit
7f9d184d9b688d13ce76eefabaddcfa76bdde2b5.

--
Ricardo

[bug#31399] [PATCH] import: elpa: Implement recursive import.

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

>> While I've tested, ‘test-tmp/db/’ directory was missing in my Guix Git
>> repository (‘test-tmp’ directory was present), tests failed until I
>> manually created by invoking ‘mkdir test-tmp/db’.  I've tried to remove
>> ‘test-tmp’, invoke ‘./configure --localstatedir=/var --prefix=’ and
>> ‘make’, but it produced only ‘test-tmp’ directory.
>
> This seems to be a problem that was introduced with commit
> 7f9d184d9b688d13ce76eefabaddcfa76bdde2b5.

I’m happy to say it was fixed earlier today in
9953685c0985c6cc4481db3e351f1cc1dbf81e8b.  :-)

Thanks,
Ludo’.