[PATCH] import: pypi: add updater

[PATCH] import: pypi: add updater

[Cyril Roelandt]

* guix/import/pypi.scm (guix-package->pypi-name,
  latest-release): New procedures.
  (pypi-updater): New variable.
* guix/scripts/refresh.scm (%updaters): Add %PYPI-UPDATER.
---
 guix/import/pypi.scm     | 50 +++++++++++++++++++++++++++++++++++++++++++++++-
 guix/scripts/refresh.scm |  4 +++-
 2 files changed, 52 insertions(+), 2 deletions(-)

diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index 647ef61..f1988a7 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -30,12 +30,16 @@
   #:use-module (guix ui)
   #:use-module (guix utils)
   #:use-module (guix import utils)
+  #:use-module ((guix download) #:prefix download:)
   #:use-module (guix import json)
   #:use-module (guix packages)
+  #:use-module (guix upstream)
   #:use-module (guix licenses)
   #:use-module (guix build-system python)
+  #:use-module (gnu packages)
   #:use-module (gnu packages python)
-  #:export (pypi->guix-package))
+  #:export (pypi->guix-package
+            %pypi-updater))
 
 (define (pypi-fetch name)
   "Return an alist representation of the PyPI metadata for the package NAME,
@@ -60,6 +64,16 @@ package."
       (snake-case name)
       (string-append "python-" (snake-case name))))
 
+(define (guix-package->pypi-name package)
+  "Given a Python PACKAGE built from pypi.python.org, return the name of the
+package on PyPI."
+  (let ((source-url (and=> (package-source package) origin-uri)))
+    ;; The URL has the form:
+    ;; 'https://pypi.python.org/packages/source/' +
+    ;; first letter of the package name +
+    ;; '/' + package name + '/' + ...
+    (substring source-url 42 (string-rindex source-url #\/))))
+
 (define (maybe-inputs package-inputs)
   "Given a list of PACKAGE-INPUTS, tries to generate the 'inputs' field of a
 package definition."
@@ -190,3 +204,37 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                (license (string->license (assoc-ref* package "info" "license"))))
            (make-pypi-sexp name version release home-page synopsis
                            description license)))))
+
+(define (pypi-package? package)
+  "Return true if PACKAGE is a Python package from PyPI."
+
+  (define (pypi-url? url)
+    (string-prefix? "https://pypi.python.org/" url))
+
+  (let ((source-url (and=> (package-source package) origin-uri))
+        (fetch-method (and=> (package-source package) origin-method)))
+    (and (eq? fetch-method download:url-fetch)
+         (match source-url
+           ((? string?)
+            (pypi-url? source-url))
+           ((source-url ...)
+            (any pypi-url? source-url))))))
+
+(define (latest-release guix-package)
+  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
+  (let* ((pypi-name (guix-package->pypi-name
+                     (specification->package guix-package)))
+         (metadata (pypi-fetch pypi-name))
+         (version (assoc-ref* metadata "info" "version"))
+         (url (assoc-ref (latest-source-release metadata) "url")))
+    (upstream-source
+     (package guix-package)
+     (version version)
+     (urls (list url)))))
+
+(define %pypi-updater
+  (upstream-updater
+   (name 'pypi)
+   (description "Updater for PyPI packages")
+   (pred pypi-package?)
+   (latest latest-release)))
diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index 0df4121..3984a0b 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -30,6 +30,7 @@
   #:use-module ((guix gnu-maintenance) #:select (%gnu-updater))
   #:use-module (guix import elpa)
   #:use-module (guix import cran)
+  #:use-module (guix import pypi)
   #:use-module (guix gnupg)
   #:use-module (gnu packages)
   #:use-module ((gnu packages commencement) #:select (%final-inputs))
@@ -152,7 +153,8 @@ specified with `--select'.\n"))
   ;; List of "updaters" used by default.  They are consulted in this order.
   (list %gnu-updater
         %elpa-updater
-        %cran-updater))
+        %cran-updater
+        %pypi-updater))
 
 (define (lookup-updater name)
   "Return the updater called NAME."
-- 
2.6.2

Re: [PATCH] import: pypi: add updater

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> * guix/import/pypi.scm (guix-package->pypi-name,
>   latest-release): New procedures.
>   (pypi-updater): New variable.
    ^
Missing ‘%’.

> * guix/scripts/refresh.scm (%updaters): Add %PYPI-UPDATER.

Could you add it to the list that appears under ‘--type’ in guix.texi?

OK to push with this change.

Thank you!

Ludo’.

Re: [PATCH] import: pypi: add updater

[Mathieu Lirzin]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

Hi,

Cyril Roelandt <tipecaml@gmail.com> writes:

> * guix/import/pypi.scm (guix-package->pypi-name,
>   latest-release): New procedures.
>   (pypi-updater): New variable.
> * guix/scripts/refresh.scm (%updaters): Add %PYPI-UPDATER.
> ---
>  guix/import/pypi.scm     | 50 +++++++++++++++++++++++++++++++++++++++++++++++-
>  guix/scripts/refresh.scm |  4 +++-
>  2 files changed, 52 insertions(+), 2 deletions(-)
>

IIUC the commit bab020d7ca50e4153cf24832d119389a37fa8f63 has made
Guile-JSON non optional.

I've succeed a clean build with the previous commit (gnu: Add
python-tempest-lib.)  but then when I have done:

  git reset bab020d
  make -j4


[-- Attachment #2: make.log --]
[-- Type: text/plain, Size: 3169 bytes --]

mthl@godel:~/src/gnu/guix$ make -j4
make  all-recursive
make[1]: Entering directory '/home/mthl/src/gnu/guix'
Making all in po/guix
make[2]: Entering directory '/home/mthl/src/gnu/guix/po/guix'
make[2]: Leaving directory '/home/mthl/src/gnu/guix/po/guix'
Making all in po/packages
make[2]: Entering directory '/home/mthl/src/gnu/guix/po/packages'
make[2]: Leaving directory '/home/mthl/src/gnu/guix/po/packages'
make[2]: Entering directory '/home/mthl/src/gnu/guix'
LANGUAGE= ./pre-inst-env	\
  /bin/bash /home/mthl/src/gnu/guix/build-aux/missing help2man --output="doc/guix.1" guix
LANGUAGE= ./pre-inst-env /bin/bash /home/mthl/src/gnu/guix/build-aux/missing help2man --output="doc/guix-refresh.1" "guix refresh"
  GUILEC guix/scripts/refresh.go
  MAKEINFO doc/guix.info
Backtrace:
In ice-9/psyntax.scm:
 279: 19 [scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]
In ice-9/eval.scm:
 411: 18 [eval # ()]
In ice-9/boot-9.scm:
2951: 17 [define-module* (guix scripts refresh) #:filename ...]
2926: 16 [resolve-imports (((guix ui)) ((guix hash)) ((guix scripts)) ...)]
2864: 15 [resolve-interface (guix import pypi) #:select ...]
2789: 14 [#<procedure b3c360 at ice-9/boot-9.scm:2777:4 (name #:optional autoload version #:key ensure)> # ...]
3065: 13 [try-module-autoload (guix import pypi) #f]
2401: 12 [save-module-excursion #<procedure 1a57bd0 at ice-9/boot-9.scm:3066:17 ()>]
3085: 11 [#<procedure 1a57bd0 at ice-9/boot-9.scm:3066:17 ()>]
In unknown file:
   ?: 10 [primitive-load-path "guix/import/pypi" ...]
In ice-9/eval.scm:
 505: 9 [#<procedure a894a0 at ice-9/eval.scm:499:4 (exp)> (define-module # # ...)]
In ice-9/psyntax.scm:
1106: 8 [expand-top-sequence ((define-module # # # ...)) () ((top)) ...]
 989: 7 [scan ((define-module (guix import pypi) #:use-module ...)) () ...]
 279: 6 help2man: impossible de récupérer l'information « --help » de guix refresh
Try `--no-discard-stderr' if option outputs to stderr
Makefile:4590: recipe for target 'doc/guix-refresh.1' failed
make[2]: [doc/guix-refresh.1] Error 1 (ignorée)
[scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]
In ice-9/eval.scm:
 411: 5 [eval # ()]
In ice-9/boot-9.scm:
2951: 4 [define-module* (guix import pypi) #:filename ...]
2926: 3 [resolve-imports ((#) (#) (#) (#) ...)]
2867: 2 [resolve-interface (json) #:select ...]
In unknown file:
   ?: 1 [scm-error misc-error #f "~A ~S" ("no code for module" (json)) #f]
In ice-9/boot-9.scm:
 106: 0 [#<procedure e3d9c0 at ice-9/boot-9.scm:97:6 (thrown-k . args)> misc-error ...]

ice-9/boot-9.scm:106:20: In procedure #<procedure e3d9c0 at ice-9/boot-9.scm:97:6 (thrown-k . args)>:
ice-9/boot-9.scm:106:20: no code for module (json)
Makefile:4526: recipe for target 'guix/scripts/refresh.go' failed
make[2]: *** [guix/scripts/refresh.go] Error 1
make[2]: *** Attente des tâches non terminées....
make[2]: Leaving directory '/home/mthl/src/gnu/guix'
Makefile:3688: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/mthl/src/gnu/guix'
Makefile:2302: recipe for target 'all' failed
make: *** [all] Error 2

[-- Attachment #3: Type: text/plain, Size: 50 bytes --]


What should we do about this?

--
Mathieu Lirzin

Re: [PATCH] import: pypi: add updater

[Cyril Roelandt]

On 11/05/2015 08:58 PM, Mathieu Lirzin wrote:
> 
> IIUC the commit bab020d7ca50e4153cf24832d119389a37fa8f63 has made
> Guile-JSON non optional.
> 

That is weird, because the PyPI importer has always been using the
guile-json module. I thought it was already mandatory.

Cyril.

Re: [PATCH] import: pypi: add updater

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 643 bytes --]

Mathieu Lirzin <mthl@gnu.org> skribis:

> Cyril Roelandt <tipecaml@gmail.com> writes:
>
>> * guix/import/pypi.scm (guix-package->pypi-name,
>>   latest-release): New procedures.
>>   (pypi-updater): New variable.
>> * guix/scripts/refresh.scm (%updaters): Add %PYPI-UPDATER.
>> ---
>>  guix/import/pypi.scm     | 50 +++++++++++++++++++++++++++++++++++++++++++++++-
>>  guix/scripts/refresh.scm |  4 +++-
>>  2 files changed, 52 insertions(+), 2 deletions(-)
>>
>
> IIUC the commit bab020d7ca50e4153cf24832d119389a37fa8f63 has made
> Guile-JSON non optional.

Yes, a mistake.

> What should we do about this?

Something like this should work:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1841 bytes --]

diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index 3984a0b..4e9d1c8 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -30,7 +30,6 @@
   #:use-module ((guix gnu-maintenance) #:select (%gnu-updater))
   #:use-module (guix import elpa)
   #:use-module (guix import cran)
-  #:use-module (guix import pypi)
   #:use-module (guix gnupg)
   #:use-module (gnu packages)
   #:use-module ((gnu packages commencement) #:select (%final-inputs))
@@ -149,12 +148,35 @@ specified with `--select'.\n"))
 ;;; Updates.
 ;;;
 
+(define-syntax maybe-updater
+  (lambda (s)
+    (syntax-case s (=>)
+      ((_ ((module => updater) rest ...) (result ...))
+       (let ((met? (false-if-exception
+                    (resolve-interface (syntax->datum #'module)))))
+         (if met?
+             #'(maybe-updater (rest ...)
+                              (result ... (@ module updater)))
+             #'(maybe-updater (rest ...) (result ...)))))
+      ((_ (updater rest ...) (result ...))
+       #'(maybe-updater (rest ...) (result ... updater)))
+      ((_ () result)
+       #'result))))
+
+(define-syntax-rule (list-updaters updaters ...)
+  "Expand to '(list UPDATERS ...)' but only the subset of UPDATERS that are
+either unconditional, or have their requirement met.
+
+This is a way to discard at macro expansion time updaters that depend on
+unavailable optional dependencies such as Guile-JSON."
+  (maybe-updater (updaters ...) (list)))
+
 (define %updaters
   ;; List of "updaters" used by default.  They are consulted in this order.
-  (list %gnu-updater
+  (list-updaters %gnu-updater
                  %elpa-updater
                  %cran-updater
-        %pypi-updater))
+                 ((guix import pypi) => %pypi-updater)))
 
 (define (lookup-updater name)
   "Return the updater called NAME."

[-- Attachment #3: Type: text/plain, Size: 141 bytes --]


Could you test in a JSON-less environment and report back?

Alternately, we could declare Guile-JSON to be mandatory.

TIA,
Ludo’.

Re: [PATCH] import: pypi: add updater

[Mathieu Lirzin]

ludo@gnu.org (Ludovic Courtès) writes:

> Mathieu Lirzin <mthl@gnu.org> skribis:
>> IIUC the commit bab020d7ca50e4153cf24832d119389a37fa8f63 has made
>> Guile-JSON non optional.
>
> Yes, a mistake.
>
>> What should we do about this?
>
> Something like this should work:
>
> diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
> index 3984a0b..4e9d1c8 100644
> --- a/guix/scripts/refresh.scm
> +++ b/guix/scripts/refresh.scm
[...]
> Could you test in a JSON-less environment and report back?

It compiles and ‘guix refresh’ seems to work fine. Thank you very much Ludo!

> Alternately, we could declare Guile-JSON to be mandatory.

Since it is not packaged in Debian IMO it is not a good idea to depend
on it.

--
Mathieu Lirzin

Re: [PATCH] import: pypi: add updater

[Ludovic Courtès]

Mathieu Lirzin <mthl@gnu.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mathieu Lirzin <mthl@gnu.org> skribis:
>>> IIUC the commit bab020d7ca50e4153cf24832d119389a37fa8f63 has made
>>> Guile-JSON non optional.
>>
>> Yes, a mistake.
>>
>>> What should we do about this?
>>
>> Something like this should work:
>>
>> diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
>> index 3984a0b..4e9d1c8 100644
>> --- a/guix/scripts/refresh.scm
>> +++ b/guix/scripts/refresh.scm
> [...]
>> Could you test in a JSON-less environment and report back?
>
> It compiles and ‘guix refresh’ seems to work fine. Thank you very much Ludo!

Committed, thanks!

Ludo’.