[PATCH 1/3] import: hackage: Silence download output.

[PATCH 1/3] import: hackage: Silence download output.

[ericbavier]

From: Eric Bavier <bavier@member.fsf.org>

* guix/import/hackage.scm (hackage-fetch): Send error output from
  url-fetch to /dev/null.
---
 guix/import/hackage.scm | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/guix/import/hackage.scm b/guix/import/hackage.scm
index 8725ffa..9401830 100644
--- a/guix/import/hackage.scm
+++ b/guix/import/hackage.scm
@@ -82,9 +82,14 @@ version."
                                     name "-" version "/" name ".cabal")
                      (string-append "http://hackage.haskell.org/package/"
                                     name "/" name ".cabal"))))
+    ;; XXX: We want to silence the download progress report, which is
+    ;; especially annoying for 'guix refresh', but we have to use a file port.
     (call-with-temporary-output-file
      (lambda (temp port)
-       (and (url-fetch url temp)
+       (and (call-with-output-file "/dev/null"
+              (lambda (null)
+                (with-error-to-port null
+                  (lambda () (url-fetch url temp)))))
             (call-with-input-file temp
               (compose read-cabal canonical-newline-port)))))))
 
-- 
2.7.3

[PATCH 2/3] import: hackage: Factorize url synthesis.

[ericbavier]

From: Eric Bavier <bavier@member.fsf.org>

* guix/import/hackage.scm (hackage-source-url, hackage-cabal-url): New
  procedures.
  (hackage-fetch, hackage-module->sexp): Use them.
---
 guix/import/hackage.scm | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

diff --git a/guix/import/hackage.scm b/guix/import/hackage.scm
index 9401830..3e9df63 100644
--- a/guix/import/hackage.scm
+++ b/guix/import/hackage.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 Federico Beffa <beffa@fbengineering.ch>
+;;; Copyright © 2016 Eric Bavier <bavier@member.fsf.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -65,6 +66,22 @@
 
 (define package-name-prefix "ghc-")
 
+(define (hackage-source-url name version)
+  "Given a Hackage package NAME and VERSION, return a url to the source
+tarball."
+  (string-append "http://hackage.haskell.org/package/" name
+                 "/" name "-" version ".tar.gz"))
+
+(define* (hackage-cabal-url name #:optional version)
+  "Given a Hackage package NAME and VERSION, return a url to the corresponding
+.cabal file on Hackage.  If VERSION is #f or missing, the url for the latest
+version is returned."
+  (if version
+      (string-append "http://hackage.haskell.org/package/"
+                     name "-" version "/" name ".cabal")
+      (string-append "http://hackage.haskell.org/package/"
+                     name "/" name ".cabal")))
+
 (define (hackage-name->package-name name)
   "Given the NAME of a Cabal package, return the corresponding Guix name."
   (if (string-prefix? package-name-prefix name)
@@ -76,12 +93,7 @@
 the version part is omitted from the package name, then return the latest
 version."
   (let*-values (((name version) (package-name->name+version name-version))
-                ((url)
-                 (if version
-                     (string-append "http://hackage.haskell.org/package/"
-                                    name "-" version "/" name ".cabal")
-                     (string-append "http://hackage.haskell.org/package/"
-                                    name "/" name ".cabal"))))
+                ((url) (hackage-cabal-url name version)))
     ;; XXX: We want to silence the download progress report, which is
     ;; especially annoying for 'guix refresh', but we have to use a file port.
     (call-with-temporary-output-file
@@ -159,8 +171,7 @@ representation of a Cabal file as produced by 'read-cabal'."
     (cabal-package-version cabal))
   
   (define source-url
-    (string-append "http://hackage.haskell.org/package/" name
-                   "/" name "-" version ".tar.gz"))
+    (hackage-source-url name version))
 
   (define dependencies
     (let ((names
-- 
2.7.3

[PATCH 3/3] import: Add Hackage updater.

[ericbavier]

From: Eric Bavier <bavier@member.fsf.org>

* guix/import/hackage.scm (guix-package->hackage-name, hackage-package?)
  (latest-release): New procedures.
  (%hackage-updater): New variable.
* guix/scripts/refresh.scm (%updaters): Add it.
* doc/guix.texi (Invoking guix refresh): Mention it.
---
 doc/guix.texi            |  2 ++
 guix/import/hackage.scm  | 65 +++++++++++++++++++++++++++++++++++++++++++++---
 guix/scripts/refresh.scm |  2 ++
 3 files changed, 66 insertions(+), 3 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 186b850..57ddf60 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -4764,6 +4764,8 @@ the updater for @uref{https://pypi.python.org, PyPI} packages.
 the updater for @uref{https://rubygems.org, RubyGems} packages.
 @item github
 the updater for @uref{https://github.com, GitHub} packages.
+@item hackage
+the updater for @uref{https://hackage.haskell.org, Hackage} packages.
 @end table
 
 For instance, the following command only checks for updates of Emacs
diff --git a/guix/import/hackage.scm b/guix/import/hackage.scm
index 3e9df63..c67ce91 100644
--- a/guix/import/hackage.scm
+++ b/guix/import/hackage.scm
@@ -19,10 +19,13 @@
 
 (define-module (guix import hackage)
   #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
   #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-1)
-  #:use-module ((guix download) #:select (download-to-store))
+  #:use-module (gnu packages)
+  #:use-module ((guix download) #:select (download-to-store url-fetch)
+                                #:prefix download:)
   #:use-module ((guix utils) #:select (package-name->name+version
                                        canonical-newline-port))
   #:use-module (guix import utils)
@@ -30,8 +33,11 @@
   #:use-module (guix store)
   #:use-module (guix hash)
   #:use-module (guix base32)
+  #:use-module (guix upstream)
+  #:use-module (guix packages)
   #:use-module ((guix utils) #:select (call-with-temporary-output-file))
-  #:export (hackage->guix-package))
+  #:export (hackage->guix-package
+            %hackage-updater))
 
 (define ghc-standard-libraries
   ;; List of libraries distributed with ghc (7.10.2). We include GHC itself as
@@ -88,6 +94,17 @@ version is returned."
       (string-downcase name)
       (string-append package-name-prefix (string-downcase name))))
 
+(define guix-package->hackage-name
+  (let ((uri-rx (make-regexp "https?://hackage.haskell.org/package/([^/]+)/.*"))
+        (name-rx (make-regexp "(.*)-[0-9\\.]+")))
+    (lambda (package)
+      "Given a Guix package name, return the corresponding Hackage name."
+      (let* ((source-url (and=> (package-source package) origin-uri))
+             (name (match:substring (regexp-exec uri-rx source-url) 1)))
+        (match (regexp-exec name-rx name)
+          (#f name)
+          (m (match:substring m 1)))))))
+
 (define (hackage-fetch name-version)
   "Return the Cabal file for the package NAME-VERSION, or #f on failure.  If
 the version part is omitted from the package name, then return the latest
@@ -199,7 +216,7 @@ representation of a Cabal file as produced by 'read-cabal'."
         '()))
 
   (let ((tarball (with-store store
-                   (download-to-store store source-url))))
+                   (download:download-to-store store source-url))))
     `(package
        (name ,(hackage-name->package-name name))
        (version ,version)
@@ -241,4 +258,46 @@ respectively."
                                     include-test-dependencies?)
                                (cut eval-cabal <> cabal-environment)))))
 
+(define (hackage-package? package)
+  "Return #t if PACKAGE is a Haskell package from Hackage."
+
+  (define haskell-url?
+    (let ((hackage-rx (make-regexp "https?://hackage.haskell.org")))
+      (lambda (url)
+        (regexp-exec hackage-rx url))))
+
+  (let ((source-url (and=> (package-source package) origin-uri))
+        (fetch-method (and=> (package-source package) origin-method)))
+    (and (eq? fetch-method download:url-fetch)
+         (match source-url
+           ((? string?)
+            (haskell-url? source-url))
+           ((source-url ...)
+            (any haskell-url? source-url))))))
+
+(define (latest-release guix-package)
+  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
+  (let* ((hackage-name (guix-package->hackage-name
+                        (specification->package guix-package)))
+         (cabal-meta (hackage-fetch hackage-name)))
+    (match cabal-meta
+      (#f
+       (format (current-error-port)
+               "warning: failed to parse ~a~%"
+               (hackage-cabal-url hackage-name))
+       #f)
+      ((_ *** ("version" (version)))
+       (let ((url (hackage-source-url hackage-name version)))
+         (upstream-source
+          (package guix-package)
+          (version version)
+          (urls (list url))))))))
+
+(define %hackage-updater
+  (upstream-updater
+   (name 'hackage)
+   (description "Updater for Hackage packages")
+   (pred hackage-package?)
+   (latest latest-release)))
+
 ;;; cabal.scm ends here
diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index e541138..0efc190 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -35,6 +35,7 @@
                 #:select (%gnu-updater %gnome-updater %xorg-updater))
   #:use-module (guix import elpa)
   #:use-module (guix import cran)
+  #:use-module (guix import hackage)
   #:use-module (guix gnupg)
   #:use-module (gnu packages)
   #:use-module ((gnu packages commencement) #:select (%final-inputs))
@@ -198,6 +199,7 @@ unavailable optional dependencies such as Guile-JSON."
                  %elpa-updater
                  %cran-updater
                  %bioconductor-updater
+                 %hackage-updater
                  ((guix import pypi) => %pypi-updater)
                  ((guix import gem) => %gem-updater)
                  ((guix import github) => %github-updater)))
-- 
2.7.3

Re: [PATCH 1/3] import: hackage: Silence download output.

[Ludovic Courtès]

ericbavier@openmailbox.org skribis:

> From: Eric Bavier <bavier@member.fsf.org>
>
> * guix/import/hackage.scm (hackage-fetch): Send error output from
>   url-fetch to /dev/null.

[...]

> +    ;; XXX: We want to silence the download progress report, which is
> +    ;; especially annoying for 'guix refresh', but we have to use a file port.
>      (call-with-temporary-output-file
>       (lambda (temp port)
> -       (and (url-fetch url temp)
> +       (and (call-with-output-file "/dev/null"
> +              (lambda (null)
> +                (with-error-to-port null
> +                  (lambda () (url-fetch url temp)))))
>              (call-with-input-file temp
>                (compose read-cabal canonical-newline-port)))))))

Would it work to directly use:

  (let* ((port   (http-fetch url))    ;from (guix http-client)
         (result (read-cabal (canonical-newline-port port))))
    (close-port port)
    result)

This avoids the creation of a temporary file.

TIA,
Ludo’.

Re: [PATCH 2/3] import: hackage: Factorize url synthesis.

[Ludovic Courtès]

ericbavier@openmailbox.org skribis:

> From: Eric Bavier <bavier@member.fsf.org>
>
> * guix/import/hackage.scm (hackage-source-url, hackage-cabal-url): New
>   procedures.
>   (hackage-fetch, hackage-module->sexp): Use them.

OK!

Ludo'.

Re: [PATCH 3/3] import: Add Hackage updater.

[Ludovic Courtès]

ericbavier@openmailbox.org skribis:

> From: Eric Bavier <bavier@member.fsf.org>
>
> * guix/import/hackage.scm (guix-package->hackage-name, hackage-package?)
>   (latest-release): New procedures.
>   (%hackage-updater): New variable.
> * guix/scripts/refresh.scm (%updaters): Add it.
> * doc/guix.texi (Invoking guix refresh): Mention it.

[...]

> +(define guix-package->hackage-name
> +  (let ((uri-rx (make-regexp "https?://hackage.haskell.org/package/([^/]+)/.*"))
> +        (name-rx (make-regexp "(.*)-[0-9\\.]+")))
> +    (lambda (package)
> +      "Given a Guix package name, return the corresponding Hackage name."
> +      (let* ((source-url (and=> (package-source package) origin-uri))
> +             (name (match:substring (regexp-exec uri-rx source-url) 1)))
> +        (match (regexp-exec name-rx name)
> +          (#f name)
> +          (m (match:substring m 1)))))))

It might be useful to honor a ‘cabal-package-name’ property in cases
where guessing doesn’t work.  That can always be added later, though.

[...]

>                   %elpa-updater
>                   %cran-updater
>                   %bioconductor-updater
> +                 %hackage-updater

Make it:

  ((guix import hackage) => %hackage-importer)

to be on the safe side.

If you confirm that ‘guix refresh -t hackage’ doesn’t crash, go for it!

Thank you!

Ludo’.

Re: [PATCH 3/3] import: Add Hackage updater.

[Eric Bavier]

On Mon, 28 Mar 2016 18:12:05 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> ericbavier@openmailbox.org skribis:
> 
> > From: Eric Bavier <bavier@member.fsf.org>
> >
> > * guix/import/hackage.scm (guix-package->hackage-name, hackage-package?)
> >   (latest-release): New procedures.
> >   (%hackage-updater): New variable.
> > * guix/scripts/refresh.scm (%updaters): Add it.
> > * doc/guix.texi (Invoking guix refresh): Mention it.  
> 
> [...]
> 
> > +(define guix-package->hackage-name
> > +  (let ((uri-rx (make-regexp "https?://hackage.haskell.org/package/([^/]+)/.*"))
> > +        (name-rx (make-regexp "(.*)-[0-9\\.]+")))
> > +    (lambda (package)
> > +      "Given a Guix package name, return the corresponding Hackage name."
> > +      (let* ((source-url (and=> (package-source package) origin-uri))
> > +             (name (match:substring (regexp-exec uri-rx source-url) 1)))
> > +        (match (regexp-exec name-rx name)
> > +          (#f name)
> > +          (m (match:substring m 1)))))))  
> 
> It might be useful to honor a ‘cabal-package-name’ property in cases
> where guessing doesn’t work.  That can always be added later, though.

Sure.  Let's save that for later.  This scheme works for all packages
that we have so far, and seems like it should work for most others on
Hackage that I've seen.

> 
> [...]
> 
> >                   %elpa-updater
> >                   %cran-updater
> >                   %bioconductor-updater
> > +                 %hackage-updater  
> 
> Make it:
> 
>   ((guix import hackage) => %hackage-importer)
> 
> to be on the safe side.

I considered that at first, but it seems like that syntax is for
updaters that rely on guile-json (or other modules which might be
missing).  I'd prefer not to unnecessarily confuse the situation.

> If you confirm that ‘guix refresh -t hackage’ doesn’t crash, go for it!

I have done so, and it seems to work quite well.  There are a few
packages whose .cabal file fail to parse, so I included the
warning output to incite some future bug-fixing.

`~Eric

Re: [PATCH 3/3] import: Add Hackage updater.

[Ludovic Courtès]

Eric Bavier <ericbavier@openmailbox.org> skribis:

> On Mon, 28 Mar 2016 18:12:05 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:

[...]

>> It might be useful to honor a ‘cabal-package-name’ property in cases
>> where guessing doesn’t work.  That can always be added later, though.
>
> Sure.  Let's save that for later.  This scheme works for all packages
> that we have so far, and seems like it should work for most others on
> Hackage that I've seen.

Sure.

>> 
>> [...]
>> 
>> >                   %elpa-updater
>> >                   %cran-updater
>> >                   %bioconductor-updater
>> > +                 %hackage-updater  
>> 
>> Make it:
>> 
>>   ((guix import hackage) => %hackage-importer)
>> 
>> to be on the safe side.
>
> I considered that at first, but it seems like that syntax is for
> updaters that rely on guile-json (or other modules which might be
> missing).  I'd prefer not to unnecessarily confuse the situation.

I thought (guix import hackage) would depend on guile-json via (guix
import utils), but apparently no, so you’re right.

>> If you confirm that ‘guix refresh -t hackage’ doesn’t crash, go for it!
>
> I have done so, and it seems to work quite well.  There are a few
> packages whose .cabal file fail to parse, so I included the
> warning output to incite some future bug-fixing.

Awesome!

Thanks,
Ludo’.