[PATCH] draft addition of github updater

[PATCH] draft addition of github updater

[Ben Woodcroft]

[-- Attachment #1: Type: text/plain, Size: 2272 bytes --]

Hi,

Importing from GitHub seems very non-trivial, but can we update? There's 
a number of issues with the attached patch but so far out of the 171 
github package in guix, it recognizes 101, and 17 are detected as out of 
date (see below).

I have two questions:

1. Some guess-work is required to get between the version as it is 
defined in guix, and that presented in the github json, where only the 
"tag_name" is available. Is it OK to be a little speculative in this 
conversion e.g. "v1.0" => "1.0"?

2. For mass-updates, it fails when it hits the abuse limit on github (60 
api requests per hour). This can be overcome by authenticating with an 
access token, but I don't think that token should go in the git 
repository. So I'm after some guidance on the best way of the user 
providing a token to the updater (or some other workaround).

Thanks,
ben

gnu/packages/xml.scm:378:13: pugixml would be upgraded from 1.6 to 1.7
gnu/packages/web.scm:685:6: sassc would be upgraded from 3.2.5 to 3.3.2
gnu/packages/video.scm:693:13: mpv would be upgraded from 0.11.0 to 0.13.0
gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to 
4.02.0+1
gnu/packages/ninja.scm:31:13: ninja would be upgraded from 1.5.3 to 1.6.0
gnu/packages/jrnl.scm:30:13: jrnl would be upgraded from 1.8.4 to 1.9.7
gnu/packages/gl.scm:453:13: libepoxy would be upgraded from 1.2 to 1.3.1
gnu/packages/game-development.scm:123:13: tiled would be upgraded from 
0.13.1 to 0.14.2
gnu/packages/fontutils.scm:285:13: libuninameslist would be upgraded 
from 0.4.20140731 to 0.5.20150701
gnu/packages/engineering.scm:58:13: librecad would be upgraded from 
2.0.6-rc to 2.0.8
gnu/packages/bioinformatics.scm:1530:13: htsjdk would be upgraded from 
1.129 to 1.140
gnu/packages/bioinformatics.scm:613:13: bowtie would be upgraded from 
2.2.4 to 2.2.6
gnu/packages/bioinformatics.scm:2925:13: vsearch would be upgraded from 
1.4.1 to 1.9.1
gnu/packages/bioinformatics.scm:1360:13: grit would be upgraded from 
2.0.2 to 2.0.5beta4
gnu/packages/bioinformatics.scm:758:13: clipper would be upgraded from 
0.3.0 to 1.0
gnu/packages/bioinformatics.scm:207:13: bedtools would be upgraded from 
2.24.0 to 2.25.0
gnu/packages/bioinformatics.scm:1610:13: idr would be upgraded from 
2.0.0 to 2.0.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-import-Add-github-updater.patch --]
[-- Type: text/x-patch; name="0001-import-Add-github-updater.patch", Size: 6473 bytes --]

From 8072ee3ac66a71b74e79af4047d4f03bac9fed48 Mon Sep 17 00:00:00 2001
From: Ben Woodcroft <donttrustben@gmail.com>
Date: Sun, 15 Nov 2015 10:18:05 +1000
Subject: [PATCH] import: Add github-updater.

* guix/import/github.scm: New file.
* guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER
---
 guix/import/github.scm   | 118 +++++++++++++++++++++++++++++++++++++++++++++++
 guix/scripts/refresh.scm |   4 +-
 2 files changed, 121 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/github.scm

diff --git a/guix/import/github.scm b/guix/import/github.scm
new file mode 100644
index 0000000..2fecb0a
--- /dev/null
+++ b/guix/import/github.scm
@@ -0,0 +1,118 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2015 Ben Woodcroft <donttrustben@gmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;; TODO: Are all of these imports used?
+(define-module (guix import github)
+  #:use-module (ice-9 binary-ports)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 pretty-print)
+  #:use-module (ice-9 regex)
+  #:use-module ((ice-9 rdelim) #:select (read-line))
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (rnrs bytevectors)
+  #:use-module (json)
+  #:use-module (web uri)
+  #:use-module (guix ui)
+  #:use-module (guix utils)
+  #:use-module ((guix download) #:prefix download:)
+  #:use-module (guix import utils)
+  #:use-module (guix import json)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (gnu packages)
+  #:export (%github-updater))
+
+(define (json-fetch* url)
+  "Return a list/hash representation of the JSON resource URL, or #f on
+failure."
+  ;; TODO: make silent
+  (call-with-temporary-output-file
+   (lambda (temp port)
+     (and (url-fetch url temp)
+          (call-with-input-file temp json->scm)))))
+
+(define (github-package? package)
+  "Return true if PACKAGE is a package from GitHub."
+
+  ;; TODO: currently requires the standard "v1.0" or "1.0" style tag names
+  ;; TODO: currently only accepts .tar.gz downloads
+  ;; TODO: should also accept alternative download URLs of style like
+  ;; https://github.com/libical/libical/releases/download/v1.0.1/libical-1.0.1.tar.gz
+  (define (github-url? url)
+    (and
+     (string-prefix? "https://github.com/" url)
+     (or
+      (string-suffix?
+       (string-append "/archive/v" (package-version package) ".tar.gz") url)
+      (string-suffix?
+       (string-append "/archive/" (package-version package) ".tar.gz") url))))
+
+  (let ((source-url (and=> (package-source package) origin-uri))
+        (fetch-method (and=> (package-source package) origin-method)))
+    (display (list "testing" source-url))
+    (display "\n")
+    (and (eq? fetch-method download:url-fetch)
+         (match source-url
+           ((? string?)
+            (github-url? source-url))
+           ((source-url ...)
+            (any github-url? source-url))))))
+
+(define (github-user-slash-repository url)
+  "Return a string e.g. arq5x/bedtools2 of the owner and the name of the
+repository separated by a forward slash, from a string URL of the form
+'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
+  (let ((splits (string-split url #\/)))
+    (string-append (list-ref splits 3) "/" (list-ref splits 4))))
+
+(define (latest-released-version url)
+  "Return a string of the newest released version name given a string URL like
+'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz', or #f if there
+is no releases"
+  ;; TODO: don't return pre-release versions, can detect this from JSON field
+  ;; 'prerelease'
+  (let ((json (json-fetch*
+               (string-append "https://api.github.com/repos/"
+                              (github-user-slash-repository url)
+                              "/releases"
+                              ;;"?access_token=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+                              ))))
+        (if (eq? (length json) 0) #f
+            (let ((tag (assoc-ref (hash-table->alist (first json)) "tag_name")))
+              (if (eq? (string-ref tag 0) #\v)
+                  (substring tag 1) tag)))))
+
+(define (latest-release guix-package)
+  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
+  (let* ((pkg (specification->package guix-package))
+         (source-uri (origin-uri (package-source pkg)))
+         (version (latest-released-version source-uri)))
+    (if version
+        (upstream-source
+         (package guix-package)
+         (version version)
+         (urls (list source-uri)))
+        #f)))
+
+(define %github-updater
+  (upstream-updater
+   (name 'github)
+   (description "Updater for GitHub packages")
+   (pred github-package?)
+   (latest latest-release)))
diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index 3161aac..f9ac0ed 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
 ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2015 Alex Kost <alezost@gmail.com>
+;;; Copyright © 2015 Ben Woodcroft <donttrustben@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -184,7 +185,8 @@ unavailable optional dependencies such as Guile-JSON."
   (list-updaters %gnu-updater
                  %elpa-updater
                  %cran-updater
-                 ((guix import pypi) => %pypi-updater)))
+                 ((guix import pypi) => %pypi-updater)
+                 ((guix import github) => %github-updater)))
 
 (define (lookup-updater name)
   "Return the updater called NAME."
-- 
2.5.0

Re: [PATCH] draft addition of github updater

[Ludovic Courtès]

Hi!

Ben Woodcroft <b.woodcroft@uq.edu.au> skribis:

> Importing from GitHub seems very non-trivial, but can we update?
> There's a number of issues with the attached patch but so far out of
> the 171 github package in guix, it recognizes 101, and 17 are detected
> as out of date (see below).

Woow, nice!

> I have two questions:
>
> 1. Some guess-work is required to get between the version as it is
> defined in guix, and that presented in the github json, where only the
> "tag_name" is available. Is it OK to be a little speculative in this
> conversion e.g. "v1.0" => "1.0"?

I guess so.  What I would do is do that conversion when the tag matches
“^v[0-9]” and leave the tag as-is in other cases.  WDYT?

We can always add more heuristics later if we find that there’s another
widely-used convention for tag names.

> 2. For mass-updates, it fails when it hits the abuse limit on github
> (60 api requests per hour). This can be overcome by authenticating
> with an access token, but I don't think that token should go in the
> git repository. So I'm after some guidance on the best way of the user
> providing a token to the updater (or some other workaround).

Argh, that’s annoying.  How does it fail exactly?  What’s the impact on
the behavior of ‘guix refresh’?

I guess (guix import github) could contain something like:

  (define %github-token
    ;; Token to be passed to Github.com to avoid the 60-request per hour
    ;; limit, or #f.
    (make-parameter (getenv "GUIX_GITHUB_TOKEN")))

and we’d need to document that, or maybe write a message hinting at it
when we know the limit has been reached.

WDYT?

> +;; TODO: Are all of these imports used?
> +(define-module (guix import github)
> +  #:use-module (ice-9 binary-ports)

By default modules are compiled with -Wunbound-variables, so you can
find out by removing modules until you get an “unbound variable”
warning.

> +(define (json-fetch* url)
> +  "Return a list/hash representation of the JSON resource URL, or #f on
> +failure."
> +  ;; TODO: make silent
> +  (call-with-temporary-output-file
> +   (lambda (temp port)
> +     (and (url-fetch url temp)
> +          (call-with-input-file temp json->scm)))))

See how ‘pypi-fetch’ makes it silent.

Overall it LGTM.

I was thinking we could have a generic Git updater that would look for
available tags upstream.  I wonder how efficient that would be compared
to using the GitHub-specific API, and if there would be other
differences.  What are your thoughts on this?

Thanks!

Ludo’.

Re: [PATCH] draft addition of github updater

[Ben Woodcroft]

[-- Attachment #1: Type: text/plain, Size: 7532 bytes --]

Thanks for the encouraging words. Here's the next revision.

On 16/11/15 19:15, Ludovic Courtès wrote:
> Hi!
>
> Ben Woodcroft <b.woodcroft@uq.edu.au> skribis:
>
>> Importing from GitHub seems very non-trivial, but can we update?
>> There's a number of issues with the attached patch but so far out of
>> the 171 github package in guix, it recognizes 101, and 17 are detected
>> as out of date (see below).
It seems I miscounted before, but now it is 129 of 146 github "release" 
packages recognised with 28 suggesting an update - see the end of email 
for details. There is one false positive:

gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to 
4.02.0+1

This happens because the newer versions were not made as official 
releases just tags, so the newer versions are omitted from the API 
response, plus there's the odd version numbering scheme. Guix is up to 
date.
>> I have two questions:
>>
>> 1. Some guess-work is required to get between the version as it is
>> defined in guix, and that presented in the github json, where only the
>> "tag_name" is available. Is it OK to be a little speculative in this
>> conversion e.g. "v1.0" => "1.0"?
> I guess so.  What I would do is do that conversion when the tag matches
> “^v[0-9]” and leave the tag as-is in other cases.  WDYT?
>
> We can always add more heuristics later if we find that there’s another
> widely-used convention for tag names.
Most seem to follow those few conventions, but there's still repos that 
decided to be different e.g.

https://github.com/vapoursynth/vapoursynth/archive/R28.tar.gz
https://github.com/synergy/synergy/archive/v1.7.4-stable.tar.gz

Having gotten this far, I wonder if I've gone about it backwards. 
Currently the updater works by asserting it is a refreshable package by 
interrogating the source URI only. But it might be easier to determine 
this with an API response on hand, by matching the current release 
version number to a tag. Then if we assume the same transformation of 
tag to version holds in the newest release, the reverse transformation 
can be used on the newest tag to convert it back into a version number. 
By transformation I mean addition of [a-z\.\-] characters before and 
after the version number. This is easier because guesswork is only 
needed to convert between the tag and version number, without reference 
to a URI.

This means more work for me, is it a good idea? As I understand it would 
involve returning #t more often from "github-package?". If #f is 
returned by an updater, do the updaters further down the chain get a 
bite at the cherry too? It doesn't matter for now since the github 
updater is last, but it might in the future.
>> 2. For mass-updates, it fails when it hits the abuse limit on github
>> (60 api requests per hour). This can be overcome by authenticating
>> with an access token, but I don't think that token should go in the
>> git repository. So I'm after some guidance on the best way of the user
>> providing a token to the updater (or some other workaround).
> Argh, that’s annoying.  How does it fail exactly?  What’s the impact on
> the behavior of ‘guix refresh’?
I didn't investigate thoroughly, but I believe it either gives a 403 or 
a more descriptive json string, dependent on the user-agent.

I added some words and errored out when json-fetch* returns #f. This was 
potentially a little lazy on my part as it might be better to detect the 
403 error as distinct from errors of other kinds, but it wasn't 
immediately obvious to me how to do this without going too deep into the 
fetching functions and/or duplicating code. WDYT?
>
> I guess (guix import github) could contain something like:
>
>    (define %github-token
>      ;; Token to be passed to Github.com to avoid the 60-request per hour
>      ;; limit, or #f.
>      (make-parameter (getenv "GUIX_GITHUB_TOKEN")))
>
> and we’d need to document that, or maybe write a message hinting at it
> when we know the limit has been reached.
>
> WDYT?
Seems we were all thinking the same thing - I've integrated this. Should 
we check that the token matches ^[0-9a-f]+$ for security and UI?
> I was thinking we could have a generic Git updater that would look for 
> available tags upstream. I wonder how efficient that would be compared 
> to using the GitHub-specific API, and if there would be other 
> differences. What are your thoughts on this?
This sounds like an excellent idea, but I was unable to find any way to 
fetch tags without a clone first. A clone could take a long time and a 
lot of bandwidth I would imagine. Also there's no way to discern regular 
releases from pre-releases I don't think. It is a bit unclear to me how 
conservative these updaters should be, are tags sufficiently synonymous 
with releases so as to be reported by refresh?

There's a number of github repos packaged that refer to git commits 
directly too, these are ignored by the current updater but might benefit 
from this approach (as well as non-github git repos of course).

Thanks,
ben



gnu/packages/xml.scm:380:13: pugixml would be upgraded from 1.6 to 1.7
gnu/packages/web.scm:353:13: libpsl would be upgraded from 0.7.1 to 0.11.0
gnu/packages/web.scm:685:6: sassc would be upgraded from 3.2.5 to 3.3.2
gnu/packages/version-control.scm:934:13: findnewest would be upgraded 
from 0.2 to 0.3
gnu/packages/telephony.scm:192:13: libsrtp would be upgraded from 1.5.2 
to 1.5.3
gnu/packages/ruby.scm:2373:13: ruby-sanitize would be upgraded from 
4.0.0 to 4.0.1
gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to 
4.02.0+1
gnu/packages/ninja.scm:31:13: ninja would be upgraded from 1.5.3 to 1.6.0
gnu/packages/maths.scm:1855:13: dealii would be upgraded from 8.2.1 to 8.3.0
gnu/packages/jrnl.scm:30:13: jrnl would be upgraded from 1.8.4 to 1.9.7
gnu/packages/gl.scm:453:13: libepoxy would be upgraded from 1.2 to 1.3.1
gnu/packages/game-development.scm:125:13: tiled would be upgraded from 
0.13.1 to 0.14.2
gnu/packages/fontutils.scm:285:13: libuninameslist would be upgraded 
from 0.4.20140731 to 0.5.20150701
gnu/packages/engineering.scm:58:13: librecad would be upgraded from 
2.0.6-rc to 2.0.8
gnu/packages/emacs.scm:436:13: haskell-mode would be upgraded from 
13.14.2 to 13.16
gnu/packages/conky.scm:35:13: conky would be upgraded from 1.10.0 to 1.10.1
gnu/packages/bioinformatics.scm:974:13: deeptools would be upgraded from 
1.5.11 to 1.5.12
gnu/packages/bioinformatics.scm:1532:13: htsjdk would be upgraded from 
1.129 to 2.0.1
gnu/packages/bioinformatics.scm:207:13: bedtools would be upgraded from 
2.24.0 to 2.25.0
gnu/packages/bioinformatics.scm:1880:13: orfm would be upgraded from 
0.4.1 to 0.5.2
gnu/packages/bioinformatics.scm:758:13: clipper would be upgraded from 
0.3.0 to 1.0
gnu/packages/bioinformatics.scm:1612:13: idr would be upgraded from 
2.0.0 to 2.0.2
gnu/packages/bioinformatics.scm:2592:13: preseq would be upgraded from 
2.0 to 2.0.2
gnu/packages/bioinformatics.scm:2978:13: vsearch would be upgraded from 
1.4.1 to 1.9.5
gnu/packages/bioinformatics.scm:1360:13: grit would be upgraded from 
2.0.2 to 2.0.4
gnu/packages/bioinformatics.scm:1577:13: htslib would be upgraded from 
1.2.1 to 1.3
gnu/packages/bioinformatics.scm:1013:13: diamond would be upgraded from 
0.7.9 to 0.7.10
gnu/packages/bioinformatics.scm:613:13: bowtie would be upgraded from 
2.2.4 to 2.2.6


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-import-Add-github-updater.patch --]
[-- Type: text/x-patch; name="0001-import-Add-github-updater.patch", Size: 11421 bytes --]

From a42eda6b9631cc28dfdd02d2c8bb02eabb2626b9 Mon Sep 17 00:00:00 2001
From: Ben Woodcroft <donttrustben@gmail.com>
Date: Sun, 15 Nov 2015 10:18:05 +1000
Subject: [PATCH] import: Add github-updater.

* guix/import/github.scm: New file.
* guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER.
* doc/guix.texi (Invoking guix refresh): Mention it.
---
 doc/guix.texi            |  14 ++++
 guix/import/github.scm   | 167 +++++++++++++++++++++++++++++++++++++++++++++++
 guix/scripts/refresh.scm |   5 +-
 3 files changed, 185 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/github.scm

diff --git a/doc/guix.texi b/doc/guix.texi
index 06d70ba..f6b7368 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -16,6 +16,7 @@ Copyright @copyright{} 2013 Nikita Karetnikov@*
 Copyright @copyright{} 2015 Mathieu Lirzin@*
 Copyright @copyright{} 2014 Pierre-Antoine Rault@*
 Copyright @copyright{} 2015 Taylan Ulrich Bayırlı/Kammer
+Copyright @copyright{} 2015 Ben Woodcroft
 
 Permission is granted to copy, distribute and/or modify this document
 under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -4354,6 +4355,16 @@ attempt is made to automatically retrieve it from a public key server;
 when it's successful, the key is added to the user's keyring; otherwise,
 @command{guix refresh} reports an error.
 
+The @code{github} updater uses the
+@uref{https://developer.github.com/v3/, GitHub API} to query for new
+releases. When used repeatedly e.g. when refreshing all packages, GitHub
+will eventually refuse to answer any further API requests. By default 60
+API requests per hour are allowed, and a full refresh on all GitHub
+packages in Guix requires more than this. Authentication with GitHub
+through the use of an API token alleviates these limits. To use an API
+token, set the environment variable @code{GUIX_GITHUB_TOKEN} to a token
+procured from @uref{https://github.com/settings/tokens} or otherwise.
+
 The following options are supported:
 
 @table @code
@@ -4415,6 +4426,8 @@ the updater for @uref{http://elpa.gnu.org/, ELPA} packages;
 the updater for @uref{http://cran.r-project.org/, CRAN} packages;
 @item pypi
 the updater for @uref{https://pypi.python.org, PyPI} packages.
+@item github
+the updater for @uref{https://github.com, GitHub} packages.
 @end table
 
 For instance, the following commands only checks for updates of Emacs
@@ -4501,6 +4514,7 @@ Use @var{host} as the OpenPGP key server when importing a public key.
 
 @end table
 
+
 @node Invoking guix lint
 @section Invoking @command{guix lint}
 The @command{guix lint} is meant to help package developers avoid common
diff --git a/guix/import/github.scm b/guix/import/github.scm
new file mode 100644
index 0000000..2ed477e
--- /dev/null
+++ b/guix/import/github.scm
@@ -0,0 +1,167 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2015 Ben Woodcroft <donttrustben@gmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;; TODO: Are all of these imports used?
+(define-module (guix import github)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (json)
+  #:use-module (guix utils)
+  #:use-module ((guix download) #:prefix download:)
+  #:use-module (guix import utils)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (gnu packages)
+  #:export (%github-updater))
+
+(define (json-fetch* url)
+  "Return a list/hash representation of the JSON resource URL, or #f on
+failure."
+  (call-with-output-file "/dev/null"
+    (lambda (null)
+      (with-error-to-port null
+        (lambda ()
+          (call-with-temporary-output-file
+           (lambda (temp port)
+             (and (url-fetch url temp)
+                  (call-with-input-file temp json->scm)))))))))
+
+;; TODO: is there some code from elsewhere in guix that can be used instead of
+;; redefining?
+(define (find-extension url)
+  "Return the extension of the archive e.g. '.tar.gz' given a URL, or
+false if none is recognized"
+  (find (lambda x (string-suffix? (first x) url))
+        (list ".tar.gz" ".tar.bz2" ".tar.xz" ".zip" ".tar")))
+
+(define (github-package? package)
+  "Return true if PACKAGE is a package from GitHub."
+
+  (define (github-url? url)
+    (and
+     (string-prefix? "https://github.com/" url)
+     (let ((ext (find-extension url)))
+       (and ext
+            (or
+             (string-suffix?
+              (string-append "/archive/v" (package-version package) ext) url)
+             (string-suffix?
+              (string-append "/archive/" (package-version package) ext) url)
+             (string-suffix?
+              (string-append "/archive/" (package-name package) "-"
+                             (package-version package) ext)
+              url)
+             (string-suffix?
+              (string-append "/releases/download/v" (package-version package)
+                             "/" (package-name package) "-"
+                             (package-version package) ext)
+              url)
+             (string-suffix?
+              (string-append "/releases/download/" (package-version package)
+                             "/" (package-name package) "-"
+                             (package-version package) ext)
+              url))))))
+
+  (let ((source-url (and=> (package-source package) origin-uri))
+        (fetch-method (and=> (package-source package) origin-method)))
+    (and (eq? fetch-method download:url-fetch)
+         (match source-url
+           ((? string?)
+            (github-url? source-url))
+           ((source-url ...)
+            (any github-url? source-url))))))
+
+(define (github-user-slash-repository url)
+  "Return a string e.g. arq5x/bedtools2 of the owner and the name of the
+repository separated by a forward slash, from a string URL of the form
+'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
+  (let ((splits (string-split url #\/)))
+    (string-append (list-ref splits 3) "/" (list-ref splits 4))))
+
+(define %github-token
+  ;; Token to be passed to Github.com to avoid the 60-request per hour
+  ;; limit, or #f.
+  ;; QUESTION: is there a need to check that the token looks like a token, for
+  ;; security, since it gets used in a fetch as is?
+  (make-parameter (getenv "GUIX_GITHUB_TOKEN")))
+
+(define (latest-released-version url package-name)
+  "Return a string of the newest released version name given a string URL like
+'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz' and the name of
+the package e.g. 'bedtools2'. Return #f if there is no releases"
+  (let* ((token (%github-token))
+         (api-url (string-append
+                   "https://api.github.com/repos/"
+                   (github-user-slash-repository url)
+                   "/releases"))
+         (json (json-fetch*
+                (if token
+                    (string-append api-url "?access_token=" token)
+                    api-url))))
+    (if (eq? json #f)
+        (if token
+            (error "Error downloading release information through the GitHub
+API when using a GitHub token")
+            (error "Error downloading release information through the GitHub
+API. This may be fixed by using an access token and setting the environment
+variable GUIX_GITHUB_TOKEN, for instance one procured from
+https://github.com/settings/tokens"))
+        (let ((proper-releases
+               (filter
+                (lambda (x)
+                  ;; example pre-release:
+                  ;; https://github.com/wwood/OrfM/releases/tag/v0.5.1
+                  ;; or an all-prerelease set
+                  ;; https://github.com/powertab/powertabeditor/releases
+                  (eq? (assoc-ref (hash-table->alist x) "prerelease") #f))
+                json)))
+          (if (eq? (length proper-releases) 0) #f ;empty releases list
+              (let*
+                  ((tag (assoc-ref (hash-table->alist (first proper-releases))
+                                   "tag_name"))
+                   (name-length (string-length package-name)))
+                ;; some tags include the name of the package e.g. "fdupes-1.51"
+                ;; so remove these
+                (if (and (< name-length (string-length tag))
+                         (string=? (string-append package-name "-")
+                                   (substring tag 0 (+ name-length 1))))
+                    (substring tag (+ name-length 1))
+                    ;; some tags start with a "v" e.g. "v0.25.0"
+                    ;; where some are just the version number
+                    (if (eq? (string-ref tag 0) #\v)
+                        (substring tag 1) tag))))))))
+
+(define (latest-release guix-package)
+  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
+  (let* ((pkg (specification->package guix-package))
+         (source-uri (origin-uri (package-source pkg)))
+         (name (package-name pkg))
+         (version (latest-released-version source-uri name)))
+    (if version
+        (upstream-source
+         (package guix-package)
+         (version version)
+         (urls (list source-uri)))
+        #f)))
+
+(define %github-updater
+  (upstream-updater
+   (name 'github)
+   (description "Updater for GitHub packages")
+   (pred github-package?)
+   (latest latest-release)))
diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index a5834d1..adbcf28 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
 ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2015 Alex Kost <alezost@gmail.com>
+;;; Copyright © 2015 Ben Woodcroft <donttrustben@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -34,6 +35,7 @@
                 #:select (%gnu-updater %gnome-updater))
   #:use-module (guix import elpa)
   #:use-module (guix import cran)
+  #:use-module (guix import github)
   #:use-module (guix gnupg)
   #:use-module (gnu packages)
   #:use-module ((gnu packages commencement) #:select (%final-inputs))
@@ -195,7 +197,8 @@ unavailable optional dependencies such as Guile-JSON."
                  %gnome-updater
                  %elpa-updater
                  %cran-updater
-                 ((guix import pypi) => %pypi-updater)))
+                 ((guix import pypi) => %pypi-updater)
+                 %github-updater))
 
 (define (lookup-updater name)
   "Return the updater called NAME."
-- 
2.5.0

Re: [PATCH] draft addition of github updater

[Ludovic Courtès]

Ben Woodcroft <b.woodcroft@uq.edu.au> skribis:

> It seems I miscounted before, but now it is 129 of 146 github
> "release" packages recognised with 28 suggesting an update - see the
> end of email for details. There is one false positive:
>
> gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to
> 4.02.0+1
>
> This happens because the newer versions were not made as official
> releases just tags, so the newer versions are omitted from the API
> response, plus there's the odd version numbering scheme. Guix is up to
> date.

I guess we could filter out such downgrades by adding a call to
‘version>?’, no?

>>> I have two questions:
>>>
>>> 1. Some guess-work is required to get between the version as it is
>>> defined in guix, and that presented in the github json, where only the
>>> "tag_name" is available. Is it OK to be a little speculative in this
>>> conversion e.g. "v1.0" => "1.0"?
>> I guess so.  What I would do is do that conversion when the tag matches
>> “^v[0-9]” and leave the tag as-is in other cases.  WDYT?
>>
>> We can always add more heuristics later if we find that there’s another
>> widely-used convention for tag names.
> Most seem to follow those few conventions, but there's still repos
> that decided to be different e.g.
>
> https://github.com/vapoursynth/vapoursynth/archive/R28.tar.gz
> https://github.com/synergy/synergy/archive/v1.7.4-stable.tar.gz
>
> Having gotten this far, I wonder if I've gone about it
> backwards. Currently the updater works by asserting it is a
> refreshable package by interrogating the source URI only. But it might
> be easier to determine this with an API response on hand, by matching
> the current release version number to a tag. Then if we assume the
> same transformation of tag to version holds in the newest release, the
> reverse transformation can be used on the newest tag to convert it
> back into a version number. By transformation I mean addition of
> [a-z\.\-] characters before and after the version number. This is
> easier because guesswork is only needed to convert between the tag and
> version number, without reference to a URI.
>
> This means more work for me, is it a good idea? As I understand it
> would involve returning #t more often from "github-package?". If #f is
> returned by an updater, do the updaters further down the chain get a
> bite at the cherry too? It doesn't matter for now since the github
> updater is last, but it might in the future.

I’m not sure I completely follow ;-), but it’s fine to hard-code the
v[0-9\.]+ convention for now, esp. if it works for most packages.

>> I guess (guix import github) could contain something like:
>>
>>    (define %github-token
>>      ;; Token to be passed to Github.com to avoid the 60-request per hour
>>      ;; limit, or #f.
>>      (make-parameter (getenv "GUIX_GITHUB_TOKEN")))
>>
>> and we’d need to document that, or maybe write a message hinting at it
>> when we know the limit has been reached.
>>
>> WDYT?
> Seems we were all thinking the same thing - I've integrated
> this. Should we check that the token matches ^[0-9a-f]+$ for security
> and UI?

I think it’s fine as is.  There’s no security issue on the client side
AFAICS.

>> I was thinking we could have a generic Git updater that would look
>> for available tags upstream. I wonder how efficient that would be
>> compared to using the GitHub-specific API, and if there would be
>> other differences. What are your thoughts on this?
> This sounds like an excellent idea, but I was unable to find any way
> to fetch tags without a clone first. A clone could take a long time
> and a lot of bandwidth I would imagine. Also there's no way to discern
> regular releases from pre-releases I don't think. It is a bit unclear
> to me how conservative these updaters should be, are tags sufficiently
> synonymous with releases so as to be reported by refresh?

I think we’d have to hard-code heuristics to distinguish release tags
from other tags.  Typically, again, considering only tags that match
‘v[0-9\.]+’.

Well, future work!  :-)

> From a42eda6b9631cc28dfdd02d2c8bb02eabb2626b9 Mon Sep 17 00:00:00 2001
> From: Ben Woodcroft <donttrustben@gmail.com>
> Date: Sun, 15 Nov 2015 10:18:05 +1000
> Subject: [PATCH] import: Add github-updater.
>
> * guix/import/github.scm: New file.
> * guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER.
> * doc/guix.texi (Invoking guix refresh): Mention it.

[...]

> +The @code{github} updater uses the
> +@uref{https://developer.github.com/v3/, GitHub API} to query for new
> +releases. When used repeatedly e.g. when refreshing all packages, GitHub
> +will eventually refuse to answer any further API requests. By default 60
> +API requests per hour are allowed, and a full refresh on all GitHub
> +packages in Guix requires more than this. Authentication with GitHub
> +through the use of an API token alleviates these limits. To use an API
> +token, set the environment variable @code{GUIX_GITHUB_TOKEN} to a token
> +procured from @uref{https://github.com/settings/tokens} or otherwise.

Good!  Please make sure to leave two spaces after end-of-sentence periods.

Also, maybe this paragraph should be moved after the @table that lists
updaters?  Otherwise it mentions the ‘github’ updater before it has been
introduced.

> +;; TODO: Are all of these imports used?
> +(define-module (guix import github)

Should be easily checked.  ;-)

> +(define (json-fetch* url)
> +  "Return a list/hash representation of the JSON resource URL, or #f on
> +failure."
> +  (call-with-output-file "/dev/null"
> +    (lambda (null)
> +      (with-error-to-port null
> +        (lambda ()
> +          (call-with-temporary-output-file
> +           (lambda (temp port)
> +             (and (url-fetch url temp)
> +                  (call-with-input-file temp json->scm)))))))))

Rather use (guix http-client) and something like:

  (let ((port (http-fetch url)))
    (dynamic-wind
      (const #t)
      (lambda ()
        (json->scm port))
      (lambda ()
        (close-port port))))

This avoids the temporary file creation etc.

> +;; TODO: is there some code from elsewhere in guix that can be used instead of
> +;; redefining?
> +(define (find-extension url)
> +  "Return the extension of the archive e.g. '.tar.gz' given a URL, or
> +false if none is recognized"
> +  (find (lambda x (string-suffix? (first x) url))
> +        (list ".tar.gz" ".tar.bz2" ".tar.xz" ".zip" ".tar")))

Remove this procedure and use (file-extension url) instead, from (guix utils).

> +(define (github-user-slash-repository url)
> +  "Return a string e.g. arq5x/bedtools2 of the owner and the name of the
> +repository separated by a forward slash, from a string URL of the form
> +'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
> +  (let ((splits (string-split url #\/)))
> +    (string-append (list-ref splits 3) "/" (list-ref splits 4))))

Rather write it as:

  (match (string-split (uri-path (string->uri url)) #\/)
    ((owner project . rest)
     (string-append owner "/" project)))

> +    (if (eq? json #f)

Rather: (if (not json).

However, ‘http-fetch’ raises an &http-error condition when something
goes wrong (it never returns #f.)  So…

> +        (if token
> +            (error "Error downloading release information through the GitHub
> +API when using a GitHub token")
> +            (error "Error downloading release information through the GitHub
> +API. This may be fixed by using an access token and setting the environment
> +variable GUIX_GITHUB_TOKEN, for instance one procured from
> +https://github.com/settings/tokens"))

… this can be removed, and the whole thing becomes:

  (guard (c ((http-get-error? c)
             (warning (_ "failed to access ~a: ~a (~a)~%")
                      (uri->string (http-get-error-uri c))
                      (http-get-error-code c)
                      (http-get-error-reason c))))
    …)

> +        (let ((proper-releases
> +               (filter
> +                (lambda (x)
> +                  ;; example pre-release:
> +                  ;; https://github.com/wwood/OrfM/releases/tag/v0.5.1
> +                  ;; or an all-prerelease set
> +                  ;; https://github.com/powertab/powertabeditor/releases
> +                  (eq? (assoc-ref (hash-table->alist x) "prerelease") #f))

Simply: (not (hash-ref x "prerelease")).

> +          (if (eq? (length proper-releases) 0) #f ;empty releases list
> +              (let*
> +                  ((tag (assoc-ref (hash-table->alist (first proper-releases))
> +                                   "tag_name"))

Rather:

  (match proper-releases
    (()                       ;empty release list
     #f)
    ((release . rest)         ;one or more releases
     (let* ((tag (hash-ref release "tag_name")) …)
       …)))

> +(define (latest-release guix-package)
> +  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
> +  (let* ((pkg (specification->package guix-package))

Someone (Ricardo?) proposed recently to pass a package object instead of
a package name to ‘latest-release’.

We should do that ideally before this patch goes in, or otherwise soon.

> -                 ((guix import pypi) => %pypi-updater)))
> +                 ((guix import pypi) => %pypi-updater)
> +                 %github-updater))

Write it as:

  ((guix import github) => %github-updater)

so that users who do not have guile-json can still use ‘guix refresh’.

Could you send an updated patch?  Looks like we’re almost there.

Thank you!

Ludo’.

Re: [PATCH] draft addition of github updater

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

>> +(define (latest-release guix-package)
>> +  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
>> +  (let* ((pkg (specification->package guix-package))
>
> Someone (Ricardo?) proposed recently to pass a package object instead of
> a package name to ‘latest-release’.
>
> We should do that ideally before this patch goes in, or otherwise soon.

Yes, it was me.  I’m still going through some of the many emails I
dropped (sorry, everone!) and hope to be able to write a patch to use
package objects soon.

~~ Ricardo

Updaters now receive package objects

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>>> +(define (latest-release guix-package)
>>> +  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
>>> +  (let* ((pkg (specification->package guix-package))
>>
>> Someone (Ricardo?) proposed recently to pass a package object instead of
>> a package name to ‘latest-release’.
>>
>> We should do that ideally before this patch goes in, or otherwise soon.
>
> Yes, it was me.

For the record, I just did that in commit
7d27a0259bc7a37c04b17ffc2953837fcc3e75ff (initially because I wanted to
clean up the GNU updater, which I did in
63e8bb12a46fe6ff493e674fd7ccceb8729c6b47.)

Ludo’.

Re: [PATCH] draft addition of github updater

[Ben Woodcroft]

[-- Attachment #1: Type: text/plain, Size: 10055 bytes --]

Hi again,

Thanks for the comments Ludo.

Unfortunately I found a further bug - the updated URL for the new 
package was actually the old URL not the updated one, and fixing this 
required some refactoring.

I'm afraid I'm almost out of time for this until the end of March, so if 
there are any further substantive changes we might have to let this slip 
the upcoming release, unless someone else can continue this work. Soz..

One way in which this could be improved in the future would be to accept 
odd source GitHub URLs and return the newest version, but error out when 
the URL needs to be guessed. That way, at least all GitHub-sourced 
packages can be checked for updates even if they cannot all be updated 
in place. I don't think this would be especially hard to implement and 
would be quite reliable.

On 04/01/16 06:46, Ludovic Courtès wrote:
> Ben Woodcroft<b.woodcroft@uq.edu.au>  skribis:
>
>> It seems I miscounted before, but now it is 129 of 146 github
>> "release" packages recognised with 28 suggesting an update - see the
>> end of email for details. There is one false positive:
>>
>> gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to
>> 4.02.0+1
>>
>> This happens because the newer versions were not made as official
>> releases just tags, so the newer versions are omitted from the API
>> response, plus there's the odd version numbering scheme. Guix is up to
>> date.
> I guess we could filter out such downgrades by adding a call to
> ‘version>?’, no?

My impression is that code elsewhere (yours?) already does this, but 
version>? does not work as intended for this corner case.

[...]

>>> I guess (guix import github) could contain something like:
>>>
>>>     (define %github-token
>>>       ;; Token to be passed to Github.com to avoid the 60-request per hour
>>>       ;; limit, or #f.
>>>       (make-parameter (getenv "GUIX_GITHUB_TOKEN")))
>>>
>>> and we’d need to document that, or maybe write a message hinting at it
>>> when we know the limit has been reached.
>>>
>>> WDYT?
>> Seems we were all thinking the same thing - I've integrated
>> this. Should we check that the token matches ^[0-9a-f]+$ for security
>> and UI?
> I think it’s fine as is.  There’s no security issue on the client side
> AFAICS.

OK

>>> I was thinking we could have a generic Git updater that would look
>>> for available tags upstream. I wonder how efficient that would be
>>> compared to using the GitHub-specific API, and if there would be
>>> other differences. What are your thoughts on this?
>> This sounds like an excellent idea, but I was unable to find any way
>> to fetch tags without a clone first. A clone could take a long time
>> and a lot of bandwidth I would imagine. Also there's no way to discern
>> regular releases from pre-releases I don't think. It is a bit unclear
>> to me how conservative these updaters should be, are tags sufficiently
>> synonymous with releases so as to be reported by refresh?
> I think we’d have to hard-code heuristics to distinguish release tags
> from other tags.  Typically, again, considering only tags that match
> ‘v[0-9\.]+’.
>
> Well, future work!  :-)

OK.

>>  From a42eda6b9631cc28dfdd02d2c8bb02eabb2626b9 Mon Sep 17 00:00:00 2001
>> From: Ben Woodcroft<donttrustben@gmail.com>
>> Date: Sun, 15 Nov 2015 10:18:05 +1000
>> Subject: [PATCH] import: Add github-updater.
>>
>> * guix/import/github.scm: New file.
>> * guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER.
>> * doc/guix.texi (Invoking guix refresh): Mention it.
> [...]
>
>> +The @code{github} updater uses the
>> +@uref{https://developer.github.com/v3/, GitHub API} to query for new
>> +releases. When used repeatedly e.g. when refreshing all packages, GitHub
>> +will eventually refuse to answer any further API requests. By default 60
>> +API requests per hour are allowed, and a full refresh on all GitHub
>> +packages in Guix requires more than this. Authentication with GitHub
>> +through the use of an API token alleviates these limits. To use an API
>> +token, set the environment variable @code{GUIX_GITHUB_TOKEN} to a token
>> +procured from @uref{https://github.com/settings/tokens} or otherwise.
> Good!  Please make sure to leave two spaces after end-of-sentence periods.
>
> Also, maybe this paragraph should be moved after the @table that lists
> updaters?  Otherwise it mentions the ‘github’ updater before it has been
> introduced.

OK. I moved it to the end of the refresh section, not just after the table.

[...]
>> +(define (json-fetch* url)
>> +  "Return a list/hash representation of the JSON resource URL, or #f on
>> +failure."
>> +  (call-with-output-file "/dev/null"
>> +    (lambda (null)
>> +      (with-error-to-port null
>> +        (lambda ()
>> +          (call-with-temporary-output-file
>> +           (lambda (temp port)
>> +             (and (url-fetch url temp)
>> +                  (call-with-input-file temp json->scm)))))))))
> Rather use (guix http-client) and something like:
>
>    (let ((port (http-fetch url)))
>      (dynamic-wind
>        (const #t)
>        (lambda ()
>          (json->scm port))
>        (lambda ()
>          (close-port port))))
>
> This avoids the temporary file creation etc.

This sounds preferable but did not work as I kept getting 403 forbidden. 
Displaying the URI for instance with (string-append uri "\n") gives the 
below. I understand the error is trivial, but just wanted to communicate 
the URI object.

ERROR: In procedure string-append:
ERROR: In procedure string-append: Wrong type (expecting string): 
#<<uri> scheme: https userinfo: #f host: "api.github.com" port: #f path: 
"/repos/torognes/vsearch/releases" query: 
"access_token=27907952ef87f3691d592b9dcd93cd4b6f20625f" fragment: #f>

>> +;; TODO: is there some code from elsewhere in guix that can be used instead of
>> +;; redefining?
>> +(define (find-extension url)
>> +  "Return the extension of the archive e.g. '.tar.gz' given a URL, or
>> +false if none is recognized"
>> +  (find (lambda x (string-suffix? (first x) url))
>> +        (list ".tar.gz" ".tar.bz2" ".tar.xz" ".zip" ".tar")))
> Remove this procedure and use (file-extension url) instead, from (guix utils).

I figured there was something out there. The problem is file-extension 
returns, for example, "gz" when we are after "tar.gz".

>> +(define (github-user-slash-repository url)
>> +  "Return a string e.g. arq5x/bedtools2 of the owner and the name of the
>> +repository separated by a forward slash, from a string URL of the form
>> +'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
>> +  (let ((splits (string-split url #\/)))
>> +    (string-append (list-ref splits 3) "/" (list-ref splits 4))))
> Rather write it as:
>
>    (match (string-split (uri-path (string->uri url)) #\/)
>      ((owner project . rest)
>       (string-append owner "/" project)))
>
>> +    (if (eq? json #f)
> Rather: (if (not json).
>
> However, ‘http-fetch’ raises an &http-error condition when something
> goes wrong (it never returns #f.)  So…

Since we aren't using http-fetch for the above reasons I tried to use 
(if (not json)), but this did not work because "(if (list of hashes))" 
throws a Wrong type to apply error.

>> +        (if token
>> +            (error "Error downloading release information through the GitHub
>> +API when using a GitHub token")
>> +            (error "Error downloading release information through the GitHub
>> +API. This may be fixed by using an access token and setting the environment
>> +variable GUIX_GITHUB_TOKEN, for instance one procured from
>> +https://github.com/settings/tokens"))
> … this can be removed, and the whole thing becomes:
>
>    (guard (c ((http-get-error? c)
>               (warning (_ "failed to access ~a: ~a (~a)~%")
>                        (uri->string (http-get-error-uri c))
>                        (http-get-error-code c)
>                        (http-get-error-reason c))))
>      …)

I've not used for now.

>> +        (let ((proper-releases
>> +               (filter
>> +                (lambda (x)
>> +                  ;; example pre-release:
>> +                  ;;https://github.com/wwood/OrfM/releases/tag/v0.5.1
>> +                  ;; or an all-prerelease set
>> +                  ;;https://github.com/powertab/powertabeditor/releases
>> +                  (eq? (assoc-ref (hash-table->alist x) "prerelease") #f))
> Simply: (not (hash-ref x "prerelease")).

OK.

>> +          (if (eq? (length proper-releases) 0) #f ;empty releases list
>> +              (let*
>> +                  ((tag (assoc-ref (hash-table->alist (first proper-releases))
>> +                                   "tag_name"))
> Rather:
>
>    (match proper-releases
>      (()                       ;empty release list
>       #f)
>      ((release . rest)         ;one or more releases
>       (let* ((tag (hash-ref release "tag_name")) …)
>         …)))

OK.

>> +(define (latest-release guix-package)
>> +  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
>> +  (let* ((pkg (specification->package guix-package))
> Someone (Ricardo?) proposed recently to pass a package object instead of
> a package name to ‘latest-release’.
>
> We should do that ideally before this patch goes in, or otherwise soon.

As discussed in the other thread, let's just proceed without waiting for 
Ricardo's efforts.

>> -                 ((guix import pypi) => %pypi-updater)))
>> +                 ((guix import pypi) => %pypi-updater)
>> +                 %github-updater))
> Write it as:
>
>    ((guix import github) => %github-updater)
>
> so that users who do not have guile-json can still use ‘guix refresh’.

OK.

> Could you send an updated patch?  Looks like we’re almost there.

Not quite there it seems.

Thanks.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-import-Add-github-updater.patch --]
[-- Type: text/x-patch; name="0001-import-Add-github-updater.patch", Size: 12694 bytes --]

From e75c5f2b76fd5a3074a230b6764eb4cc879fa582 Mon Sep 17 00:00:00 2001
From: Ben Woodcroft <donttrustben@gmail.com>
Date: Sun, 15 Nov 2015 10:18:05 +1000
Subject: [PATCH] import: Add github-updater.

* guix/import/github.scm: New file.
* guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER.
* doc/guix.texi (Invoking guix refresh): Mention it.
---
 doc/guix.texi            |  15 ++++
 guix/import/github.scm   | 198 +++++++++++++++++++++++++++++++++++++++++++++++
 guix/scripts/refresh.scm |   4 +-
 3 files changed, 216 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/github.scm

diff --git a/doc/guix.texi b/doc/guix.texi
index b991cc1..0b76dac 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -17,6 +17,7 @@ Copyright @copyright{} 2015 Mathieu Lirzin@*
 Copyright @copyright{} 2014 Pierre-Antoine Rault@*
 Copyright @copyright{} 2015 Taylan Ulrich Bayırlı/Kammer@*
 Copyright @copyright{} 2015, 2016 Leo Famulari
+Copyright @copyright{} 2016 Ben Woodcroft
 
 Permission is granted to copy, distribute and/or modify this document
 under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -4573,6 +4574,8 @@ the updater for @uref{http://www.bioconductor.org/, Bioconductor} R packages;
 the updater for @uref{https://pypi.python.org, PyPI} packages.
 @item gem
 the updater for @uref{https://rubygems.org, RubyGems} packages.
+@item github
+the updater for @uref{https://github.com, GitHub} packages.
 @end table
 
 For instance, the following command only checks for updates of Emacs
@@ -4659,6 +4662,18 @@ Use @var{host} as the OpenPGP key server when importing a public key.
 
 @end table
 
+The @code{github} updater uses the
+@uref{https://developer.github.com/v3/, GitHub API} to query for new
+releases.  When used repeatedly e.g. when refreshing all packages,
+GitHub will eventually refuse to answer any further API requests.  By
+default 60 API requests per hour are allowed, and a full refresh on all
+GitHub packages in Guix requires more than this.  Authentication with
+GitHub through the use of an API token alleviates these limits.  To use
+an API token, set the environment variable @code{GUIX_GITHUB_TOKEN} to a
+token procured from @uref{https://github.com/settings/tokens} or
+otherwise.
+
+
 @node Invoking guix lint
 @section Invoking @command{guix lint}
 The @command{guix lint} command is meant to help package developers avoid
diff --git a/guix/import/github.scm b/guix/import/github.scm
new file mode 100644
index 0000000..c696dcb
--- /dev/null
+++ b/guix/import/github.scm
@@ -0,0 +1,198 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import github)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (json)
+  #:use-module (guix utils)
+  #:use-module ((guix download) #:prefix download:)
+  #:use-module (guix import utils)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (gnu packages)
+  #:use-module (web uri)
+  #:export (%github-updater))
+
+(define (json-fetch* url)
+  "Return a list/hash representation of the JSON resource URL, or #f on
+failure."
+  (call-with-output-file "/dev/null"
+    (lambda (null)
+      (with-error-to-port null
+        (lambda ()
+          (call-with-temporary-output-file
+           (lambda (temp port)
+             (and (url-fetch url temp)
+                  (call-with-input-file temp json->scm)))))))))
+
+(define (find-extension url)
+  "Return the extension of the archive e.g. '.tar.gz' given a URL, or
+false if none is recognized"
+  (find (lambda x (string-suffix? (first x) url))
+        (list ".tar.gz" ".tar.bz2" ".tar.xz" ".zip" ".tar")))
+
+(define (updated-github-url old-package new-version)
+  ;; Return a url for the OLD-PACKAGE with NEW-VERSION.  If no source url in
+  ;; the OLD-PACKAGE is a GitHub url, then return false.
+
+  (define (updated-url url)
+    (if (string-prefix? "https://github.com/" url)
+        (let ((ext     (find-extension url))
+              (name    (package-name old-package))
+              (version (package-version old-package))
+              (prefix  (string-append "https://github.com/"
+                                      (github-user-slash-repository url)))
+              (repo    (github-repository url)))
+          (cond
+           ((string-suffix? (string-append "/tarball/v" version) url)
+            (string-append prefix "/tarball/v" new-version))
+           ((string-suffix? (string-append "/tarball/" version) url)
+            (string-append prefix "/tarball/" new-version))
+           ((string-suffix? (string-append "/archive/v" version ext) url)
+            (string-append prefix "/archive/v" new-version ext))
+           ((string-suffix? (string-append "/archive/" version ext) url)
+            (string-append prefix "/archive/" new-version ext))
+           ((string-suffix? (string-append "/archive/" name "-" version ext)
+                            url)
+            (string-append prefix "/archive/" name "-" new-version ext))
+           ((string-suffix? (string-append "/releases/download/v" version "/"
+                                           name "-" version ext)
+                            url)
+            (string-append prefix "/releases/download/v" new-version "/" name
+                           "-" new-version ext))
+           ((string-suffix? (string-append "/releases/download/" version "/"
+                                           name "-" version ext)
+                            url)
+            (string-append prefix "/releases/download/" new-version "/" name
+                           "-" new-version ext))
+           ((string-suffix? (string-append "/releases/download/" version "/"
+                                           repo "-" version ext)
+                            url)
+            (string-append prefix "/releases/download/" new-version "/" repo
+                           "-" new-version ext))
+           ((string-suffix? (string-append "/releases/download/" repo "-"
+                                           version "/" repo "-" version ext)
+                            url)
+            (string-append "/releases/download/" repo "-" version "/" repo "-"
+                           version ext))
+           (#t #f))) ; Some URLs are not recognised.
+        #f))
+
+  (let ((source-url (and=> (package-source old-package) origin-uri))
+        (fetch-method (and=> (package-source old-package) origin-method)))
+    (if (eq? fetch-method download:url-fetch)
+        (match source-url
+          ((? string?)
+           (updated-url source-url))
+          ((source-url ...)
+           (find updated-url source-url)))
+        #f)))
+
+(define (github-package? package)
+  "Return true if PACKAGE is a package from GitHub, else false."
+  (not (eq? #f (updated-github-url package "dummy"))))
+
+(define (github-repository url)
+  "Return a string e.g. bedtools2 of the name of the repository, from a string
+URL of the form 'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
+  (match (string-split (uri-path (string->uri url)) #\/)
+    ((_ owner project . rest)
+     (string-append project))))
+
+(define (github-user-slash-repository url)
+  "Return a string e.g. arq5x/bedtools2 of the owner and the name of the
+repository separated by a forward slash, from a string URL of the form
+'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
+  (match (string-split (uri-path (string->uri url)) #\/)
+    ((_ owner project . rest)
+     (string-append owner "/" project))))
+
+(define %github-token
+  ;; Token to be passed to Github.com to avoid the 60-request per hour
+  ;; limit, or #f.
+  (make-parameter (getenv "GUIX_GITHUB_TOKEN")))
+
+(define (latest-released-version url package-name)
+  "Return a string of the newest released version name given a string URL like
+'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz' and the name of
+the package e.g. 'bedtools2'.  Return #f if there is no releases"
+  (let* ((token (%github-token))
+         (api-url (string-append
+                   "https://api.github.com/repos/"
+                   (github-user-slash-repository url)
+                   "/releases"))
+         (json (json-fetch*
+                (if token
+                    (string-append api-url "?access_token=" token)
+                    api-url))))
+    (if (eq? json #f)
+        (if token
+            (error "Error downloading release information through the GitHub
+API when using a GitHub token")
+            (error "Error downloading release information through the GitHub
+API. This may be fixed by using an access token and setting the environment
+variable GUIX_GITHUB_TOKEN, for instance one procured from
+https://github.com/settings/tokens"))
+        (let ((proper-releases
+               (filter
+                (lambda (x)
+                  ;; example pre-release:
+                  ;; https://github.com/wwood/OrfM/releases/tag/v0.5.1
+                  ;; or an all-prerelease set
+                  ;; https://github.com/powertab/powertabeditor/releases
+                  (not (hash-ref x "prerelease")))
+                json)))
+          (match proper-releases
+            (()                       ;empty release list
+             #f)
+            ((release . rest)         ;one or more releases
+             (let ((tag (hash-ref release "tag_name"))
+                   (name-length (string-length package-name)))
+               ;; some tags include the name of the package e.g. "fdupes-1.51"
+               ;; so remove these
+               (if (and (< name-length (string-length tag))
+                        (string=? (string-append package-name "-")
+                                  (substring tag 0 (+ name-length 1))))
+                   (substring tag (+ name-length 1))
+                   ;; some tags start with a "v" e.g. "v0.25.0"
+                   ;; where some are just the version number
+                   (if (eq? (string-ref tag 0) #\v)
+                       (substring tag 1) tag)))))))))
+
+(define (latest-release guix-package)
+  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
+  (let* ((pkg (specification->package guix-package))
+         (source-uri (origin-uri (package-source pkg)))
+         (name (package-name pkg))
+         (newest-version (latest-released-version source-uri name)))
+    (if newest-version
+        (upstream-source
+         (package pkg)
+         (version newest-version)
+         (urls (list (updated-github-url pkg newest-version))))
+        #f))) ; On GitHub but no proper releases
+
+(define %github-updater
+  (upstream-updater
+   (name 'github)
+   (description "Updater for GitHub packages")
+   (pred github-package?)
+   (latest latest-release)))
+
+
diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index bb38f09..24e97c7 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -43,6 +43,7 @@
   #:use-module (ice-9 regex)
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 format)
+  #:use-module (guix import github)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
@@ -199,7 +200,8 @@ unavailable optional dependencies such as Guile-JSON."
                  %cran-updater
                  %bioconductor-updater
                  ((guix import pypi) => %pypi-updater)
-                 ((guix import gem) => %gem-updater)))
+                 ((guix import gem) => %gem-updater)
+                 ((guix import github) => %github-updater)))
 
 (define (lookup-updater name)
   "Return the updater called NAME."
-- 
2.6.3

Re: [PATCH] draft addition of github updater

[Ben Woodcroft]

[-- Attachment #1: Type: text/plain, Size: 153 bytes --]

On 21/02/16 13:13, Ben Woodcroft wrote:
> +  #:use-module (guix import github)
Oops, forgot to remove this line which I was using for debug. Reattached.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-import-Add-github-updater.patch --]
[-- Type: text/x-patch; name="0001-import-Add-github-updater.patch", Size: 12448 bytes --]

From 29dc5a809e6d8796279911a993ef1b2237c810ca Mon Sep 17 00:00:00 2001
From: Ben Woodcroft <donttrustben@gmail.com>
Date: Sun, 15 Nov 2015 10:18:05 +1000
Subject: [PATCH] import: Add github-updater.

* guix/import/github.scm: New file.
* guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER.
* doc/guix.texi (Invoking guix refresh): Mention it.
---
 doc/guix.texi            |  15 ++++
 guix/import/github.scm   | 198 +++++++++++++++++++++++++++++++++++++++++++++++
 guix/scripts/refresh.scm |   3 +-
 3 files changed, 215 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/github.scm

diff --git a/doc/guix.texi b/doc/guix.texi
index b991cc1..0b76dac 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -17,6 +17,7 @@ Copyright @copyright{} 2015 Mathieu Lirzin@*
 Copyright @copyright{} 2014 Pierre-Antoine Rault@*
 Copyright @copyright{} 2015 Taylan Ulrich Bayırlı/Kammer@*
 Copyright @copyright{} 2015, 2016 Leo Famulari
+Copyright @copyright{} 2016 Ben Woodcroft
 
 Permission is granted to copy, distribute and/or modify this document
 under the terms of the GNU Free Documentation License, Version 1.3 or
@@ -4573,6 +4574,8 @@ the updater for @uref{http://www.bioconductor.org/, Bioconductor} R packages;
 the updater for @uref{https://pypi.python.org, PyPI} packages.
 @item gem
 the updater for @uref{https://rubygems.org, RubyGems} packages.
+@item github
+the updater for @uref{https://github.com, GitHub} packages.
 @end table
 
 For instance, the following command only checks for updates of Emacs
@@ -4659,6 +4662,18 @@ Use @var{host} as the OpenPGP key server when importing a public key.
 
 @end table
 
+The @code{github} updater uses the
+@uref{https://developer.github.com/v3/, GitHub API} to query for new
+releases.  When used repeatedly e.g. when refreshing all packages,
+GitHub will eventually refuse to answer any further API requests.  By
+default 60 API requests per hour are allowed, and a full refresh on all
+GitHub packages in Guix requires more than this.  Authentication with
+GitHub through the use of an API token alleviates these limits.  To use
+an API token, set the environment variable @code{GUIX_GITHUB_TOKEN} to a
+token procured from @uref{https://github.com/settings/tokens} or
+otherwise.
+
+
 @node Invoking guix lint
 @section Invoking @command{guix lint}
 The @command{guix lint} command is meant to help package developers avoid
diff --git a/guix/import/github.scm b/guix/import/github.scm
new file mode 100644
index 0000000..c696dcb
--- /dev/null
+++ b/guix/import/github.scm
@@ -0,0 +1,198 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import github)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (json)
+  #:use-module (guix utils)
+  #:use-module ((guix download) #:prefix download:)
+  #:use-module (guix import utils)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (gnu packages)
+  #:use-module (web uri)
+  #:export (%github-updater))
+
+(define (json-fetch* url)
+  "Return a list/hash representation of the JSON resource URL, or #f on
+failure."
+  (call-with-output-file "/dev/null"
+    (lambda (null)
+      (with-error-to-port null
+        (lambda ()
+          (call-with-temporary-output-file
+           (lambda (temp port)
+             (and (url-fetch url temp)
+                  (call-with-input-file temp json->scm)))))))))
+
+(define (find-extension url)
+  "Return the extension of the archive e.g. '.tar.gz' given a URL, or
+false if none is recognized"
+  (find (lambda x (string-suffix? (first x) url))
+        (list ".tar.gz" ".tar.bz2" ".tar.xz" ".zip" ".tar")))
+
+(define (updated-github-url old-package new-version)
+  ;; Return a url for the OLD-PACKAGE with NEW-VERSION.  If no source url in
+  ;; the OLD-PACKAGE is a GitHub url, then return false.
+
+  (define (updated-url url)
+    (if (string-prefix? "https://github.com/" url)
+        (let ((ext     (find-extension url))
+              (name    (package-name old-package))
+              (version (package-version old-package))
+              (prefix  (string-append "https://github.com/"
+                                      (github-user-slash-repository url)))
+              (repo    (github-repository url)))
+          (cond
+           ((string-suffix? (string-append "/tarball/v" version) url)
+            (string-append prefix "/tarball/v" new-version))
+           ((string-suffix? (string-append "/tarball/" version) url)
+            (string-append prefix "/tarball/" new-version))
+           ((string-suffix? (string-append "/archive/v" version ext) url)
+            (string-append prefix "/archive/v" new-version ext))
+           ((string-suffix? (string-append "/archive/" version ext) url)
+            (string-append prefix "/archive/" new-version ext))
+           ((string-suffix? (string-append "/archive/" name "-" version ext)
+                            url)
+            (string-append prefix "/archive/" name "-" new-version ext))
+           ((string-suffix? (string-append "/releases/download/v" version "/"
+                                           name "-" version ext)
+                            url)
+            (string-append prefix "/releases/download/v" new-version "/" name
+                           "-" new-version ext))
+           ((string-suffix? (string-append "/releases/download/" version "/"
+                                           name "-" version ext)
+                            url)
+            (string-append prefix "/releases/download/" new-version "/" name
+                           "-" new-version ext))
+           ((string-suffix? (string-append "/releases/download/" version "/"
+                                           repo "-" version ext)
+                            url)
+            (string-append prefix "/releases/download/" new-version "/" repo
+                           "-" new-version ext))
+           ((string-suffix? (string-append "/releases/download/" repo "-"
+                                           version "/" repo "-" version ext)
+                            url)
+            (string-append "/releases/download/" repo "-" version "/" repo "-"
+                           version ext))
+           (#t #f))) ; Some URLs are not recognised.
+        #f))
+
+  (let ((source-url (and=> (package-source old-package) origin-uri))
+        (fetch-method (and=> (package-source old-package) origin-method)))
+    (if (eq? fetch-method download:url-fetch)
+        (match source-url
+          ((? string?)
+           (updated-url source-url))
+          ((source-url ...)
+           (find updated-url source-url)))
+        #f)))
+
+(define (github-package? package)
+  "Return true if PACKAGE is a package from GitHub, else false."
+  (not (eq? #f (updated-github-url package "dummy"))))
+
+(define (github-repository url)
+  "Return a string e.g. bedtools2 of the name of the repository, from a string
+URL of the form 'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
+  (match (string-split (uri-path (string->uri url)) #\/)
+    ((_ owner project . rest)
+     (string-append project))))
+
+(define (github-user-slash-repository url)
+  "Return a string e.g. arq5x/bedtools2 of the owner and the name of the
+repository separated by a forward slash, from a string URL of the form
+'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
+  (match (string-split (uri-path (string->uri url)) #\/)
+    ((_ owner project . rest)
+     (string-append owner "/" project))))
+
+(define %github-token
+  ;; Token to be passed to Github.com to avoid the 60-request per hour
+  ;; limit, or #f.
+  (make-parameter (getenv "GUIX_GITHUB_TOKEN")))
+
+(define (latest-released-version url package-name)
+  "Return a string of the newest released version name given a string URL like
+'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz' and the name of
+the package e.g. 'bedtools2'.  Return #f if there is no releases"
+  (let* ((token (%github-token))
+         (api-url (string-append
+                   "https://api.github.com/repos/"
+                   (github-user-slash-repository url)
+                   "/releases"))
+         (json (json-fetch*
+                (if token
+                    (string-append api-url "?access_token=" token)
+                    api-url))))
+    (if (eq? json #f)
+        (if token
+            (error "Error downloading release information through the GitHub
+API when using a GitHub token")
+            (error "Error downloading release information through the GitHub
+API. This may be fixed by using an access token and setting the environment
+variable GUIX_GITHUB_TOKEN, for instance one procured from
+https://github.com/settings/tokens"))
+        (let ((proper-releases
+               (filter
+                (lambda (x)
+                  ;; example pre-release:
+                  ;; https://github.com/wwood/OrfM/releases/tag/v0.5.1
+                  ;; or an all-prerelease set
+                  ;; https://github.com/powertab/powertabeditor/releases
+                  (not (hash-ref x "prerelease")))
+                json)))
+          (match proper-releases
+            (()                       ;empty release list
+             #f)
+            ((release . rest)         ;one or more releases
+             (let ((tag (hash-ref release "tag_name"))
+                   (name-length (string-length package-name)))
+               ;; some tags include the name of the package e.g. "fdupes-1.51"
+               ;; so remove these
+               (if (and (< name-length (string-length tag))
+                        (string=? (string-append package-name "-")
+                                  (substring tag 0 (+ name-length 1))))
+                   (substring tag (+ name-length 1))
+                   ;; some tags start with a "v" e.g. "v0.25.0"
+                   ;; where some are just the version number
+                   (if (eq? (string-ref tag 0) #\v)
+                       (substring tag 1) tag)))))))))
+
+(define (latest-release guix-package)
+  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
+  (let* ((pkg (specification->package guix-package))
+         (source-uri (origin-uri (package-source pkg)))
+         (name (package-name pkg))
+         (newest-version (latest-released-version source-uri name)))
+    (if newest-version
+        (upstream-source
+         (package pkg)
+         (version newest-version)
+         (urls (list (updated-github-url pkg newest-version))))
+        #f))) ; On GitHub but no proper releases
+
+(define %github-updater
+  (upstream-updater
+   (name 'github)
+   (description "Updater for GitHub packages")
+   (pred github-package?)
+   (latest latest-release)))
+
+
diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index bb38f09..6ca2d79 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -199,7 +199,8 @@ unavailable optional dependencies such as Guile-JSON."
                  %cran-updater
                  %bioconductor-updater
                  ((guix import pypi) => %pypi-updater)
-                 ((guix import gem) => %gem-updater)))
+                 ((guix import gem) => %gem-updater)
+                 ((guix import github) => %github-updater)))
 
 (define (lookup-updater name)
   "Return the updater called NAME."
-- 
2.6.3

Re: [PATCH] draft addition of github updater

[Ludovic Courtès]

Ben Woodcroft <b.woodcroft@uq.edu.au> skribis:

> Unfortunately I found a further bug - the updated URL for the new
> package was actually the old URL not the updated one, and fixing this
> required some refactoring.

OK.

> I'm afraid I'm almost out of time for this until the end of March, so
> if there are any further substantive changes we might have to let this
> slip the upcoming release, unless someone else can continue this
> work. Soz..

No problem.  It’s OK to leave improvements for later.  We can always add
this version now as long as it’s functional and doesn’t break anything.

> One way in which this could be improved in the future would be to
> accept odd source GitHub URLs and return the newest version, but error
> out when the URL needs to be guessed. That way, at least all
> GitHub-sourced packages can be checked for updates even if they cannot
> all be updated in place. I don't think this would be especially hard
> to implement and would be quite reliable.

OK.

> On 04/01/16 06:46, Ludovic Courtès wrote:
>> Ben Woodcroft<b.woodcroft@uq.edu.au>  skribis:
>>
>>> It seems I miscounted before, but now it is 129 of 146 github
>>> "release" packages recognised with 28 suggesting an update - see the
>>> end of email for details. There is one false positive:
>>>
>>> gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to
>>> 4.02.0+1
>>>
>>> This happens because the newer versions were not made as official
>>> releases just tags, so the newer versions are omitted from the API
>>> response, plus there's the odd version numbering scheme. Guix is up to
>>> date.
>> I guess we could filter out such downgrades by adding a call to
>> ‘version>?’, no?
>
> My impression is that code elsewhere (yours?) already does this, but
> version>? does not work as intended for this corner case.

Indeed:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (version>? "4.02+6" "4.02.0+1")
$2 = #f
--8<---------------cut here---------------end--------------->8---

I would argue that upstream chose a confusing numbering scheme is
4.02.0+1 is supposed to be older…

>> Rather use (guix http-client) and something like:
>>
>>    (let ((port (http-fetch url)))
>>      (dynamic-wind
>>        (const #t)
>>        (lambda ()
>>          (json->scm port))
>>        (lambda ()
>>          (close-port port))))
>>
>> This avoids the temporary file creation etc.
>
> This sounds preferable but did not work as I kept getting 403
> forbidden. Displaying the URI for instance with (string-append uri
> "\n") gives the below. I understand the error is trivial, but just
> wanted to communicate the URI object.
>
> ERROR: In procedure string-append:
> ERROR: In procedure string-append: Wrong type (expecting string):
> #<<uri> scheme: https userinfo: #f host: "api.github.com" port: #f
> path: "/repos/torognes/vsearch/releases" query:
> "access_token=27907952ef87f3691d592b9dcd93cd4b6f20625f" fragment: #f>

That’s because this is a URI object, not a string.

>>> +;; TODO: is there some code from elsewhere in guix that can be used instead of
>>> +;; redefining?
>>> +(define (find-extension url)
>>> +  "Return the extension of the archive e.g. '.tar.gz' given a URL, or
>>> +false if none is recognized"
>>> +  (find (lambda x (string-suffix? (first x) url))
>>> +        (list ".tar.gz" ".tar.bz2" ".tar.xz" ".zip" ".tar")))
>> Remove this procedure and use (file-extension url) instead, from (guix utils).
>
> I figured there was something out there. The problem is file-extension
> returns, for example, "gz" when we are after "tar.gz".

Oh, I see.

> From 29dc5a809e6d8796279911a993ef1b2237c810ca Mon Sep 17 00:00:00 2001
> From: Ben Woodcroft <donttrustben@gmail.com>
> Date: Sun, 15 Nov 2015 10:18:05 +1000
> Subject: [PATCH] import: Add github-updater.
>
> * guix/import/github.scm: New file.
> * guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER.
> * doc/guix.texi (Invoking guix refresh): Mention it.

Make sure to add github.scm in Makefile.am.  Otherwise LGTM!

Once this is in, I’ll see if I can make that ‘http-fetch’ change I was
suggesting.

Thank you!

Ludo’.

Re: [PATCH] draft addition of github updater

[Ben Woodcroft]

On 23/02/16 08:22, Ludovic Courtès wrote:
[ ... ]
>> On 04/01/16 06:46, Ludovic Courtès wrote:
>>> Ben Woodcroft<b.woodcroft@uq.edu.au>  skribis:
>>>
>>>> It seems I miscounted before, but now it is 129 of 146 github
>>>> "release" packages recognised with 28 suggesting an update - see the
>>>> end of email for details. There is one false positive:
>>>>
>>>> gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to
>>>> 4.02.0+1
>>>>
>>>> This happens because the newer versions were not made as official
>>>> releases just tags, so the newer versions are omitted from the API
>>>> response, plus there's the odd version numbering scheme. Guix is up to
>>>> date.
>>> I guess we could filter out such downgrades by adding a call to
>>> ‘version>?’, no?
>> My impression is that code elsewhere (yours?) already does this, but
>> version>? does not work as intended for this corner case.
> Indeed:
>
> --8<---------------cut here---------------start------------->8---
> scheme@(guile-user)> (version>? "4.02+6" "4.02.0+1")
> $2 = #f
> --8<---------------cut here---------------end--------------->8---
>
> I would argue that upstream chose a confusing numbering scheme is
> 4.02.0+1 is supposed to be older…

Indeed, I think it is OK to leave this.

>>> Rather use (guix http-client) and something like:
>>>
>>>     (let ((port (http-fetch url)))
>>>       (dynamic-wind
>>>         (const #t)
>>>         (lambda ()
>>>           (json->scm port))
>>>         (lambda ()
>>>           (close-port port))))
>>>
>>> This avoids the temporary file creation etc.
>> This sounds preferable but did not work as I kept getting 403
>> forbidden. Displaying the URI for instance with (string-append uri
>> "\n") gives the below. I understand the error is trivial, but just
>> wanted to communicate the URI object.
>>
>> ERROR: In procedure string-append:
>> ERROR: In procedure string-append: Wrong type (expecting string):
>> #<<uri> scheme: https userinfo: #f host: "api.github.com" port: #f
>> path: "/repos/torognes/vsearch/releases" query:
>> "access_token=27907952ef87f3691d592b9dcd93cd4b6f20625f" fragment: #f>
> That’s because this is a URI object, not a string.

I had a fresh crack at this, trying among other things:

(define (json-fetch* url)
   "Return a list/hash representation of the JSON resource URL, or #f on
failure."
   (display (string-append url "\n"))
   (let ((port (http-fetch url)))
     (dynamic-wind
       (const #t)
       (lambda ()
         (json->scm port))
       (lambda ()
         (close-port port)))))

and got
$ ./pre-inst-env guix refresh -t github vsearch
https://api.github.com/repos/torognes/vsearch/releases?access_token=27907952ef87f3691d592b9dcd93cd4b6f20625f
guix refresh: error: download failed

>>  From 29dc5a809e6d8796279911a993ef1b2237c810ca Mon Sep 17 00:00:00 2001
>> From: Ben Woodcroft <donttrustben@gmail.com>
>> Date: Sun, 15 Nov 2015 10:18:05 +1000
>> Subject: [PATCH] import: Add github-updater.
>>
>> * guix/import/github.scm: New file.
>> * guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER.
>> * doc/guix.texi (Invoking guix refresh): Mention it.
> Make sure to add github.scm in Makefile.am.  Otherwise LGTM!
>
> Once this is in, I’ll see if I can make that ‘http-fetch’ change I was
> suggesting.
>
> Thank you!

No problem at all, thanks for babying me through this.

Pushed as 917a2a58.
ben

Re: [PATCH] draft addition of github updater

[Ricardo Wurmus]

Ben Woodcroft <b.woodcroft@uq.edu.au> writes:

> Pushed as 917a2a58.

Yay!  This allows us to strike off another item from the list of
features to add before the 0.9.1 release.

Thank you!

~~ Ricardo

Re: [PATCH] draft addition of github updater

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Ben Woodcroft <b.woodcroft@uq.edu.au> writes:
>
>> Pushed as 917a2a58.
>
> Yay!  This allows us to strike off another item from the list of
> features to add before the 0.9.1 release.

+1  :-)

We have even fewer excuses to have outdated packages.  ;-)

Ludo’.

Re: [PATCH] draft addition of github updater

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3505 bytes --]

On Sun, 15 Nov 2015 10:32:40 +1000
Ben Woodcroft <b.woodcroft@uq.edu.au> wrote:

> Hi,
> 
> Importing from GitHub seems very non-trivial, but can we update? There's 
> a number of issues with the attached patch but so far out of the 171 
> github package in guix, it recognizes 101, and 17 are detected as out of 
> date (see below).
> 
> I have two questions:
> 
> 1. Some guess-work is required to get between the version as it is 
> defined in guix, and that presented in the github json, where only the 
> "tag_name" is available. Is it OK to be a little speculative in this 
> conversion e.g. "v1.0" => "1.0"?

from what I've seen, there are two sets of urls for github, and looking at
the notes in your patch, it looks like you've seen that too. I think your
v1.0 => 1.0 change is ok, it's also part of github's url structure. 

> 2. For mass-updates, it fails when it hits the abuse limit on github (60 
> api requests per hour). This can be overcome by authenticating with an 
> access token, but I don't think that token should go in the git 
> repository. So I'm after some guidance on the best way of the user 
> providing a token to the updater (or some other workaround).

Ouch. Will it accept an environmental variable for the access token? Or an
entry in .netrc or in .ssh/config? If I already have an ssh key registered
with github does that change anything? Not directly relevant, but minitube
accepts either having an API key embedded in the compiled version (what
debian does iirc) or querying an environmental variable.

> Thanks,
> ben
> 
> gnu/packages/xml.scm:378:13: pugixml would be upgraded from 1.6 to 1.7
> ...
> gnu/packages/bioinformatics.scm:1610:13: idr would be upgraded from 
> 2.0.0 to 2.0.2

My email client won't let me comment on patches, so I'll add it here:
+  ;; TODO: currently requires the standard "v1.0" or "1.0" style tag names
+  ;; TODO: currently only accepts .tar.gz downloads


+  ;; TODO: should also accept alternative download URLs of style like
+  ;; https://github.com/libical/libical/releases/download/v1.0.1/libical-1.0.1.tar.gz

tilda's url is: https://github.com/lanoxx/tilda/archive/tilda-1.2.4.tar.gz
looking at your example and my example, we have:
(string-suffix?
  (string-append "/download/v" (package-version package) "/" (package-name package) "-" (package-version package) ".tar.gz"))
(string-suffix?
  (string-append "/archive/" (package-name package) "-" (package-version package) ".tar.gz"))

+  (define (github-url? url)
+    (and
+     (string-prefix? "https://github.com/" url)
+     (or
+      (string-suffix?
+       (string-append "/archive/v" (package-version package) ".tar.gz") url)
+      (string-suffix?
+       (string-append "/archive/" (package-version package) ".tar.gz") url))))

I don't know that archive/${version}.tar.gz is actually anywhere, but a quick search of videos.scm shouldn't by itself rule it out.


After my search through videos.scm, I see that github and sourceforge have similar url endings.
/${name}-${version}.ending, with the ocassional ${name}_${version}.ending or ${version}.ending. However, every single package ended in ${version}.ending, no matter where it was from.

Have you found the secret of updates?

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]