From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Woodcroft Subject: Re: [PATCH] draft addition of github updater Date: Sun, 20 Dec 2015 10:42:22 +1000 Message-ID: <5675F96E.4090609@uq.edu.au> References: <5647D2A8.8040603@uq.edu.au> <87h9kmb8zs.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------030602050100070705020400" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:56319) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aAS5A-0004aj-Tp for guix-devel@gnu.org; Sat, 19 Dec 2015 19:42:43 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aAS57-0005Bx-J0 for guix-devel@gnu.org; Sat, 19 Dec 2015 19:42:40 -0500 In-Reply-To: <87h9kmb8zs.fsf@gnu.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: =?UTF-8?Q?Ludovic_Court=c3=a8s?= Cc: "guix-devel@gnu.org" This is a multi-part message in MIME format. --------------030602050100070705020400 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by newmailhub.uq.edu.au id tBK0gPnB036044 Thanks for the encouraging words. Here's the next revision. On 16/11/15 19:15, Ludovic Court=C3=A8s wrote: > Hi! > > Ben Woodcroft skribis: > >> Importing from GitHub seems very non-trivial, but can we update? >> There's a number of issues with the attached patch but so far out of >> the 171 github package in guix, it recognizes 101, and 17 are detected >> as out of date (see below). It seems I miscounted before, but now it is 129 of 146 github "release"=20 packages recognised with 28 suggesting an update - see the end of email=20 for details. There is one false positive: gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to=20 4.02.0+1 This happens because the newer versions were not made as official=20 releases just tags, so the newer versions are omitted from the API=20 response, plus there's the odd version numbering scheme. Guix is up to=20 date. >> I have two questions: >> >> 1. Some guess-work is required to get between the version as it is >> defined in guix, and that presented in the github json, where only the >> "tag_name" is available. Is it OK to be a little speculative in this >> conversion e.g. "v1.0" =3D> "1.0"? > I guess so. What I would do is do that conversion when the tag matches > =E2=80=9C^v[0-9]=E2=80=9D and leave the tag as-is in other cases. WDYT= ? > > We can always add more heuristics later if we find that there=E2=80=99s= another > widely-used convention for tag names. Most seem to follow those few conventions, but there's still repos that=20 decided to be different e.g. https://github.com/vapoursynth/vapoursynth/archive/R28.tar.gz https://github.com/synergy/synergy/archive/v1.7.4-stable.tar.gz Having gotten this far, I wonder if I've gone about it backwards.=20 Currently the updater works by asserting it is a refreshable package by=20 interrogating the source URI only. But it might be easier to determine=20 this with an API response on hand, by matching the current release=20 version number to a tag. Then if we assume the same transformation of=20 tag to version holds in the newest release, the reverse transformation=20 can be used on the newest tag to convert it back into a version number.=20 By transformation I mean addition of [a-z\.\-] characters before and=20 after the version number. This is easier because guesswork is only=20 needed to convert between the tag and version number, without reference=20 to a URI. This means more work for me, is it a good idea? As I understand it would=20 involve returning #t more often from "github-package?". If #f is=20 returned by an updater, do the updaters further down the chain get a=20 bite at the cherry too? It doesn't matter for now since the github=20 updater is last, but it might in the future. >> 2. For mass-updates, it fails when it hits the abuse limit on github >> (60 api requests per hour). This can be overcome by authenticating >> with an access token, but I don't think that token should go in the >> git repository. So I'm after some guidance on the best way of the user >> providing a token to the updater (or some other workaround). > Argh, that=E2=80=99s annoying. How does it fail exactly? What=E2=80=99= s the impact on > the behavior of =E2=80=98guix refresh=E2=80=99? I didn't investigate thoroughly, but I believe it either gives a 403 or=20 a more descriptive json string, dependent on the user-agent. I added some words and errored out when json-fetch* returns #f. This was=20 potentially a little lazy on my part as it might be better to detect the=20 403 error as distinct from errors of other kinds, but it wasn't=20 immediately obvious to me how to do this without going too deep into the=20 fetching functions and/or duplicating code. WDYT? > > I guess (guix import github) could contain something like: > > (define %github-token > ;; Token to be passed to Github.com to avoid the 60-request per ho= ur > ;; limit, or #f. > (make-parameter (getenv "GUIX_GITHUB_TOKEN"))) > > and we=E2=80=99d need to document that, or maybe write a message hintin= g at it > when we know the limit has been reached. > > WDYT? Seems we were all thinking the same thing - I've integrated this. Should=20 we check that the token matches ^[0-9a-f]+$ for security and UI? > I was thinking we could have a generic Git updater that would look for=20 > available tags upstream. I wonder how efficient that would be compared=20 > to using the GitHub-specific API, and if there would be other=20 > differences. What are your thoughts on this? This sounds like an excellent idea, but I was unable to find any way to=20 fetch tags without a clone first. A clone could take a long time and a=20 lot of bandwidth I would imagine. Also there's no way to discern regular=20 releases from pre-releases I don't think. It is a bit unclear to me how=20 conservative these updaters should be, are tags sufficiently synonymous=20 with releases so as to be reported by refresh? There's a number of github repos packaged that refer to git commits=20 directly too, these are ignored by the current updater but might benefit=20 from this approach (as well as non-github git repos of course). Thanks, ben gnu/packages/xml.scm:380:13: pugixml would be upgraded from 1.6 to 1.7 gnu/packages/web.scm:353:13: libpsl would be upgraded from 0.7.1 to 0.11.= 0 gnu/packages/web.scm:685:6: sassc would be upgraded from 3.2.5 to 3.3.2 gnu/packages/version-control.scm:934:13: findnewest would be upgraded=20 from 0.2 to 0.3 gnu/packages/telephony.scm:192:13: libsrtp would be upgraded from 1.5.2=20 to 1.5.3 gnu/packages/ruby.scm:2373:13: ruby-sanitize would be upgraded from=20 4.0.0 to 4.0.1 gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to=20 4.02.0+1 gnu/packages/ninja.scm:31:13: ninja would be upgraded from 1.5.3 to 1.6.0 gnu/packages/maths.scm:1855:13: dealii would be upgraded from 8.2.1 to 8.= 3.0 gnu/packages/jrnl.scm:30:13: jrnl would be upgraded from 1.8.4 to 1.9.7 gnu/packages/gl.scm:453:13: libepoxy would be upgraded from 1.2 to 1.3.1 gnu/packages/game-development.scm:125:13: tiled would be upgraded from=20 0.13.1 to 0.14.2 gnu/packages/fontutils.scm:285:13: libuninameslist would be upgraded=20 from 0.4.20140731 to 0.5.20150701 gnu/packages/engineering.scm:58:13: librecad would be upgraded from=20 2.0.6-rc to 2.0.8 gnu/packages/emacs.scm:436:13: haskell-mode would be upgraded from=20 13.14.2 to 13.16 gnu/packages/conky.scm:35:13: conky would be upgraded from 1.10.0 to 1.10= .1 gnu/packages/bioinformatics.scm:974:13: deeptools would be upgraded from=20 1.5.11 to 1.5.12 gnu/packages/bioinformatics.scm:1532:13: htsjdk would be upgraded from=20 1.129 to 2.0.1 gnu/packages/bioinformatics.scm:207:13: bedtools would be upgraded from=20 2.24.0 to 2.25.0 gnu/packages/bioinformatics.scm:1880:13: orfm would be upgraded from=20 0.4.1 to 0.5.2 gnu/packages/bioinformatics.scm:758:13: clipper would be upgraded from=20 0.3.0 to 1.0 gnu/packages/bioinformatics.scm:1612:13: idr would be upgraded from=20 2.0.0 to 2.0.2 gnu/packages/bioinformatics.scm:2592:13: preseq would be upgraded from=20 2.0 to 2.0.2 gnu/packages/bioinformatics.scm:2978:13: vsearch would be upgraded from=20 1.4.1 to 1.9.5 gnu/packages/bioinformatics.scm:1360:13: grit would be upgraded from=20 2.0.2 to 2.0.4 gnu/packages/bioinformatics.scm:1577:13: htslib would be upgraded from=20 1.2.1 to 1.3 gnu/packages/bioinformatics.scm:1013:13: diamond would be upgraded from=20 0.7.9 to 0.7.10 gnu/packages/bioinformatics.scm:613:13: bowtie would be upgraded from=20 2.2.4 to 2.2.6 --------------030602050100070705020400 Content-Type: text/x-patch; name="0001-import-Add-github-updater.patch" Content-Disposition: attachment; filename="0001-import-Add-github-updater.patch" Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by newmailhub.uq.edu.au id tBK0gPnB036044 >From a42eda6b9631cc28dfdd02d2c8bb02eabb2626b9 Mon Sep 17 00:00:00 2001 From: Ben Woodcroft Date: Sun, 15 Nov 2015 10:18:05 +1000 Subject: [PATCH] import: Add github-updater. * guix/import/github.scm: New file. * guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER. * doc/guix.texi (Invoking guix refresh): Mention it. --- doc/guix.texi | 14 ++++ guix/import/github.scm | 167 +++++++++++++++++++++++++++++++++++++++++= ++++++ guix/scripts/refresh.scm | 5 +- 3 files changed, 185 insertions(+), 1 deletion(-) create mode 100644 guix/import/github.scm diff --git a/doc/guix.texi b/doc/guix.texi index 06d70ba..f6b7368 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -16,6 +16,7 @@ Copyright @copyright{} 2013 Nikita Karetnikov@* Copyright @copyright{} 2015 Mathieu Lirzin@* Copyright @copyright{} 2014 Pierre-Antoine Rault@* Copyright @copyright{} 2015 Taylan Ulrich Bay=C4=B1rl=C4=B1/Kammer +Copyright @copyright{} 2015 Ben Woodcroft =20 Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -4354,6 +4355,16 @@ attempt is made to automatically retrieve it from = a public key server; when it's successful, the key is added to the user's keyring; otherwise, @command{guix refresh} reports an error. =20 +The @code{github} updater uses the +@uref{https://developer.github.com/v3/, GitHub API} to query for new +releases. When used repeatedly e.g. when refreshing all packages, GitHub +will eventually refuse to answer any further API requests. By default 60 +API requests per hour are allowed, and a full refresh on all GitHub +packages in Guix requires more than this. Authentication with GitHub +through the use of an API token alleviates these limits. To use an API +token, set the environment variable @code{GUIX_GITHUB_TOKEN} to a token +procured from @uref{https://github.com/settings/tokens} or otherwise. + The following options are supported: =20 @table @code @@ -4415,6 +4426,8 @@ the updater for @uref{http://elpa.gnu.org/, ELPA} p= ackages; the updater for @uref{http://cran.r-project.org/, CRAN} packages; @item pypi the updater for @uref{https://pypi.python.org, PyPI} packages. +@item github +the updater for @uref{https://github.com, GitHub} packages. @end table =20 For instance, the following commands only checks for updates of Emacs @@ -4501,6 +4514,7 @@ Use @var{host} as the OpenPGP key server when impor= ting a public key. =20 @end table =20 + @node Invoking guix lint @section Invoking @command{guix lint} The @command{guix lint} is meant to help package developers avoid common diff --git a/guix/import/github.scm b/guix/import/github.scm new file mode 100644 index 0000000..2ed477e --- /dev/null +++ b/guix/import/github.scm @@ -0,0 +1,167 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2015 Ben Woodcroft +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (a= t +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +;; TODO: Are all of these imports used? +(define-module (guix import github) + #:use-module (ice-9 match) + #:use-module (srfi srfi-1) + #:use-module (json) + #:use-module (guix utils) + #:use-module ((guix download) #:prefix download:) + #:use-module (guix import utils) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (gnu packages) + #:export (%github-updater)) + +(define (json-fetch* url) + "Return a list/hash representation of the JSON resource URL, or #f on +failure." + (call-with-output-file "/dev/null" + (lambda (null) + (with-error-to-port null + (lambda () + (call-with-temporary-output-file + (lambda (temp port) + (and (url-fetch url temp) + (call-with-input-file temp json->scm))))))))) + +;; TODO: is there some code from elsewhere in guix that can be used inst= ead of +;; redefining? +(define (find-extension url) + "Return the extension of the archive e.g. '.tar.gz' given a URL, or +false if none is recognized" + (find (lambda x (string-suffix? (first x) url)) + (list ".tar.gz" ".tar.bz2" ".tar.xz" ".zip" ".tar"))) + +(define (github-package? package) + "Return true if PACKAGE is a package from GitHub." + + (define (github-url? url) + (and + (string-prefix? "https://github.com/" url) + (let ((ext (find-extension url))) + (and ext + (or + (string-suffix? + (string-append "/archive/v" (package-version package) ext)= url) + (string-suffix? + (string-append "/archive/" (package-version package) ext) = url) + (string-suffix? + (string-append "/archive/" (package-name package) "-" + (package-version package) ext) + url) + (string-suffix? + (string-append "/releases/download/v" (package-version pac= kage) + "/" (package-name package) "-" + (package-version package) ext) + url) + (string-suffix? + (string-append "/releases/download/" (package-version pack= age) + "/" (package-name package) "-" + (package-version package) ext) + url)))))) + + (let ((source-url (and=3D> (package-source package) origin-uri)) + (fetch-method (and=3D> (package-source package) origin-method))) + (and (eq? fetch-method download:url-fetch) + (match source-url + ((? string?) + (github-url? source-url)) + ((source-url ...) + (any github-url? source-url)))))) + +(define (github-user-slash-repository url) + "Return a string e.g. arq5x/bedtools2 of the owner and the name of the +repository separated by a forward slash, from a string URL of the form +'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'" + (let ((splits (string-split url #\/))) + (string-append (list-ref splits 3) "/" (list-ref splits 4)))) + +(define %github-token + ;; Token to be passed to Github.com to avoid the 60-request per hour + ;; limit, or #f. + ;; QUESTION: is there a need to check that the token looks like a toke= n, for + ;; security, since it gets used in a fetch as is? + (make-parameter (getenv "GUIX_GITHUB_TOKEN"))) + +(define (latest-released-version url package-name) + "Return a string of the newest released version name given a string UR= L like +'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz' and the name= of +the package e.g. 'bedtools2'. Return #f if there is no releases" + (let* ((token (%github-token)) + (api-url (string-append + "https://api.github.com/repos/" + (github-user-slash-repository url) + "/releases")) + (json (json-fetch* + (if token + (string-append api-url "?access_token=3D" token) + api-url)))) + (if (eq? json #f) + (if token + (error "Error downloading release information through the Gi= tHub +API when using a GitHub token") + (error "Error downloading release information through the Gi= tHub +API. This may be fixed by using an access token and setting the environm= ent +variable GUIX_GITHUB_TOKEN, for instance one procured from +https://github.com/settings/tokens")) + (let ((proper-releases + (filter + (lambda (x) + ;; example pre-release: + ;; https://github.com/wwood/OrfM/releases/tag/v0.5.1 + ;; or an all-prerelease set + ;; https://github.com/powertab/powertabeditor/releases + (eq? (assoc-ref (hash-table->alist x) "prerelease") #f= )) + json))) + (if (eq? (length proper-releases) 0) #f ;empty releases list + (let* + ((tag (assoc-ref (hash-table->alist (first proper-rele= ases)) + "tag_name")) + (name-length (string-length package-name))) + ;; some tags include the name of the package e.g. "fdupe= s-1.51" + ;; so remove these + (if (and (< name-length (string-length tag)) + (string=3D? (string-append package-name "-") + (substring tag 0 (+ name-length 1)))) + (substring tag (+ name-length 1)) + ;; some tags start with a "v" e.g. "v0.25.0" + ;; where some are just the version number + (if (eq? (string-ref tag 0) #\v) + (substring tag 1) tag)))))))) + +(define (latest-release guix-package) + "Return an for the latest release of GUIX-PACKAGE." + (let* ((pkg (specification->package guix-package)) + (source-uri (origin-uri (package-source pkg))) + (name (package-name pkg)) + (version (latest-released-version source-uri name))) + (if version + (upstream-source + (package guix-package) + (version version) + (urls (list source-uri))) + #f))) + +(define %github-updater + (upstream-updater + (name 'github) + (description "Updater for GitHub packages") + (pred github-package?) + (latest latest-release))) diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm index a5834d1..adbcf28 100644 --- a/guix/scripts/refresh.scm +++ b/guix/scripts/refresh.scm @@ -3,6 +3,7 @@ ;;; Copyright =C2=A9 2013 Nikita Karetnikov ;;; Copyright =C2=A9 2014 Eric Bavier ;;; Copyright =C2=A9 2015 Alex Kost +;;; Copyright =C2=A9 2015 Ben Woodcroft ;;; ;;; This file is part of GNU Guix. ;;; @@ -34,6 +35,7 @@ #:select (%gnu-updater %gnome-updater)) #:use-module (guix import elpa) #:use-module (guix import cran) + #:use-module (guix import github) #:use-module (guix gnupg) #:use-module (gnu packages) #:use-module ((gnu packages commencement) #:select (%final-inputs)) @@ -195,7 +197,8 @@ unavailable optional dependencies such as Guile-JSON.= " %gnome-updater %elpa-updater %cran-updater - ((guix import pypi) =3D> %pypi-updater))) + ((guix import pypi) =3D> %pypi-updater) + %github-updater)) =20 (define (lookup-updater name) "Return the updater called NAME." --=20 2.5.0 --------------030602050100070705020400--