From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Woodcroft Subject: [PATCH] draft addition of github updater Date: Sun, 15 Nov 2015 10:32:40 +1000 Message-ID: <5647D2A8.8040603@uq.edu.au> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------050205060107030600030705" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:58536) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZxlFW-0002Hz-JK for guix-devel@gnu.org; Sat, 14 Nov 2015 19:32:56 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZxlFS-0005lC-8m for guix-devel@gnu.org; Sat, 14 Nov 2015 19:32:54 -0500 Received: from mailhub1.soe.uq.edu.au ([130.102.132.208]:60600 helo=newmailhub.uq.edu.au) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZxlFR-0005jt-N5 for guix-devel@gnu.org; Sat, 14 Nov 2015 19:32:50 -0500 Received: from smtp2.soe.uq.edu.au (smtp2.soe.uq.edu.au [10.138.113.41]) by newmailhub.uq.edu.au (8.14.5/8.14.5) with ESMTP id tAF0WgfQ033323 for ; Sun, 15 Nov 2015 10:32:43 +1000 Received: from [192.168.1.105] ([103.25.181.216]) (authenticated bits=0) by smtp2.soe.uq.edu.au (8.14.5/8.14.5) with ESMTP id tAF0WfRc023758 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Sun, 15 Nov 2015 10:32:42 +1000 List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: "guix-devel@gnu.org" This is a multi-part message in MIME format. --------------050205060107030600030705 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, Importing from GitHub seems very non-trivial, but can we update? There's a number of issues with the attached patch but so far out of the 171 github package in guix, it recognizes 101, and 17 are detected as out of date (see below). I have two questions: 1. Some guess-work is required to get between the version as it is defined in guix, and that presented in the github json, where only the "tag_name" is available. Is it OK to be a little speculative in this conversion e.g. "v1.0" => "1.0"? 2. For mass-updates, it fails when it hits the abuse limit on github (60 api requests per hour). This can be overcome by authenticating with an access token, but I don't think that token should go in the git repository. So I'm after some guidance on the best way of the user providing a token to the updater (or some other workaround). Thanks, ben gnu/packages/xml.scm:378:13: pugixml would be upgraded from 1.6 to 1.7 gnu/packages/web.scm:685:6: sassc would be upgraded from 3.2.5 to 3.3.2 gnu/packages/video.scm:693:13: mpv would be upgraded from 0.11.0 to 0.13.0 gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to 4.02.0+1 gnu/packages/ninja.scm:31:13: ninja would be upgraded from 1.5.3 to 1.6.0 gnu/packages/jrnl.scm:30:13: jrnl would be upgraded from 1.8.4 to 1.9.7 gnu/packages/gl.scm:453:13: libepoxy would be upgraded from 1.2 to 1.3.1 gnu/packages/game-development.scm:123:13: tiled would be upgraded from 0.13.1 to 0.14.2 gnu/packages/fontutils.scm:285:13: libuninameslist would be upgraded from 0.4.20140731 to 0.5.20150701 gnu/packages/engineering.scm:58:13: librecad would be upgraded from 2.0.6-rc to 2.0.8 gnu/packages/bioinformatics.scm:1530:13: htsjdk would be upgraded from 1.129 to 1.140 gnu/packages/bioinformatics.scm:613:13: bowtie would be upgraded from 2.2.4 to 2.2.6 gnu/packages/bioinformatics.scm:2925:13: vsearch would be upgraded from 1.4.1 to 1.9.1 gnu/packages/bioinformatics.scm:1360:13: grit would be upgraded from 2.0.2 to 2.0.5beta4 gnu/packages/bioinformatics.scm:758:13: clipper would be upgraded from 0.3.0 to 1.0 gnu/packages/bioinformatics.scm:207:13: bedtools would be upgraded from 2.24.0 to 2.25.0 gnu/packages/bioinformatics.scm:1610:13: idr would be upgraded from 2.0.0 to 2.0.2 --------------050205060107030600030705 Content-Type: text/x-patch; name="0001-import-Add-github-updater.patch" Content-Disposition: attachment; filename="0001-import-Add-github-updater.patch" Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by newmailhub.uq.edu.au id tAF0WgfQ033323 >From 8072ee3ac66a71b74e79af4047d4f03bac9fed48 Mon Sep 17 00:00:00 2001 From: Ben Woodcroft Date: Sun, 15 Nov 2015 10:18:05 +1000 Subject: [PATCH] import: Add github-updater. * guix/import/github.scm: New file. * guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER --- guix/import/github.scm | 118 +++++++++++++++++++++++++++++++++++++++++= ++++++ guix/scripts/refresh.scm | 4 +- 2 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 guix/import/github.scm diff --git a/guix/import/github.scm b/guix/import/github.scm new file mode 100644 index 0000000..2fecb0a --- /dev/null +++ b/guix/import/github.scm @@ -0,0 +1,118 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2015 Ben Woodcroft +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (a= t +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +;; TODO: Are all of these imports used? +(define-module (guix import github) + #:use-module (ice-9 binary-ports) + #:use-module (ice-9 match) + #:use-module (ice-9 pretty-print) + #:use-module (ice-9 regex) + #:use-module ((ice-9 rdelim) #:select (read-line)) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) + #:use-module (rnrs bytevectors) + #:use-module (json) + #:use-module (web uri) + #:use-module (guix ui) + #:use-module (guix utils) + #:use-module ((guix download) #:prefix download:) + #:use-module (guix import utils) + #:use-module (guix import json) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (gnu packages) + #:export (%github-updater)) + +(define (json-fetch* url) + "Return a list/hash representation of the JSON resource URL, or #f on +failure." + ;; TODO: make silent + (call-with-temporary-output-file + (lambda (temp port) + (and (url-fetch url temp) + (call-with-input-file temp json->scm))))) + +(define (github-package? package) + "Return true if PACKAGE is a package from GitHub." + + ;; TODO: currently requires the standard "v1.0" or "1.0" style tag nam= es + ;; TODO: currently only accepts .tar.gz downloads + ;; TODO: should also accept alternative download URLs of style like + ;; https://github.com/libical/libical/releases/download/v1.0.1/libical= -1.0.1.tar.gz + (define (github-url? url) + (and + (string-prefix? "https://github.com/" url) + (or + (string-suffix? + (string-append "/archive/v" (package-version package) ".tar.gz") = url) + (string-suffix? + (string-append "/archive/" (package-version package) ".tar.gz") u= rl)))) + + (let ((source-url (and=3D> (package-source package) origin-uri)) + (fetch-method (and=3D> (package-source package) origin-method))) + (display (list "testing" source-url)) + (display "\n") + (and (eq? fetch-method download:url-fetch) + (match source-url + ((? string?) + (github-url? source-url)) + ((source-url ...) + (any github-url? source-url)))))) + +(define (github-user-slash-repository url) + "Return a string e.g. arq5x/bedtools2 of the owner and the name of the +repository separated by a forward slash, from a string URL of the form +'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'" + (let ((splits (string-split url #\/))) + (string-append (list-ref splits 3) "/" (list-ref splits 4)))) + +(define (latest-released-version url) + "Return a string of the newest released version name given a string UR= L like +'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz', or #f if th= ere +is no releases" + ;; TODO: don't return pre-release versions, can detect this from JSON = field + ;; 'prerelease' + (let ((json (json-fetch* + (string-append "https://api.github.com/repos/" + (github-user-slash-repository url) + "/releases" + ;;"?access_token=3Daaaaaaaaaaaaaaaaaaaaaaa= aaaaaaaaaaaaaaaaa" + )))) + (if (eq? (length json) 0) #f + (let ((tag (assoc-ref (hash-table->alist (first json)) "tag_= name"))) + (if (eq? (string-ref tag 0) #\v) + (substring tag 1) tag))))) + +(define (latest-release guix-package) + "Return an for the latest release of GUIX-PACKAGE." + (let* ((pkg (specification->package guix-package)) + (source-uri (origin-uri (package-source pkg))) + (version (latest-released-version source-uri))) + (if version + (upstream-source + (package guix-package) + (version version) + (urls (list source-uri))) + #f))) + +(define %github-updater + (upstream-updater + (name 'github) + (description "Updater for GitHub packages") + (pred github-package?) + (latest latest-release))) diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm index 3161aac..f9ac0ed 100644 --- a/guix/scripts/refresh.scm +++ b/guix/scripts/refresh.scm @@ -3,6 +3,7 @@ ;;; Copyright =C2=A9 2013 Nikita Karetnikov ;;; Copyright =C2=A9 2014 Eric Bavier ;;; Copyright =C2=A9 2015 Alex Kost +;;; Copyright =C2=A9 2015 Ben Woodcroft ;;; ;;; This file is part of GNU Guix. ;;; @@ -184,7 +185,8 @@ unavailable optional dependencies such as Guile-JSON.= " (list-updaters %gnu-updater %elpa-updater %cran-updater - ((guix import pypi) =3D> %pypi-updater))) + ((guix import pypi) =3D> %pypi-updater) + ((guix import github) =3D> %github-updater))) =20 (define (lookup-updater name) "Return the updater called NAME." --=20 2.5.0 --------------050205060107030600030705--