all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* [PATCH] draft addition of github updater
@ 2015-11-15  0:32 Ben Woodcroft
  2015-11-16  9:15 ` Ludovic Courtès
  2015-11-16 14:14 ` Efraim Flashner
  0 siblings, 2 replies; 13+ messages in thread
From: Ben Woodcroft @ 2015-11-15  0:32 UTC (permalink / raw)
  To: guix-devel@gnu.org

[-- Attachment #1: Type: text/plain, Size: 2272 bytes --]

Hi,

Importing from GitHub seems very non-trivial, but can we update? There's 
a number of issues with the attached patch but so far out of the 171 
github package in guix, it recognizes 101, and 17 are detected as out of 
date (see below).

I have two questions:

1. Some guess-work is required to get between the version as it is 
defined in guix, and that presented in the github json, where only the 
"tag_name" is available. Is it OK to be a little speculative in this 
conversion e.g. "v1.0" => "1.0"?

2. For mass-updates, it fails when it hits the abuse limit on github (60 
api requests per hour). This can be overcome by authenticating with an 
access token, but I don't think that token should go in the git 
repository. So I'm after some guidance on the best way of the user 
providing a token to the updater (or some other workaround).

Thanks,
ben

gnu/packages/xml.scm:378:13: pugixml would be upgraded from 1.6 to 1.7
gnu/packages/web.scm:685:6: sassc would be upgraded from 3.2.5 to 3.3.2
gnu/packages/video.scm:693:13: mpv would be upgraded from 0.11.0 to 0.13.0
gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to 
4.02.0+1
gnu/packages/ninja.scm:31:13: ninja would be upgraded from 1.5.3 to 1.6.0
gnu/packages/jrnl.scm:30:13: jrnl would be upgraded from 1.8.4 to 1.9.7
gnu/packages/gl.scm:453:13: libepoxy would be upgraded from 1.2 to 1.3.1
gnu/packages/game-development.scm:123:13: tiled would be upgraded from 
0.13.1 to 0.14.2
gnu/packages/fontutils.scm:285:13: libuninameslist would be upgraded 
from 0.4.20140731 to 0.5.20150701
gnu/packages/engineering.scm:58:13: librecad would be upgraded from 
2.0.6-rc to 2.0.8
gnu/packages/bioinformatics.scm:1530:13: htsjdk would be upgraded from 
1.129 to 1.140
gnu/packages/bioinformatics.scm:613:13: bowtie would be upgraded from 
2.2.4 to 2.2.6
gnu/packages/bioinformatics.scm:2925:13: vsearch would be upgraded from 
1.4.1 to 1.9.1
gnu/packages/bioinformatics.scm:1360:13: grit would be upgraded from 
2.0.2 to 2.0.5beta4
gnu/packages/bioinformatics.scm:758:13: clipper would be upgraded from 
0.3.0 to 1.0
gnu/packages/bioinformatics.scm:207:13: bedtools would be upgraded from 
2.24.0 to 2.25.0
gnu/packages/bioinformatics.scm:1610:13: idr would be upgraded from 
2.0.0 to 2.0.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-import-Add-github-updater.patch --]
[-- Type: text/x-patch; name="0001-import-Add-github-updater.patch", Size: 6473 bytes --]

From 8072ee3ac66a71b74e79af4047d4f03bac9fed48 Mon Sep 17 00:00:00 2001
From: Ben Woodcroft <donttrustben@gmail.com>
Date: Sun, 15 Nov 2015 10:18:05 +1000
Subject: [PATCH] import: Add github-updater.

* guix/import/github.scm: New file.
* guix/scripts/refresh.scm (%updaters): Add %GITHUB-UPDATER
---
 guix/import/github.scm   | 118 +++++++++++++++++++++++++++++++++++++++++++++++
 guix/scripts/refresh.scm |   4 +-
 2 files changed, 121 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/github.scm

diff --git a/guix/import/github.scm b/guix/import/github.scm
new file mode 100644
index 0000000..2fecb0a
--- /dev/null
+++ b/guix/import/github.scm
@@ -0,0 +1,118 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2015 Ben Woodcroft <donttrustben@gmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;; TODO: Are all of these imports used?
+(define-module (guix import github)
+  #:use-module (ice-9 binary-ports)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 pretty-print)
+  #:use-module (ice-9 regex)
+  #:use-module ((ice-9 rdelim) #:select (read-line))
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (rnrs bytevectors)
+  #:use-module (json)
+  #:use-module (web uri)
+  #:use-module (guix ui)
+  #:use-module (guix utils)
+  #:use-module ((guix download) #:prefix download:)
+  #:use-module (guix import utils)
+  #:use-module (guix import json)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module (gnu packages)
+  #:export (%github-updater))
+
+(define (json-fetch* url)
+  "Return a list/hash representation of the JSON resource URL, or #f on
+failure."
+  ;; TODO: make silent
+  (call-with-temporary-output-file
+   (lambda (temp port)
+     (and (url-fetch url temp)
+          (call-with-input-file temp json->scm)))))
+
+(define (github-package? package)
+  "Return true if PACKAGE is a package from GitHub."
+
+  ;; TODO: currently requires the standard "v1.0" or "1.0" style tag names
+  ;; TODO: currently only accepts .tar.gz downloads
+  ;; TODO: should also accept alternative download URLs of style like
+  ;; https://github.com/libical/libical/releases/download/v1.0.1/libical-1.0.1.tar.gz
+  (define (github-url? url)
+    (and
+     (string-prefix? "https://github.com/" url)
+     (or
+      (string-suffix?
+       (string-append "/archive/v" (package-version package) ".tar.gz") url)
+      (string-suffix?
+       (string-append "/archive/" (package-version package) ".tar.gz") url))))
+
+  (let ((source-url (and=> (package-source package) origin-uri))
+        (fetch-method (and=> (package-source package) origin-method)))
+    (display (list "testing" source-url))
+    (display "\n")
+    (and (eq? fetch-method download:url-fetch)
+         (match source-url
+           ((? string?)
+            (github-url? source-url))
+           ((source-url ...)
+            (any github-url? source-url))))))
+
+(define (github-user-slash-repository url)
+  "Return a string e.g. arq5x/bedtools2 of the owner and the name of the
+repository separated by a forward slash, from a string URL of the form
+'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz'"
+  (let ((splits (string-split url #\/)))
+    (string-append (list-ref splits 3) "/" (list-ref splits 4))))
+
+(define (latest-released-version url)
+  "Return a string of the newest released version name given a string URL like
+'https://github.com/arq5x/bedtools2/archive/v2.24.0.tar.gz', or #f if there
+is no releases"
+  ;; TODO: don't return pre-release versions, can detect this from JSON field
+  ;; 'prerelease'
+  (let ((json (json-fetch*
+               (string-append "https://api.github.com/repos/"
+                              (github-user-slash-repository url)
+                              "/releases"
+                              ;;"?access_token=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+                              ))))
+        (if (eq? (length json) 0) #f
+            (let ((tag (assoc-ref (hash-table->alist (first json)) "tag_name")))
+              (if (eq? (string-ref tag 0) #\v)
+                  (substring tag 1) tag)))))
+
+(define (latest-release guix-package)
+  "Return an <upstream-source> for the latest release of GUIX-PACKAGE."
+  (let* ((pkg (specification->package guix-package))
+         (source-uri (origin-uri (package-source pkg)))
+         (version (latest-released-version source-uri)))
+    (if version
+        (upstream-source
+         (package guix-package)
+         (version version)
+         (urls (list source-uri)))
+        #f)))
+
+(define %github-updater
+  (upstream-updater
+   (name 'github)
+   (description "Updater for GitHub packages")
+   (pred github-package?)
+   (latest latest-release)))
diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm
index 3161aac..f9ac0ed 100644
--- a/guix/scripts/refresh.scm
+++ b/guix/scripts/refresh.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2013 Nikita Karetnikov <nikita@karetnikov.org>
 ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2015 Alex Kost <alezost@gmail.com>
+;;; Copyright © 2015 Ben Woodcroft <donttrustben@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -184,7 +185,8 @@ unavailable optional dependencies such as Guile-JSON."
   (list-updaters %gnu-updater
                  %elpa-updater
                  %cran-updater
-                 ((guix import pypi) => %pypi-updater)))
+                 ((guix import pypi) => %pypi-updater)
+                 ((guix import github) => %github-updater)))
 
 (define (lookup-updater name)
   "Return the updater called NAME."
-- 
2.5.0


^ permalink raw reply related	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2016-04-15  8:42 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-15  0:32 [PATCH] draft addition of github updater Ben Woodcroft
2015-11-16  9:15 ` Ludovic Courtès
2015-12-20  0:42   ` Ben Woodcroft
2016-01-03 20:46     ` Ludovic Courtès
2016-01-05 16:05       ` Ricardo Wurmus
2016-04-15  8:42         ` Updaters now receive package objects Ludovic Courtès
2016-02-21  3:13       ` [PATCH] draft addition of github updater Ben Woodcroft
2016-02-21  3:17         ` Ben Woodcroft
2016-02-23 13:22         ` Ludovic Courtès
2016-02-27  3:14           ` Ben Woodcroft
2016-02-27 11:55             ` Ricardo Wurmus
2016-02-28 14:35               ` Ludovic Courtès
2015-11-16 14:14 ` Efraim Flashner

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.