From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id EBakNbREMmHMUwEAgWs5BA (envelope-from ) for ; Fri, 03 Sep 2021 17:52:20 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id gGB1MbREMmGjdwAAB5/wlQ (envelope-from ) for ; Fri, 03 Sep 2021 15:52:20 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D687E24BAC for ; Fri, 3 Sep 2021 17:52:19 +0200 (CEST) Received: from localhost ([::1]:51314 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mMBUI-0006wC-QQ for larch@yhetil.org; Fri, 03 Sep 2021 11:52:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56764) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mMBU3-0006uT-3O for guix-patches@gnu.org; Fri, 03 Sep 2021 11:52:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:34242) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mMBU2-0005cf-Rk for guix-patches@gnu.org; Fri, 03 Sep 2021 11:52:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mMBU2-0007y3-I2 for guix-patches@gnu.org; Fri, 03 Sep 2021 11:52:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50359] [PATCH] import: Add 'generic-git' updater. Resent-From: Xinglu Chen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 03 Sep 2021 15:52:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 50359 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 50359@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.163068428430581 (code B ref -1); Fri, 03 Sep 2021 15:52:02 +0000 Received: (at submit) by debbugs.gnu.org; 3 Sep 2021 15:51:24 +0000 Received: from localhost ([127.0.0.1]:45788 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMBTP-0007xA-Ar for submit@debbugs.gnu.org; Fri, 03 Sep 2021 11:51:23 -0400 Received: from lists.gnu.org ([209.51.188.17]:53982) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMBTN-0007x3-TK for submit@debbugs.gnu.org; Fri, 03 Sep 2021 11:51:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56522) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mMBTM-0006Q2-VL for guix-patches@gnu.org; Fri, 03 Sep 2021 11:51:21 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:58080 helo=mail.yoctocell.xyz) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mMBTJ-0004iq-L0 for guix-patches@gnu.org; Fri, 03 Sep 2021 11:51:20 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1630684266; bh=cdtrGN9Y8W0hGMyXnMzGssKIL6kIknZ9jKKMhgeCheg=; h=From:To:Subject:Date; b=FwV1KD+rP9LxCUJ7+HnDaOh1f8ePrHqNIdQtltOERgyZRm2C2o9vH9D84sO4ClQ6H J4NWCChfgbAgLHQCFKrBBYNQ1AjtlVgSMIUeUioc93ezzwQ77P5S7U5AtaX0QvpviM QBEz3uekTyR2cezX71GwZRBGeVDi2K7ubxuOgayw= Message-Id: Date: Fri, 03 Sep 2021 17:50:56 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=87.96.130.155; envelope-from=public@yoctocell.xyz; helo=mail.yoctocell.xyz X-Spam_score_int: 30 X-Spam_score: 3.0 X-Spam_bar: +++ X-Spam_report: (3.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FROM_SUSPICIOUS_NTLD=0.5, FROM_SUSPICIOUS_NTLD_FP=1.598, PDS_OTHER_BAD_TLD=1.999, RDNS_DYNAMIC=0.982, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1630684340; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=dmTCETeDbKPmFuIdC/ZikZE7GNhIsJSlfsdzoBQI3m4=; b=ovK+HAKihS2iRfWovLUz/t7P382a2Wo+rgioVhk/eZ3os19iVa1opBhyJE2o423BlSxNJ2 qgo7gnTLYkGvPGCaNtbsXmKtD4TA2dHidTK7G+w0uxN8ZvmjmO8I4gJkPV47Qn2Op0LVYB xR67pXrNr550IYbwUm2FRWcBh2qv2D72UxKPjJLCXLX1b87OixkJua2Jy4xEaRueVgXlJB LuZNdHIsBp3TQ736ZIWmaE8pAJoOxGODtSdiNG3aTppubr2HFjcqkPFZyB7zNCGs0h08ZH vHlqaIRP614we2xnmFXNsUSc6TMgz+jkrFOz8Ug/bFkbVNm1nzsWLBLoPcIw6g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1630684340; a=rsa-sha256; cv=none; b=ew2wCoKwwvfWWSkKPJuiKRnZQ6ucOXrJ4qckib/vBWVtbWFf9iFTgfPbVeDLMGDdlng3jE 6mjFA3qsV4gIqDBhjL+jXXVNvHalt7cSoyD+nDhAbg7UsGUE7wWoluOoItmjEpwimTMO/U 6DuS+29qrHC5O2+gBKqeQYy7hByVApFUm1EaXlXiORMLQMrLpx9zuOuC9BHFY1IufGMOrE vfXkVQf8eZoBO1r5ah9EkWYGZg+v8Zi3y+jCKhrNBHL0iecD+Da95F7P7aodmJGBMq9EUs /mWQbX4ck1XTbjPZ3nH2YQi3j/a+qSEUjEnqRJIMP24YWZRY2X4ebcOKQsRhhQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=yoctocell.xyz header.s=mail header.b=FwV1KD+r; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -0.32 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=yoctocell.xyz header.s=mail header.b=FwV1KD+r; dmarc=fail reason="SPF not aligned (relaxed)" header.from=yoctocell.xyz (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: D687E24BAC X-Spam-Score: -0.32 X-Migadu-Scanner: scn1.migadu.com X-TUID: 7FIjm6+SH8Bx * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document it. * Makefile.am (MODULES): Register it. --- This patch adds a new =E2=80=98generic-git=E2=80=99 updater which can check= for new tags for package hosted on Git repos. However, it cannot download Git repos and update the package definitions, i.e. =E2=80=98guix refresh -u=E2=80=99.= There is a pending patch that would add this feature though[1]. =E2=80=98guix refresh -L=E2=80=99 now reports Available updaters: [=E2=80=A6] 94.5% of the packages are covered by these updaters. We are getting close to 100% :-) See it in action! --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix refresh harmonist scdoc gmnisrv gnu/packages/web.scm:7931:4: warning: no tags were found for package `gmnis= rv' gnu/packages/web.scm:7931:4: warning: 'generic-git' updater failed to deter= mine available releases for gmnisrv gnu/packages/man.scm:339:12: scdoc would be upgraded from 1.10.1 to 1.11.1 gnu/packages/games.scm:9433:2: warning: failed to fetch Git repository for = package `harmonist' gnu/packages/games.scm:9433:2: warning: 'generic-git' updater failed to det= ermine available releases for harmonist --8<---------------cut here---------------end--------------->8--- =20=20 [1]: Makefile.am | 1 + doc/guix.texi | 27 ++++++ guix/import/git.scm | 223 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 251 insertions(+) create mode 100644 guix/import/git.scm diff --git a/Makefile.am b/Makefile.am index 3c79760734..c4d3a456b1 100644 --- a/Makefile.am +++ b/Makefile.am @@ -254,6 +254,7 @@ MODULES =3D \ guix/import/egg.scm \ guix/import/elpa.scm \ guix/import/gem.scm \ + guix/import/git.scm \ guix/import/github.scm \ guix/import/gnome.scm \ guix/import/gnu.scm \ diff --git a/doc/guix.texi b/doc/guix.texi index 36a0c7f5ec..26afb1607a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, Launc= hpad} packages. @item generic-html a generic updater that crawls the HTML page where the source tarball of the package is hosted, when applicable. +@item generic-git +a generic updater for packages hosted on Git repositories. It tries to +be smart about parsing Git tag names, but if it is not able to parse the +tag name and compare tags correctly, users can define the following +properties for a package. + +@itemize +@item @code{tag-prefix}: a regular expression for matching a prefix of +the tag name. + +@item @code{tag-suffix}: a regular expression for matching a suffix of +the tag name. + +@item @code{tag-version-delimiter}: a string used as the delimiter in +the tag name for separating the numbers of the version. +@end itemize + +@lisp +(package + (name "foo") + ;; ... + (properties + '((tag-prefix . "^release0-") + (tag-suffix . "[a-z]?$") + (tag-version-delimiter . ":")))) +@end lisp=20=20=20=20=20=20 + @end table =20 For instance, the following command only checks for updates of Emacs diff --git a/guix/import/git.scm b/guix/import/git.scm new file mode 100644 index 0000000000..9a654c1972 --- /dev/null +++ b/guix/import/git.scm @@ -0,0 +1,223 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2021 Xinglu Chen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import git) + #:use-module (git) + #:use-module (guix build utils) + #:use-module (guix diagnostics) + #:use-module (guix git) + #:use-module (guix git-download) + #:use-module (guix i18n) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-28) + #:use-module (srfi srfi-34) + #:use-module (srfi srfi-35) + #:use-module (srfi srfi-71) + #:export (%generic-git-updater)) + +;;; Commentary: +;;; +;;; This module provides a generic package updater for packages hosted on = Git +;;; repositories. +;;; +;;; It tries to be smart about tag names, but if it is not automatically a= ble +;;; to parse the tag names correctly, users can set the `tag-prefix', +;;; `tag-suffix' and `tag-version-delimiter' properties of the package to = make +;;; the updater parse the Git tag name correctly. +;;; +;;; Code: + +;;; Errors & warnings + +(define-condition-type &git-tag-error &error + git-tag-error? + (kind git-tag-error-kind)) + +(define (git-tag-error kind) + (raise (condition (&message (message (format "bad `~a' property"))) + (&git-tag-error + (kind kind))))) + +(define (git-tag-warning package c) + (warning (package-location package) + (G_ "~a for package `~a'~%") + (condition-message c) + (package-name package))) + +(define-condition-type &git-no-tags-error &error + git-no-tags-error?) + +(define (git-no-tags-error) + (raise (condition (&message (message "no tags were found")) + (&git-no-tags-error)))) + +(define (git-no-tags-warning package c) + (warning (package-location package) + (G_ "~a for package `~a'~%") + (condition-message c) + (package-name package))) + +(define (git-fetch-warning package) + (warning (package-location package) + (G_ "failed to fetch Git repository for package `~a'~%") + (package-name package))) + + +;;; Helper functions + +(define (string-split* str delim) + "Like `string-split', but DELIM is a string instead of a +char-set." + (filter (lambda (str) (not (equal? str ""))) + (string-split str (string->char-set delim)))) + +(define* (get-version package tag #:key prefix suffix delim) + (define delim* (if delim delim ".")) + (define prefix-regexp "^[^0-9]*") + (define suffix-regexp (string-append "[^0-9" (regexp-quote delim*) "]*$"= )) + (define delim-regexp (string-append "^[0-9]+" (regexp-quote delim*) "[0-= 9]+")) + + (define no-prefix + (let ((match (string-match (or prefix prefix-regexp) tag))) + (if match + (regexp-substitute #f match 'post) + (git-tag-error 'tag-prefix)))) + + (define no-suffix + (let ((match (string-match (or suffix suffix-regexp) no-prefix))) + (if match + (regexp-substitute #f match 'pre) + (git-tag-error 'tag-suffix)))) + + (define no-delims + (if (string-match delim-regexp no-suffix) + (string-split* no-suffix delim*) + (git-tag-error 'tag-version-delimiter))) + + (string-join no-delims ".")) + +(define (sort-tags tags) + "Sort TAGS, a list if Git tags, such that the latest tag is the last ele= ment." + (sort tags (lambda (a b) + (eq? (version-compare a b) '<)))) + + +;;; Updater + +(define (get-remote url git-uri) + "Given a URL and GIT-URI, a record, return the ``origin'= ' remote." + (let* ((checkout (update-cached-checkout url + #:recursive? + (git-reference-recursive? git-u= ri))) + (repository (repository-open checkout))) + (remote-lookup repository "origin"))) + +(define (get-latest-tag remote) + "Given a Git REMOTE, return that latest tag available." + (remote-connect remote) + + (define tags + (sort-tags + (map (lambda (tag) + (string-drop tag (string-length "refs/tags/"))) + (filter (lambda (ref) + ;; Every tag has two refs: + ;; + ;; * refs/tags/1.2.3^{} + ;; * refs/tags/1.2.3 + ;; + ;; remove the one with the trailing ^{} + (and (not (string-suffix? "^{}" ref)) + (string-prefix? "refs/tags/" ref))) + (map (lambda (remote-head) + (remote-head-name remote-head)) + (remote-ls remote)))))) + + (remote-disconnect remote) + + (if (null? tags) + (git-no-tags-error) + (last tags))) + +(define (latest-git-tag-version package tag-prefix tag-suffix + tag-version-delimiter) + "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, and TAG-VERSION-DELIMITER +properties of PACKAGE, returns the latest version of PACKAGE." + (guard (c ((eq? (exception-kind c) 'git-error) + (git-fetch-warning package) + #f) + ((git-tag-error? c) + (git-tag-warning package c) + #f) + ((git-no-tags-error? c) + (git-no-tags-warning package c) + #f)) + (let* ((source (package-source package)) + (git-uri (origin-uri source)) + (url (git-reference-url (origin-uri source))) + (remote (get-remote url git-uri)) + (latest-tag (get-latest-tag remote))) + (get-version package + latest-tag + #:prefix tag-prefix + #:suffix tag-suffix + #:delim tag-version-delimiter)))) + +(define (git-package? package) + "Whether the origin of PACKAGE is a Git repostiory." + (match (package-source package) + ((? origin? origin) + (and (eq? (origin-method origin) git-fetch) + (git-reference? (origin-uri origin)))) + (_ #f))) + +(define (latest-git-release package) + "Return the latest release of PACKAGE." + (let* ((name (package-name package)) + (properties (package-properties package)) + (tag-prefix (assq-ref properties 'tag-prefix)) + (tag-suffix (assq-ref properties 'tag-suffix)) + (tag-version-delimiter (assq-ref properties 'tag-version-delimite= r)) + (old-version (package-version package)) + (url (git-reference-url (origin-uri (package-source package)))) + (new-version (latest-git-tag-version package + tag-prefix + tag-suffix + tag-version-delimiter))) + + (if new-version + (upstream-source + (package name) + (version new-version) + (urls (list url))) + ;; No new release or no tags available. + #f))) + +(define %generic-git-updater + (upstream-updater + (name 'generic-git) + (description "Updater for packages hosted on Git repositories") + (pred git-package?) + (latest latest-git-release))) base-commit: 9540323458de87b0b8aa421e449a4fe27af7c393 --=20 2.33.0