From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id +RumJz4NNGHYcAEAgWs5BA (envelope-from ) for ; Sun, 05 Sep 2021 02:20:14 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id gHLMIj4NNGELYQAA1q6Kng (envelope-from ) for ; Sun, 05 Sep 2021 00:20:14 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id BDFD515B25 for ; Sun, 5 Sep 2021 02:20:13 +0200 (CEST) Received: from localhost ([::1]:48770 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mMftM-0002KE-Fh for larch@yhetil.org; Sat, 04 Sep 2021 20:20:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49756) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mMftC-0002Jr-F7 for guix-patches@gnu.org; Sat, 04 Sep 2021 20:20:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:37322) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mMftC-0005y4-7F for guix-patches@gnu.org; Sat, 04 Sep 2021 20:20:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mMftB-0008Nb-SM for guix-patches@gnu.org; Sat, 04 Sep 2021 20:20:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50359] [PATCH] import: Add 'generic-git' updater. In-Reply-To: Resent-From: Sarah Morgensen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 05 Sep 2021 00:20:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50359 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Xinglu Chen Cc: 50359@debbugs.gnu.org Received: via spool by 50359-submit@debbugs.gnu.org id=B50359.163080119032190 (code B ref 50359); Sun, 05 Sep 2021 00:20:01 +0000 Received: (at 50359) by debbugs.gnu.org; 5 Sep 2021 00:19:50 +0000 Received: from localhost ([127.0.0.1]:48868 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMfsz-0008N7-MW for submit@debbugs.gnu.org; Sat, 04 Sep 2021 20:19:50 -0400 Received: from out2.migadu.com ([188.165.223.204]:31959) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMfsw-0008Mx-8W for 50359@debbugs.gnu.org; Sat, 04 Sep 2021 20:19:48 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1630801184; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=MKFB+NpBec8yWeoNs7VQUyviuqZy0NVyZz//X2MEXDk=; b=N7dMN8FCGRaqZ4MVYnaa0ZkhEz6dwSP6219YhbpdiasHZj7T2amr2GrNyTHWnL6nvYVECP o2kMA4ywPhE8kgITkLTq7R0ZKTqQjWS/bj3SvMoPk08s/icXgtpPyG1VHg4VFN0LlDiRxW 0sGXZAvfPDH35SoIFg1PQIsooxcs+eg= From: Sarah Morgensen References: Date: Sat, 04 Sep 2021 17:19:40 -0700 Message-ID: <86k0jvkh5v.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Migadu-Auth-User: iskarian@mgsn.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1630801214; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=MKFB+NpBec8yWeoNs7VQUyviuqZy0NVyZz//X2MEXDk=; b=gYC2avOkQyQ5GV8L14IWaP+DL7W38TGuuoqAK1cs31xsgXofRGKiCxgGkklOoECAqrOvhF uHFSWQlL4ndtnGoV+Kke/AdyGnMniwbJc97j+s8ufU9WyCyrvAWoM4OeTvMyRHqB4+13dM +F1D1yHUgtCfl6FZtqbulp6AUWU3hA9z1nwYVjFKhlePPKgeeqp16Y33Hr+VqktwmCr8pw KNj8sUjPIFYZaz27A5PunzZOc/JWhmH7emTXNaeuvfDpSi+bJphlCmsCpYyDLQpf6fFJtJ qM/PuFACZb7i98zzykh5zaw8U5E25uW37rhHnv3Nqh+8gFeVHZWUrEQC9UfTMA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1630801214; a=rsa-sha256; cv=none; b=SZRoR7AhXxeqlAHXUE6P4k/zKxL6K2l5wn1p2cvr/8YZZ1d5uckSyqrljjQPWchIVuGgYH L7GQki4Q6+JJHP/DmFrpJ2pr/cRP9HP6Z9mYLFGZcp4CgEmIOTWAPbktJ+AYp7a2x6zGa8 aq6SwqzJQZJN6zvCmGazjMv32zM71nNmmSOfulVUZBCwmcRacoI72e/4denURtQlYhbsQp ddCA8+rPuCxU+dC1c3nw9afsdKyC9fu29vXYpNRyuAsHmcz36thoPCst7AwwaouNpx3nKc 1xAzG9xQqa1yQgwLeVJIjGXIUS4R3AZv5+Dm2wRR3LUulda0xtdrKW86IcGb8w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=N7dMN8FC; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: 0.18 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=N7dMN8FC; dmarc=fail reason="SPF not aligned (relaxed)" header.from=mgsn.dev (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: BDFD515B25 X-Spam-Score: 0.18 X-Migadu-Scanner: scn0.migadu.com X-TUID: oTjUbagvmzmq --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello, Thanks for the patch! Glad to see this idea becoming more polished. Xinglu Chen writes: > * guix/import/git.scm: New file. > * doc/guix.texi (Invoking guix refresh): Document it. > * Makefile.am (MODULES): Register it. > --- > This patch adds a new =E2=80=98generic-git=E2=80=99 updater which can che= ck for new tags > for package hosted on Git repos. However, it cannot download Git repos > and update the package definitions, i.e. =E2=80=98guix refresh -u=E2=80= =99. There is a > pending patch that would add this feature though[1]. > > =E2=80=98guix refresh -L=E2=80=99 now reports > > Available updaters: > [=E2=80=A6] > 94.5% of the packages are covered by these updaters. > > We are getting close to 100% :-) Wow, that is close! > > See it in action! > > $ ./pre-inst-env guix refresh harmonist scdoc gmnisrv > gnu/packages/web.scm:7931:4: warning: no tags were found for package `gmn= isrv' > gnu/packages/web.scm:7931:4: warning: 'generic-git' updater failed to det= ermine available releases for gmnisrv > gnu/packages/man.scm:339:12: scdoc would be upgraded from 1.10.1 to 1.11.1 > gnu/packages/games.scm:9433:2: warning: failed to fetch Git repository fo= r package `harmonist' > gnu/packages/games.scm:9433:2: warning: 'generic-git' updater failed to d= etermine available releases for harmonist FWIW, harmonist and a few other packages fail to work because they use an old git protocol which is not supported by libgit2. [...] > + > +@itemize > +@item @code{tag-prefix}: a regular expression for matching a prefix of > +the tag name. > + > +@item @code{tag-suffix}: a regular expression for matching a suffix of > +the tag name. > + > +@item @code{tag-version-delimiter}: a string used as the delimiter in > +the tag name for separating the numbers of the version. > +@end itemize > + > +@lisp > +(package > + (name "foo") > + ;; ... > + (properties > + '((tag-prefix . "^release0-") > + (tag-suffix . "[a-z]?$") > + (tag-version-delimiter . ":")))) > +@end lisp ^ extra whitespace I do like the selection of (prefix, suffix, delimiter), though I think there are only one or two packages which use a different delimiter. [...] > +;;; Errors & warnings > + > +(define-condition-type &git-tag-error &error > + git-tag-error? > + (kind git-tag-error-kind)) > + > +(define (git-tag-error kind) > + (raise (condition (&message (message (format "bad `~a' property"))) > + (&git-tag-error > + (kind kind))))) When I trigger this error, I get: --8<---------------cut here---------------start------------->8--- In ice-9/exceptions.scm: 406:15 6 (latest-git-release _) In ice-9/boot-9.scm: 1752:10 5 (with-exception-handler _ _ #:unwind? _ # _) In guix/import/git.scm: 59:39 4 (get-version _ _ #:prefix _ #:suffix _ #:delim _) In unknown file: 3 (simple-format #f "bad `~a' property") In ice-9/boot-9.scm: 1685:16 2 (raise-exception _ #:continuable? _) 1683:16 1 (raise-exception _ #:continuable? _) 1685:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1685:16: In procedure raise-exception: In procedure simple-format: FORMAT: Missing argument for ~a --8<---------------cut here---------------end--------------->8--- > + > +(define (git-tag-warning package c) > + (warning (package-location package) > + (G_ "~a for package `~a'~%") > + (condition-message c) > + (package-name package))) > + > +(define-condition-type &git-no-tags-error &error > + git-no-tags-error?) > + > +(define (git-no-tags-error) > + (raise (condition (&message (message "no tags were found")) > + (&git-no-tags-error)))) > + > +(define (git-no-tags-warning package c) > + (warning (package-location package) > + (G_ "~a for package `~a'~%") > + (condition-message c) > + (package-name package))) > + > +(define (git-fetch-warning package) > + (warning (package-location package) > + (G_ "failed to fetch Git repository for package `~a'~%") > + (package-name package))) > + > + > +;;; Helper functions > + > +(define (string-split* str delim) > + "Like `string-split', but DELIM is a string instead of a > +char-set." > + (filter (lambda (str) (not (equal? str ""))) > + (string-split str (string->char-set delim)))) (string-split* "1:2.3" ":.") -> ("1" "2" "3") (string-split* "1a2b3" "ab") -> ("1" "2" "3") Is this what you intended? The documentation above makes it sound like the whole string serves as the delimiter. > + > +(define* (get-version package tag #:key prefix suffix delim) PACKAGE is not used by this procedure. > + (define delim* (if delim delim ".")) > + (define prefix-regexp "^[^0-9]*") > + (define suffix-regexp (string-append "[^0-9" (regexp-quote delim*) "]*= $")) With a delimiter of '.', this would say the suffix of '1.2.3.prerelease' is 'prerelease', not '.prerelease'. Is this correct? (I would be tempted to just remove delim* from this.) > + (define delim-regexp (string-append "^[0-9]+" (regexp-quote delim*) "[= 0-9]+")) This fails to account for versions which use non-numerics, such as (all taken from the package-version field of packages using git-fetch and which use this version as the tag): 1.0.0-beta.0 0.0.9.4f 4.4-git.1 5.2.0-alpha 0.2.0-alpha-199-g3e7a475 20200701.154658.b0d6223 12-068oasis4 4.0.0.dev8 0.32-14-gcdfe14e 2.8-fix-2 There are about 50-60 packages like this. I'm not sure how much effort should be spent including them, and for some of them I'm not sure what our ideal behavior *is*. Even if we could reliably detect them, should "alpha" or "dev" packages be returned by the updater? Upon investigation, there is a deeper problem: version-compare thinks "5.2.0" is a lower version than "5.2.0-alpha", and that "4.0.0" is lower than "4.0.0.dev8". scheme@(guile-user)> (version-compare "5.1.9" "5.2.0") $5 =3D < scheme@(guile-user)> (version-compare "5.2.0" "5.2.0-alpha") $6 =3D < scheme@(guile-user)> (version-compare "4.0.0" "4.0.0.dev8") $7 =3D < > + > + (define no-prefix > + (let ((match (string-match (or prefix prefix-regexp) tag))) > + (if match > + (regexp-substitute #f match 'post) > + (git-tag-error 'tag-prefix)))) > + > + (define no-suffix > + (let ((match (string-match (or suffix suffix-regexp) no-prefix))) > + (if match > + (regexp-substitute #f match 'pre) > + (git-tag-error 'tag-suffix)))) > + > + (define no-delims > + (if (string-match delim-regexp no-suffix) > + (string-split* no-suffix delim*) > + (git-tag-error 'tag-version-delimiter))) This throws an error if the version doesn't have any delimiter. Actually, it throws an error in a lot of other cases too, often saying the 'tag-version-delimiter is wrong when it's something else. Consider the tags from the "openjpeg" package, sorted by 'sort-tags': arelease opj0-97 start v2.1.1 v2.1.2 v2.2.0 v2.3.0 v2.3.1 v2.4.0 version.1.0 version.1.1 version.1.2 version.1.3 version.1.4 version.1.5 version.1.5.1 version.1.5.2 version.2.0 version.2.0.1 version.2.1 wg1n6848 At first, 'get-version' throws an error because "wg1n6848" doesn't have a delimiter. But even disregarding that, it would return "version.2.1" -> "2.1" as the latest version. Probably we should process all tags with 'get-version' (simply skipping any that don't parse) and use that to sort the tags. If none parse with 'get-version' we could use the "no tags" error or have a separate error for "there were tags but we couldn't process them". And this lets us just do something like (untested): (define* (get-version tag #:key prefix suffix delim) (define delim-rx (regexp-quote (or delim "."))) (define prefix-rx (or prefix "[^[:digit:]]*")) (define suffix-rx (or suffix ".*")) (define version-char-rx (string-append "[^" delim-rx "[:punct:]]")) (define tag-rx (string-append "^" prefix "(" version-char-rx "+(" delim-rx version-char-rx ")*)" suffix-rx "$")) (and=3D> (string-match tag-rx tag) (cut match-substring <> 1))) Though at this point, 'tag-rx' should probably be constructed and compiled outside the loop. > + > + (string-join no-delims ".")) > + > +(define (sort-tags tags) > + "Sort TAGS, a list if Git tags, such that the latest tag is the last e= lement." > + (sort tags (lambda (a b) > + (eq? (version-compare a b) '<)))) > + > + > +;;; Updater > + > +(define (get-remote url git-uri) > + "Given a URL and GIT-URI, a record, return the ``origi= n'' remote." > + (let* ((checkout (update-cached-checkout url > + #:recursive? > + (git-reference-recursive? git= -uri))) > + (repository (repository-open checkout))) > + (remote-lookup repository "origin"))) We surely don't want 'update-cached-checkout' since that fetches the whole repo history! I've attached a patch below (based on top of this one) which brings the total time-per-package to under 1s. I moved it to (guix git) to make use of 'with-libgit2' which ensures we use system certificates. Apologies for such a long reply. I hope it was helpful :) -- Sarah --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-git-Add-ls-remote-refs.patch Content-Description: git: Add 'ls-remote-refs'. >From 0b0973034711e15b52702c0aec0c653dfd41928c Mon Sep 17 00:00:00 2001 Message-Id: <0b0973034711e15b52702c0aec0c653dfd41928c.1630800771.git.iskarian@mgsn.dev> From: Sarah Morgensen Date: Fri, 3 Sep 2021 22:40:02 -0700 Subject: [PATCH] git: Add 'ls-remote-refs'. --- guix/git.scm | 33 +++++++++++++++++++++++++++++++ guix/import/git.scm | 47 ++++++++++----------------------------------- 2 files changed, 43 insertions(+), 37 deletions(-) diff --git a/guix/git.scm b/guix/git.scm index 9c6f326c36..b784fd6d20 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -56,6 +56,8 @@ commit-difference commit-relation + ls-remote-refs + git-checkout git-checkout? git-checkout-url @@ -556,6 +558,37 @@ objects: 'ancestor (meaning that OLD is an ancestor of NEW), 'descendant, or (if (set-contains? oldest new) 'descendant 'unrelated)))))) + +;; +;;; Remote operations. +;;; + +(define* (ls-remote-refs url #:key tags?) + "Return the list of references advertised at Git repository URL. If TAGS? +is true, limit to only refs/tags." + (define (ref? ref) + ;; Like `git ls-remote --refs', only show actual references. + (and (string-prefix? "refs/" ref) + (not (string-suffix? "^{}" ref)))) + + (define (tag? ref) + (string-prefix? "refs/tags/" ref)) + + (define (include? ref) + (and ref? + (or (not tags?) (tag? ref)))) + + (with-libgit2 + (with-temporary-directory + (lambda (cache-directory) + (let* ((repository (repository-init cache-directory)) + ;; Create an in-memory remote so we don't touch disk. + (remote (remote-create-anonymous repository url))) + (remote-connect remote) + (remote-disconnect remote) + (repository-close! repository) + + (filter include? (map remote-head-name (remote-ls remote)))))))) ;;; diff --git a/guix/import/git.scm b/guix/import/git.scm index 9a654c1972..097a2f70bc 100644 --- a/guix/import/git.scm +++ b/guix/import/git.scm @@ -17,7 +17,6 @@ ;;; along with GNU Guix. If not, see . (define-module (guix import git) - #:use-module (git) #:use-module (guix build utils) #:use-module (guix diagnostics) #:use-module (guix git) @@ -126,40 +125,15 @@ char-set." ;;; Updater -(define (get-remote url git-uri) - "Given a URL and GIT-URI, a record, return the ``origin'' remote." - (let* ((checkout (update-cached-checkout url - #:recursive? - (git-reference-recursive? git-uri))) - (repository (repository-open checkout))) - (remote-lookup repository "origin"))) - -(define (get-latest-tag remote) - "Given a Git REMOTE, return that latest tag available." - (remote-connect remote) - - (define tags - (sort-tags - (map (lambda (tag) - (string-drop tag (string-length "refs/tags/"))) - (filter (lambda (ref) - ;; Every tag has two refs: - ;; - ;; * refs/tags/1.2.3^{} - ;; * refs/tags/1.2.3 - ;; - ;; remove the one with the trailing ^{} - (and (not (string-suffix? "^{}" ref)) - (string-prefix? "refs/tags/" ref))) - (map (lambda (remote-head) - (remote-head-name remote-head)) - (remote-ls remote)))))) - - (remote-disconnect remote) - - (if (null? tags) - (git-no-tags-error) - (last tags))) +(define (get-latest-tag url) + "Return the latest tag available from the Git repository at URL." + (let ((tags (map (cut string-drop <> (string-length "refs/tags/")) + (ls-remote-refs url #:tags? #t)))) + + (if (null? tags) + (git-no-tags-error) + (last (sort-tags tags))))) + (define (latest-git-tag-version package tag-prefix tag-suffix tag-version-delimiter) @@ -177,8 +151,7 @@ properties of PACKAGE, returns the latest version of PACKAGE." (let* ((source (package-source package)) (git-uri (origin-uri source)) (url (git-reference-url (origin-uri source))) - (remote (get-remote url git-uri)) - (latest-tag (get-latest-tag remote))) + (latest-tag (get-latest-tag url))) (get-version package latest-tag #:prefix tag-prefix base-commit: 522a3bf99cbc21a9093f63280b9508cd69b94ff0 prerequisite-patch-id: c60e771d96884a78a014e145723562a619c1a0e0 -- 2.32.0 --=-=-=--