From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id bDOyIQK6N2H8cwAAgWs5BA (envelope-from ) for ; Tue, 07 Sep 2021 21:14:10 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id WIbSHAK6N2GAfwAAbx9fmQ (envelope-from ) for ; Tue, 07 Sep 2021 19:14:10 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3C62711570 for ; Tue, 7 Sep 2021 21:14:10 +0200 (CEST) Received: from localhost ([::1]:37906 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mNgXp-0005ja-7M for larch@yhetil.org; Tue, 07 Sep 2021 15:14:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34968) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNgXi-0005hw-No for guix-patches@gnu.org; Tue, 07 Sep 2021 15:14:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:46226) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNgXi-000544-GS for guix-patches@gnu.org; Tue, 07 Sep 2021 15:14:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mNgXi-0005M8-C8 for guix-patches@gnu.org; Tue, 07 Sep 2021 15:14:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50359] [PATCH] import: Add 'generic-git' updater. Resent-From: Xinglu Chen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 07 Sep 2021 19:14:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50359 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Sarah Morgensen Cc: 50359@debbugs.gnu.org Received: via spool by 50359-submit@debbugs.gnu.org id=B50359.163104201820539 (code B ref 50359); Tue, 07 Sep 2021 19:14:02 +0000 Received: (at 50359) by debbugs.gnu.org; 7 Sep 2021 19:13:38 +0000 Received: from localhost ([127.0.0.1]:57771 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNgXJ-0005LB-6p for submit@debbugs.gnu.org; Tue, 07 Sep 2021 15:13:38 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:51562 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNgXG-0005Kt-4E for 50359@debbugs.gnu.org; Tue, 07 Sep 2021 15:13:35 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631042005; bh=DLb7xkBOXXKVJSPIRcO86Zhb4/KB0Iu8cDZSok1Pkhk=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=i9smTY8/QkOYWKejtO5vC8Fy24djHjLWQYhEKwznR3WHQZ3AwM/8KLYNMJErr2fr+ Ky+4vIx/iRGF9oInQj9zfPgyH8LPq/DqUsIgPqYvEAuLgyiv+Lo+zRVmvSGy+bEVYB 1RzSz3b7JCBn5Mf5yVAZtmUXXEXm9m+8k7qovVW8= In-Reply-To: <86pmtli4hn.fsf@mgsn.dev> References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> <87y28928vh.fsf@yoctocell.xyz> <86pmtli4hn.fsf@mgsn.dev> Date: Tue, 07 Sep 2021 21:13:22 +0200 Message-ID: <87wnnsyzal.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -4.00 Authentication-Results: aspmx1.migadu.com; none X-Migadu-Queue-Id: 3C62711570 X-Spam-Score: -4.00 X-Migadu-Scanner: scn0.migadu.com X-TUID: n07GkfAZPCzL --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Mon, Sep 06 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >> Any trick you used to find all of there weird version numbers? :-) > > This monstrosity: > > rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(version "([^\n"]*[^0-9\.][^"= \n]*)".*(\n.*){0,10}commit.*version' gnu/packages > > and to show just the versions: > > rg -Uor '$2' --pcre2 --no-filename --no-line-number=20 Wow! I will try that and see for myself! :-) >>> IMO, just get rid of the delimiter. If we wanted to be *that* flexible, >>> we could make it so they provide a tag->version proc instead of (prefix, >>> suffix, delimiter). >> >> a =E2=80=98tag->version=E2=80=99 procedure would probably make things a = bit too >> complicated for the people writing package definitions. For example, >> having a delimiter would make it easy to match a tag like >> =E2=80=9C2021-01-01-release=E2=80=9D >> >> Delimiter is =E2=80=9C.=E2=80=9D (sorry if this hurts your eyes ;-)) >> >> scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:pun= ct:]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1) >> $28 =3D "2021" >> >> Delimiter is =E2=80=9C-=E2=80=9D >> >> scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punct= :]]+(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1) >> $29 =3D "2021-01-01-release" >> >> And then, setting the suffix to =E2=80=9C-release=E2=80=9D would match j= ust the version >> part. > > Right. I missed that. > > In that vein, should we keep the dashes in "2021-01-01" or convert them > to periods? Having periods would be more consistent, then could have a =E2=80=98date->version=E2=80=99 procedure that replaces the hyphens with do= ts and have (git-reference (url "https://git.example.org") (commit (date->version version))) > What about when a tag has underscores? Hmm, not sure about that, below is a list of packages I could find which had underscores as delimiters gnu/packages/graphics.scm 239: (commit "DIRECTFB_1_7_7"))) gnu/packages/gstreamer.scm 326: (commit "ESOUND_0_2_41"))) gnu/packages/java.scm 13925: (commit "jboss-transaction-api_1.2_spec-1.1.1.F= inal"))) They all seem to use periods in the =E2=80=98version=E2=80=99 field, though= , so I would say that the underscroes, should also be converted to periods. > What if a repo has tags in both formats? Then "3.0.1" would be > considered older than "2011-01-01". That=E2=80=99s tricky, there isn=E2=80=99t really a way to know how old =E2= =80=9C3.0.1=E2=80=9D is, without looking at the metadata of the tag. Maybe this is one of those corner cases which can=E2=80=99t really automatically determine the latest release. Should we have a =E2=80=98no-refresh?=E2=80=99 property to tell t= he refresh to not try to update the package? > Maybe we should just add an extra bit to detect a date format and only > consider it when there's no "proper versions"? That could be a good idea! > Aaaand I fell down a rabbit hole after that :) I've attached a patch > with what I've done. It still has lots of issues--it requires the tag > to contain at least one version delimiter, it requires the first > character of the version to be a number... it might not even be better > than before I touched it, and even so the added complexity might not be > worth it. But if you'd like to take it for a spin, I've attached it (it > applies straight on master). Great! I will try it out and see how it compares to my current WIP version. Not having characters in the first version number probably isn=E2=80=99t su= ch a big deal, most version that contain characters end with a character. E.g., =E2=80=9C1.2.3a=E2=80=9D is not to uncommon, but =E2=80=9Ca1.2.3=E2= =80=9D is rarely seen. > -- > Sarah > > From 08bd59a7fa1aa9735a1794672ce8d1f683d3d6db Mon Sep 17 00:00:00 2001 > Message-Id: <08bd59a7fa1aa9735a1794672ce8d1f683d3d6db.1630975873.git.iska= rian@mgsn.dev> > From: Xinglu Chen > Date: Fri, 3 Sep 2021 17:50:56 +0200 > Subject: [PATCH] import: Add 'generic-git' updater. > > * guix/import/git.scm: New file. > * doc/guix.texi (Invoking guix refresh): Document it. > * Makefile.am (MODULES): Register it. > --- > Makefile.am | 1 + > doc/guix.texi | 27 +++++++ > guix/git.scm | 33 ++++++++ > guix/import/git.scm | 191 ++++++++++++++++++++++++++++++++++++++++++++ > 4 files changed, 252 insertions(+) > create mode 100644 guix/import/git.scm > > diff --git a/Makefile.am b/Makefile.am > index 3c79760734..c4d3a456b1 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -254,6 +254,7 @@ MODULES =3D \ > guix/import/egg.scm \ > guix/import/elpa.scm \ > guix/import/gem.scm \ > + guix/import/git.scm \ > guix/import/github.scm \ > guix/import/gnome.scm \ > guix/import/gnu.scm \ > diff --git a/doc/guix.texi b/doc/guix.texi > index 36a0c7f5ec..26afb1607a 100644 > --- a/doc/guix.texi > +++ b/doc/guix.texi > @@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, Lau= nchpad} packages. > @item generic-html > a generic updater that crawls the HTML page where the source tarball of > the package is hosted, when applicable. > +@item generic-git > +a generic updater for packages hosted on Git repositories. It tries to > +be smart about parsing Git tag names, but if it is not able to parse the > +tag name and compare tags correctly, users can define the following > +properties for a package. > + > +@itemize > +@item @code{tag-prefix}: a regular expression for matching a prefix of > +the tag name. > + > +@item @code{tag-suffix}: a regular expression for matching a suffix of > +the tag name. > + > +@item @code{tag-version-delimiter}: a string used as the delimiter in > +the tag name for separating the numbers of the version. > +@end itemize > + > +@lisp > +(package > + (name "foo") > + ;; ... > + (properties > + '((tag-prefix . "^release0-") > + (tag-suffix . "[a-z]?$") > + (tag-version-delimiter . ":")))) > +@end lisp=20=20=20=20=20=20 > + > @end table >=20=20 > For instance, the following command only checks for updates of Emacs > diff --git a/guix/git.scm b/guix/git.scm > index 9c6f326c36..c5d0d2da8e 100644 > --- a/guix/git.scm > +++ b/guix/git.scm > @@ -56,6 +56,8 @@ > commit-difference > commit-relation >=20=20 > + ls-remote-refs > + > git-checkout > git-checkout? > git-checkout-url > @@ -556,6 +558,37 @@ objects: 'ancestor (meaning that OLD is an ancestor = of NEW), 'descendant, or > (if (set-contains? oldest new) > 'descendant > 'unrelated)))))) > + > +;; > +;;; Remote operations. > +;;; > + > +(define* (ls-remote-refs url #:key tags?) > + "Return the list of references advertised at Git repository URL. If T= AGS? > +is true, limit to only refs/tags." > + (define (ref? ref) > + ;; Like `git ls-remote --refs', only show actual references. > + (and (string-prefix? "refs/" ref) > + (not (string-suffix? "^{}" ref)))) > + > + (define (tag? ref) > + (string-prefix? "refs/tags/" ref)) > + > + (define (include? ref) > + (and (ref? ref) > + (or (not tags?) (tag? ref)))) > + > + (with-libgit2 > + (call-with-temporary-directory > + (lambda (cache-directory) > + (let* ((repository (repository-init cache-directory)) > + ;; Create an in-memory remote so we don't touch disk. > + (remote (remote-create-anonymous repository url))) > + (remote-connect remote) > + (remote-disconnect remote) > + (repository-close! repository) > + > + (filter include? (map remote-head-name (remote-ls remote)))))))) >=20=20 > > ;;; > diff --git a/guix/import/git.scm b/guix/import/git.scm > new file mode 100644 > index 0000000000..8568981af2 > --- /dev/null > +++ b/guix/import/git.scm > @@ -0,0 +1,191 @@ > +;;; GNU Guix --- Functional package management for GNU > +;;; Copyright =C2=A9 2021 Xinglu Chen > +;;; Copyright =C2=A9 2021 Sarah Morgensen > +;;; > +;;; This file is part of GNU Guix. > +;;; > +;;; GNU Guix is free software; you can redistribute it and/or modify it > +;;; under the terms of the GNU General Public License as published by > +;;; the Free Software Foundation; either version 3 of the License, or (at > +;;; your option) any later version. > +;;; > +;;; GNU Guix is distributed in the hope that it will be useful, but > +;;; WITHOUT ANY WARRANTY; without even the implied warranty of > +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +;;; GNU General Public License for more details. > +;;; > +;;; You should have received a copy of the GNU General Public License > +;;; along with GNU Guix. If not, see . > + > +(define-module (guix import git) > + #:use-module (guix build utils) > + #:use-module (guix diagnostics) > + #:use-module (guix git) > + #:use-module (guix git-download) > + #:use-module (guix i18n) > + #:use-module (guix packages) > + #:use-module (guix upstream) > + #:use-module (guix utils) > + #:use-module (ice-9 format) > + #:use-module (ice-9 match) > + #:use-module (ice-9 rdelim) > + #:use-module (ice-9 regex) > + #:use-module (srfi srfi-1) > + #:use-module (srfi srfi-26) > + #:use-module (srfi srfi-34) > + #:use-module (srfi srfi-35) > + #:use-module (srfi srfi-71) > + #:export (%generic-git-updater)) > + > +;;; Commentary: > +;;; > +;;; This module provides a generic package updater for packages hosted o= n Git > +;;; repositories. > +;;; > +;;; It tries to be smart about tag names, but if it is not automatically= able > +;;; to parse the tag names correctly, users can set the `tag-prefix', > +;;; `tag-suffix' and `tag-version-delimiter' properties of the package t= o make > +;;; the updater parse the Git tag name correctly. > +;;; > +;;; Code: > + > +;;; Errors & warnings > + > +(define-condition-type &git-no-valid-tags-error &error > + git-no-valid-tags-error?) > + > +(define (git-no-valid-tags-error) > + (raise (condition (&message (message "no valid tags found")) > + (&git-no-valid-tags-error)))) > + > +(define-condition-type &git-no-tags-error &error > + git-no-tags-error?) > + > +(define (git-no-tags-error) > + (raise (condition (&message (message "no tags were found")) > + (&git-no-tags-error)))) > + > + > +;;; Updater > + > +(define* (get-version-mapping tags #:key prefix suffix delim) > + (define (guess-delim) > + (let ((total (length tags)) > + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) > + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) > + (underscores (reduce + 0 (map (cut string-count <> #\_) tags))= )) > + ;; (format #t "total: ~d, dots: ~d, dashes ~d, underscores ~d~%" > + ;; total dots dashes underscores) > + (cond > + ((> dots (* total 0.35)) ".") > + ((> dashes (* total 0.8)) "-") > + ((> underscores (* total 0.8)) "_") > + (else ".")))) These numbers seem rather arbitrary, how did you arrive at them? Also, AFAICS, versions without delimiters won=E2=80=99t be matched. > + (define delim-rx (regexp-quote (or delim (guess-delim)))) > + (define suffix-rx (or suffix "")) > + > + (define prefix-rx > + (make-regexp (string-append "^" (or prefix ".*") "$"))) Why is there a $ here? > + (define tag-rx > + (make-regexp > + (string-append "([[:digit:]][^" delim-rx "[:punct:]]*" > + "(" delim-rx "[^" delim-rx "]+)+" > + ")" suffix-rx "$"))) > + > + (define (get-version tag) > + (let ((tag-match (regexp-exec tag-rx tag))) > + (and tag-match > + (regexp-exec prefix-rx (match:prefix tag-match)) > + (regexp-substitute/global > + #f delim-rx (match:substring tag-match 1) > + 'pre "." 'post)))) > + > + (define (entry + (eq? (version-compare (car a) (car b)) '<)) > + > + (let ((mapping (fold alist-cons '() (map get-version tags) tags))) > + (stable-sort! (filter car mapping) entry + > +(define* (get-latest-tag url #:key prefix suffix delim) > + "Return the latest tag available from the Git repository at URL." > + (define (pre-release? tag) > + (any (cut string-contains tag <>) > + '("alpha" "beta" "rc" "dev" "test"))) This would only match lower-case characters, but sometimes upper-case characters are used. (define (pre-release? tag) (any (lambda (rx) (regexp-exec (make-regexp rx regexp/icase) tag)) '("alpha" "beta" "rc" "dev" "test"))) =20=20=20=20=20=20=20=20=20=20=20=20=20=20 > + (let* ((tags (map (cut string-drop <> (string-length "refs/tags/")) > + (ls-remote-refs url #:tags? #t))) > + (versions->tags > + (get-version-mapping (filter (negate pre-release?) tags) > + #:prefix prefix > + #:suffix suffix > + #:delim delim))) > + (cond > + ((null? tags) > + (git-no-tags-error)) > + ((null? versions->tags) > + (git-no-valid-tags-error)) > + (else > + (match (last versions->tags) > + ((version . tag) > + (values version tag))))))) > + > +(define (latest-git-tag-version package tag-prefix tag-suffix > + tag-version-delimiter) > + "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, and TAG-VERSION-DELIMITER > +properties of PACKAGE, returns the latest version of PACKAGE." > + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) > + (warning (or (package-field-location package 'source) > + (package-location package)) > + (G_ "~a for ~a~%") > + (condition-message c) > + (package-name package)) > + #f) > + ((eq? (exception-kind c) 'git-error) > + (warning (or (package-field-location package 'source) > + (package-location package)) > + (G_ "failed to fetch Git repository for ~a~%") > + (package-name package)) > + #f)) > + (let* ((source (package-source package)) > + (url (git-reference-url (origin-uri source)))) > + ;;(format #t "~a~%" (package-name package)) > + (get-latest-tag url #:prefix tag-prefix #:suffix tag-suffix > + #:delim tag-version-delimiter)))) > + > +(define (git-package? package) > + "Whether the origin of PACKAGE is a Git repostiory." > + (match (package-source package) > + ((? origin? origin) > + (and (eq? (origin-method origin) git-fetch) > + (git-reference? (origin-uri origin)))) > + (_ #f))) > + > +(define (latest-git-release package) > + "Return the latest release of PACKAGE." > + (let* ((name (package-name package)) > + (properties (package-properties package)) > + (tag-prefix (assq-ref properties 'tag-prefix)) > + (tag-suffix (assq-ref properties 'tag-suffix)) > + (tag-version-delimiter (assq-ref properties 'tag-version-delimi= ter)) > + (old-version (package-version package)) > + (url (git-reference-url (origin-uri (package-source package)))) > + (new-version (latest-git-tag-version package > + tag-prefix > + tag-suffix > + tag-version-delimiter))) > + > + (if new-version > + (upstream-source > + (package name) > + (version new-version) > + (urls (list url))) > + ;; No new release or no tags available. > + #f))) > + > +(define %generic-git-updater > + (upstream-updater > + (name 'generic-git) > + (description "Updater for packages hosted on Git repositories") > + (pred git-package?) > + (latest latest-git-release))) > > base-commit: 522a3bf99cbc21a9093f63280b9508cd69b94ff0 > --=20 > 2.32.0 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE3udIVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x59rgP/i9zonrR51YoZ4Dw68xfhW66m2op K7Xf9B45E+pAYKxo2R31HjO2B9CNYUXb5LBLAMvkqaH0KmP7Yv3LpBUpOm6EOmMh gtgTJTBlLetvjxrQmcOHBd6n/vIFp74wlukV17vPrxtlrQVCM50IBYvPcVuq8GtU eXQZhL3HYNc8K5ddKSRGwHNogZg3casFP3O2/x9J3wImFhIqA57IHan4n+mNAOUk ddUMhiypYNTM74gYQZ4tqspJNyWD7l7YNF2CA+mpsn+Vadz+tCALffKZTO6f1mQT ibuOV75ZbtuAcEkR+43o3dtWFamaQOuUZWLMBXFYU8iy91Suyqh3I1QazoKE0gXU 3hJO2x0+TdQV/bI8U/Hoj7nimXDbicJovFH9KCzOp2fPhcFxLCLwIMjc++BlMLx8 0TfkIZfHti962MlnYFgo0I0N/F0SiI6fFQBJ7L7i0nm3/el1wB23IIbjrNCQ9841 fLNw/Vz8QsO1xBNEFhrLhoFerzHLqmy+TggKVCzdlnlwPlrK5Y9cZZ1FIJq5LKli sirHcyfi22YL7B19sTRhqZYBY1Gttucykn1ttuRZg56JhJvPoDunoluZyTtYXUQt QkSpbjD/9VkyODCKgAjvm+3+e68uNlVePPCN5W/AroVPxw8oMjCeLakUTrKavM6w vl+l3fGzkXs14GR0 =d2MH -----END PGP SIGNATURE----- --=-=-=--