From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id YFuoI9m5NmEJEwAAgWs5BA (envelope-from ) for ; Tue, 07 Sep 2021 03:01:13 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 2LJTH9m5NmEcBwAAB5/wlQ (envelope-from ) for ; Tue, 07 Sep 2021 01:01:13 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A93381A993 for ; Tue, 7 Sep 2021 03:01:12 +0200 (CEST) Received: from localhost ([::1]:50270 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mNPU7-0002L4-Lz for larch@yhetil.org; Mon, 06 Sep 2021 21:01:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36016) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNPTz-0002IH-1D for guix-patches@gnu.org; Mon, 06 Sep 2021 21:01:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:43277) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNPTy-0003wS-PN for guix-patches@gnu.org; Mon, 06 Sep 2021 21:01:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mNPTy-0004YB-Og for guix-patches@gnu.org; Mon, 06 Sep 2021 21:01:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50359] [PATCH] import: Add 'generic-git' updater. Resent-From: Sarah Morgensen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 07 Sep 2021 01:01:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50359 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Xinglu Chen Cc: 50359@debbugs.gnu.org Received: via spool by 50359-submit@debbugs.gnu.org id=B50359.163097646117463 (code B ref 50359); Tue, 07 Sep 2021 01:01:02 +0000 Received: (at 50359) by debbugs.gnu.org; 7 Sep 2021 01:01:01 +0000 Received: from localhost ([127.0.0.1]:54822 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNPTw-0004XY-33 for submit@debbugs.gnu.org; Mon, 06 Sep 2021 21:01:00 -0400 Received: from out2.migadu.com ([188.165.223.204]:62202) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNPTs-0004XL-RF for 50359@debbugs.gnu.org; Mon, 06 Sep 2021 21:00:58 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1630976455; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=g9fElk3LvhkfJh78D55tZ1OXS453jmAETo9WcoQbHpk=; b=ktJw6RDJpOrKXdxxSvG+IYg6UFk76jZUmg2MEy1uc6S/6qqaZb/GRz0q6iA8IVlL/QyqsS nRDsxV5rBwjfHyFKordYG21MVSKF4TS56QrlSToI2xfLcKbTcZuffIIcyX5mwcjnMIzyiW xtjygFP2c0riO009Hvzg6Cj4hNHkglo= From: Sarah Morgensen References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> <87y28928vh.fsf@yoctocell.xyz> Date: Mon, 06 Sep 2021 18:00:52 -0700 In-Reply-To: <87y28928vh.fsf@yoctocell.xyz> (Xinglu Chen's message of "Mon, 06 Sep 2021 14:20:34 +0200 (5 hours, 48 minutes ago)") Message-ID: <86pmtli4hn.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Migadu-Auth-User: iskarian@mgsn.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1630976472; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=g9fElk3LvhkfJh78D55tZ1OXS453jmAETo9WcoQbHpk=; b=ndfrLjA1OpQwACDwhkaXfG6GYzv4Ltmju+YsO8vSE+moF/a8l11KBimuk24PMfkLES/yqE 74PjkI/ywGVzAx1Z7fiJ6gEHRw6rtrwGnEyOc15IlVTiPSOIkTQGIe21tYZjzwEYhsynSx 5EAn+i89PL/3lrp1O52fwvnXprOdTgdJxkAbiOWbxiTcD89CgDRY9C6VGA1K07ttmgiRB/ by42N6RHS2e2JXZsCiJWdNFJgv2/yPM7yQRKrysWi8PW+6UgBkZY5jk3pe/bPZ14ZUapIR tpcPBZbumkiQgkcGh9mN6/Z6a9X3FQELx5h8vbrqNUGmD/EwQlM5aznRrWhv2w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1630976472; a=rsa-sha256; cv=none; b=dt9TydBrcxdaLJeeizHF4HCzCfJDB4qRNnwZTwxKd5i8LAfqvuXeKtFOiAj11yhN8tXOnc u9gBR0JB6eqJwlD7Mb5pJIMemn2ZPF4OvL6QkE4BbjM4ICS6yxmc/4fh8jsKhPNl6IbXPI hyzuCNnWa7QnYenkS/83Bgxj1pWlnBQNCFgLc7J0d9cBED2E4ZW6vaWwnqV8OWv33yy49v AaR482TJ/p3f+f3ukAmbfPaYOYMdB1VQVH/JXZohSz3e8KYFbrw0zzMoeuS5yPnskMNsVi iDgO1tK3Eb54lGfAJHzhwcu8PtpOjKns0pxvLk5dj+lmgc4Af4obGCp4hFxNyw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=ktJw6RDJ; dmarc=fail reason="SPF not aligned (relaxed)" header.from=mgsn.dev (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.32 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=ktJw6RDJ; dmarc=fail reason="SPF not aligned (relaxed)" header.from=mgsn.dev (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: A93381A993 X-Spam-Score: -1.32 X-Migadu-Scanner: scn0.migadu.com X-TUID: va1WCgTNqpTD --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Xinglu Chen writes: > Any trick you used to find all of there weird version numbers? :-) This monstrosity: rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(version "([^\n"]*[^0-9\.][^"\n= ]*)".*(\n.*){0,10}commit.*version' gnu/packages and to show just the versions: rg -Uor '$2' --pcre2 --no-filename --no-line-number=20 >> IMO, just get rid of the delimiter. If we wanted to be *that* flexible, >> we could make it so they provide a tag->version proc instead of (prefix, >> suffix, delimiter). > > a =E2=80=98tag->version=E2=80=99 procedure would probably make things a b= it too > complicated for the people writing package definitions. For example, > having a delimiter would make it easy to match a tag like > =E2=80=9C2021-01-01-release=E2=80=9D > > Delimiter is =E2=80=9C.=E2=80=9D (sorry if this hurts your eyes ;-)) > > scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:punc= t:]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1) > $28 =3D "2021" > > Delimiter is =E2=80=9C-=E2=80=9D > > scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punct:= ]]+(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1) > $29 =3D "2021-01-01-release" > > And then, setting the suffix to =E2=80=9C-release=E2=80=9D would match ju= st the version > part. Right. I missed that. In that vein, should we keep the dashes in "2021-01-01" or convert them to periods? What about when a tag has underscores? What if a repo has tags in both formats? Then "3.0.1" would be considered older than "2011-01-01". Maybe we should just add an extra bit to detect a date format and only consider it when there's no "proper versions"? Aaaand I fell down a rabbit hole after that :) I've attached a patch with what I've done. It still has lots of issues--it requires the tag to contain at least one version delimiter, it requires the first character of the version to be a number... it might not even be better than before I touched it, and even so the added complexity might not be worth it. But if you'd like to take it for a spin, I've attached it (it applies straight on master). -- Sarah --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=0001-import-Add-generic-git-updater.patch Content-Transfer-Encoding: quoted-printable Content-Description: the rabbit hole >From 08bd59a7fa1aa9735a1794672ce8d1f683d3d6db Mon Sep 17 00:00:00 2001 Message-Id: <08bd59a7fa1aa9735a1794672ce8d1f683d3d6db.1630975873.git.iskari= an@mgsn.dev> From: Xinglu Chen Date: Fri, 3 Sep 2021 17:50:56 +0200 Subject: [PATCH] import: Add 'generic-git' updater. * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document it. * Makefile.am (MODULES): Register it. --- Makefile.am | 1 + doc/guix.texi | 27 +++++++ guix/git.scm | 33 ++++++++ guix/import/git.scm | 191 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 252 insertions(+) create mode 100644 guix/import/git.scm diff --git a/Makefile.am b/Makefile.am index 3c79760734..c4d3a456b1 100644 --- a/Makefile.am +++ b/Makefile.am @@ -254,6 +254,7 @@ MODULES =3D \ guix/import/egg.scm \ guix/import/elpa.scm \ guix/import/gem.scm \ + guix/import/git.scm \ guix/import/github.scm \ guix/import/gnome.scm \ guix/import/gnu.scm \ diff --git a/doc/guix.texi b/doc/guix.texi index 36a0c7f5ec..26afb1607a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, Launc= hpad} packages. @item generic-html a generic updater that crawls the HTML page where the source tarball of the package is hosted, when applicable. +@item generic-git +a generic updater for packages hosted on Git repositories. It tries to +be smart about parsing Git tag names, but if it is not able to parse the +tag name and compare tags correctly, users can define the following +properties for a package. + +@itemize +@item @code{tag-prefix}: a regular expression for matching a prefix of +the tag name. + +@item @code{tag-suffix}: a regular expression for matching a suffix of +the tag name. + +@item @code{tag-version-delimiter}: a string used as the delimiter in +the tag name for separating the numbers of the version. +@end itemize + +@lisp +(package + (name "foo") + ;; ... + (properties + '((tag-prefix . "^release0-") + (tag-suffix . "[a-z]?$") + (tag-version-delimiter . ":")))) +@end lisp=20=20=20=20=20=20 + @end table =20 For instance, the following command only checks for updates of Emacs diff --git a/guix/git.scm b/guix/git.scm index 9c6f326c36..c5d0d2da8e 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -56,6 +56,8 @@ commit-difference commit-relation =20 + ls-remote-refs + git-checkout git-checkout? git-checkout-url @@ -556,6 +558,37 @@ objects: 'ancestor (meaning that OLD is an ancestor of= NEW), 'descendant, or (if (set-contains? oldest new) 'descendant 'unrelated)))))) + +;; +;;; Remote operations. +;;; + +(define* (ls-remote-refs url #:key tags?) + "Return the list of references advertised at Git repository URL. If TAG= S? +is true, limit to only refs/tags." + (define (ref? ref) + ;; Like `git ls-remote --refs', only show actual references. + (and (string-prefix? "refs/" ref) + (not (string-suffix? "^{}" ref)))) + + (define (tag? ref) + (string-prefix? "refs/tags/" ref)) + + (define (include? ref) + (and (ref? ref) + (or (not tags?) (tag? ref)))) + + (with-libgit2 + (call-with-temporary-directory + (lambda (cache-directory) + (let* ((repository (repository-init cache-directory)) + ;; Create an in-memory remote so we don't touch disk. + (remote (remote-create-anonymous repository url))) + (remote-connect remote) + (remote-disconnect remote) + (repository-close! repository) + + (filter include? (map remote-head-name (remote-ls remote)))))))) =20 ;;; diff --git a/guix/import/git.scm b/guix/import/git.scm new file mode 100644 index 0000000000..8568981af2 --- /dev/null +++ b/guix/import/git.scm @@ -0,0 +1,191 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import git) + #:use-module (guix build utils) + #:use-module (guix diagnostics) + #:use-module (guix git) + #:use-module (guix git-download) + #:use-module (guix i18n) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 format) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-34) + #:use-module (srfi srfi-35) + #:use-module (srfi srfi-71) + #:export (%generic-git-updater)) + +;;; Commentary: +;;; +;;; This module provides a generic package updater for packages hosted on = Git +;;; repositories. +;;; +;;; It tries to be smart about tag names, but if it is not automatically a= ble +;;; to parse the tag names correctly, users can set the `tag-prefix', +;;; `tag-suffix' and `tag-version-delimiter' properties of the package to = make +;;; the updater parse the Git tag name correctly. +;;; +;;; Code: + +;;; Errors & warnings + +(define-condition-type &git-no-valid-tags-error &error + git-no-valid-tags-error?) + +(define (git-no-valid-tags-error) + (raise (condition (&message (message "no valid tags found")) + (&git-no-valid-tags-error)))) + +(define-condition-type &git-no-tags-error &error + git-no-tags-error?) + +(define (git-no-tags-error) + (raise (condition (&message (message "no tags were found")) + (&git-no-tags-error)))) + + +;;; Updater + +(define* (get-version-mapping tags #:key prefix suffix delim) + (define (guess-delim) + (let ((total (length tags)) + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) + (underscores (reduce + 0 (map (cut string-count <> #\_) tags)))) + ;; (format #t "total: ~d, dots: ~d, dashes ~d, underscores ~d~%" + ;; total dots dashes underscores) + (cond + ((> dots (* total 0.35)) ".") + ((> dashes (* total 0.8)) "-") + ((> underscores (* total 0.8)) "_") + (else ".")))) + + (define delim-rx (regexp-quote (or delim (guess-delim)))) + (define suffix-rx (or suffix "")) + + (define prefix-rx + (make-regexp (string-append "^" (or prefix ".*") "$"))) + (define tag-rx + (make-regexp + (string-append "([[:digit:]][^" delim-rx "[:punct:]]*" + "(" delim-rx "[^" delim-rx "]+)+" + ")" suffix-rx "$"))) + + (define (get-version tag) + (let ((tag-match (regexp-exec tag-rx tag))) + (and tag-match + (regexp-exec prefix-rx (match:prefix tag-match)) + (regexp-substitute/global + #f delim-rx (match:substring tag-match 1) + 'pre "." 'post)))) + + (define (entry) + '("alpha" "beta" "rc" "dev" "test"))) + + (let* ((tags (map (cut string-drop <> (string-length "refs/tags/")) + (ls-remote-refs url #:tags? #t))) + (versions->tags + (get-version-mapping (filter (negate pre-release?) tags) + #:prefix prefix + #:suffix suffix + #:delim delim))) + (cond + ((null? tags) + (git-no-tags-error)) + ((null? versions->tags) + (git-no-valid-tags-error)) + (else + (match (last versions->tags) + ((version . tag) + (values version tag))))))) + +(define (latest-git-tag-version package tag-prefix tag-suffix + tag-version-delimiter) + "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, and TAG-VERSION-DELIMITER +properties of PACKAGE, returns the latest version of PACKAGE." + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) + (warning (or (package-field-location package 'source) + (package-location package)) + (G_ "~a for ~a~%") + (condition-message c) + (package-name package)) + #f) + ((eq? (exception-kind c) 'git-error) + (warning (or (package-field-location package 'source) + (package-location package)) + (G_ "failed to fetch Git repository for ~a~%") + (package-name package)) + #f)) + (let* ((source (package-source package)) + (url (git-reference-url (origin-uri source)))) + ;;(format #t "~a~%" (package-name package)) + (get-latest-tag url #:prefix tag-prefix #:suffix tag-suffix + #:delim tag-version-delimiter)))) + +(define (git-package? package) + "Whether the origin of PACKAGE is a Git repostiory." + (match (package-source package) + ((? origin? origin) + (and (eq? (origin-method origin) git-fetch) + (git-reference? (origin-uri origin)))) + (_ #f))) + +(define (latest-git-release package) + "Return the latest release of PACKAGE." + (let* ((name (package-name package)) + (properties (package-properties package)) + (tag-prefix (assq-ref properties 'tag-prefix)) + (tag-suffix (assq-ref properties 'tag-suffix)) + (tag-version-delimiter (assq-ref properties 'tag-version-delimite= r)) + (old-version (package-version package)) + (url (git-reference-url (origin-uri (package-source package)))) + (new-version (latest-git-tag-version package + tag-prefix + tag-suffix + tag-version-delimiter))) + + (if new-version + (upstream-source + (package name) + (version new-version) + (urls (list url))) + ;; No new release or no tags available. + #f))) + +(define %generic-git-updater + (upstream-updater + (name 'generic-git) + (description "Updater for packages hosted on Git repositories") + (pred git-package?) + (latest latest-git-release))) base-commit: 522a3bf99cbc21a9093f63280b9508cd69b94ff0 --=20 2.32.0 --=-=-=--