From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id Kf0vGxEKQ2ExMgEAgWs5BA (envelope-from ) for ; Thu, 16 Sep 2021 11:10:41 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id yKl3FhEKQ2E9UwAA1q6Kng (envelope-from ) for ; Thu, 16 Sep 2021 09:10:41 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 96ACA19BEA for ; Thu, 16 Sep 2021 11:10:40 +0200 (CEST) Received: from localhost ([::1]:52534 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQnPj-00082F-MD for larch@yhetil.org; Thu, 16 Sep 2021 05:10:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55398) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQnP9-000822-0V for guix-patches@gnu.org; Thu, 16 Sep 2021 05:10:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:42261) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mQnP8-0003lG-Op for guix-patches@gnu.org; Thu, 16 Sep 2021 05:10:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mQnP8-0007Or-FP for guix-patches@gnu.org; Thu, 16 Sep 2021 05:10:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50359] [PATCH 3/3] import: Add 'generic-git' updater. Resent-From: Sarah Morgensen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 16 Sep 2021 09:10:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50359 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Xinglu Chen Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 50359@debbugs.gnu.org Received: via spool by 50359-submit@debbugs.gnu.org id=B50359.163178338928420 (code B ref 50359); Thu, 16 Sep 2021 09:10:02 +0000 Received: (at 50359) by debbugs.gnu.org; 16 Sep 2021 09:09:49 +0000 Received: from localhost ([127.0.0.1]:53807 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQnOu-0007OJ-Nf for submit@debbugs.gnu.org; Thu, 16 Sep 2021 05:09:49 -0400 Received: from out0.migadu.com ([94.23.1.103]:33137) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQnOn-0007O5-Qm for 50359@debbugs.gnu.org; Thu, 16 Sep 2021 05:09:47 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1631783379; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+4Td9ov4xszFVhGp2zWH36ghExMeFtjA/PLoyjKvFGo=; b=V1lRUYIkBVYPIrNh38PMkwtxAIO5d5rpIfzBP5kiJSTcrI5yvZNPLwlxByXbq2G9TMM0Oh Jj4a1F1YrMIfNojwBp02Dmee8k1Wg1nVIo5liypWQKYGqJMY2MVhptVi1wXe9mbp5WAbuq m947wjhYsSNEOLyZq5z/tYdvmJVSZ0Q= From: Sarah Morgensen References: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> Date: Thu, 16 Sep 2021 02:09:32 -0700 In-Reply-To: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> (Xinglu Chen's message of "Fri, 10 Sep 2021 18:21:30 +0200 (5 days, 15 hours, 34 minutes ago)") Message-ID: <86czp8j38z.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1631783440; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=+4Td9ov4xszFVhGp2zWH36ghExMeFtjA/PLoyjKvFGo=; b=JAbNGko9dOtFhh3cRJsBsT7pXxDKwgIxaKzUZiieSgFIPUlkuaAoxusYJ+vWhAbsyrPyQM vO4F+IVMmPIoHKknqpCf4EQe2P+ySaUAL/zc0yeBS0Sj36MHHhomQihuj3nHuW3XZ7rboA VUG+gLYb2IXg0mLRCRtrjiNWOMatbOaGhHr5TCAAgsY1PxAfTbnrlMs/wZ3i7tBWTPo8iR QMbUaMihFiHd66EpGV16o6T/VBDzlT86N08nPHwRRP2uQWl5FfbfVioGzDsotjUpGrbuXp vXLqhBJteuU18t79ipWDWs5FDfEJNW1/WUoNLVHJiZMNyMI42gqjvulu/WUPCg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1631783440; a=rsa-sha256; cv=none; b=X4EDXV8A1e5+aSKQZzP6COl5MRT7o+5cm2uZB6h/AKR4kb9/edlAUzLPBi6DWXuOoC+lNb 6h3tzIoF3nOdL5s1QCUNP99UAtM0qHVZFlv9zVRv25MOG8FqA27ApT7eLhCzHli06n/I9d L02dQTlz8IymYBUq7YUpA7pPWviMRqskUyXrczedmsat30HaSwBrBiWuA3ZUXQWVjO2Xer NRGU6fXJWYDgO0SNb//1VFFHQQAesQtkOptW6T8j+wvFY8+gez6eLPMTOrI/Rpr6AWn5rB wiXKCfW5eSPAn48kcY3T4eoZ2HZLpjPyJP8w5IMeYBCMoR0t0GNhWAgw/EbC0w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=V1lRUYIk; dmarc=fail reason="SPF not aligned (relaxed)" header.from=mgsn.dev (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.30 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=V1lRUYIk; dmarc=fail reason="SPF not aligned (relaxed)" header.from=mgsn.dev (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 96ACA19BEA X-Spam-Score: -1.30 X-Migadu-Scanner: scn0.migadu.com X-TUID: 1/1ewNTpqwNw Hello, Here are my promised nits. Xinglu Chen writes: > * guix/git.scm (ls-remote-refs): New procedure. > * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. > * guix/import/git.scm: New file. > * doc/guix.texi (Invoking guix refresh): Document it. > * tests/import-git.scm: New test file. > * Makefile.am (MODULES, SCM_TESTS): Register the new files. > > Co-authored-by: Sarah Morgensen Again, much thanks for writing tests. > +@item generic-git > +a generic updater for packages hosted on Git repositories. It tries to > +be smart about parsing Git tag names, but if it is not able to parse the > +tag name and compare tags correctly, users can define the following > +properties for a package. > + > +@itemize > +@item @code{release-tag-prefix}: a regular expression for matching a prefix of > +the tag name. > + > +@item @code{release-tag-suffix}: a regular expression for matching a suffix of > +the tag name. > + > +@item @code{release-tag-version-delimiter}: a string used as the delimiter in > +the tag name for separating the numbers of the version. > +@end itemize > + > +@lisp > +(package > + (name "foo") > + ;; ... > + (properties > + '((release-tag-prefix . "^release0-") > + (release-tag-suffix . "[a-z]?$") > + (release-tag-version-delimiter . ":")))) > +@end lisp > + > +By default, the updater will ignore pre-releases; to make it also look > +for pre-releases, set the @code{accept-pre-releases?} property to > +@code{#t}. Should this be itemized above? > + > +;; > +;;; Remote operations. > +;;; > + > +(define* (remote-refs url #:key tags?) > + "Return the list of references advertised at Git repository URL. If TAGS? > +is true, limit to only refs/tags." > + (define (ref? ref) > + ;; Like `git ls-remote --refs', only show actual references. > + (and (string-prefix? "refs/" ref) > + (not (string-suffix? "^{}" ref)))) > + > + (define (tag? ref) > + (string-prefix? "refs/tags/" ref)) > + > + (define (include? ref) > + (and (ref? ref) > + (or (not tags?) (tag? ref)))) > + > + (with-libgit2 > + (call-with-temporary-directory > + (lambda (cache-directory) > + (let* ((repository (repository-init cache-directory)) > + ;; Create an in-memory remote so we don't touch disk. > + (remote (remote-create-anonymous repository url))) > + (remote-connect remote) > + (remote-disconnect remote) > + (repository-close! repository) > + > + (filter-map (lambda (remote) > + (let ((name (remote-head-name remote))) > + (and (include? name) > + name))) > + (remote-ls remote))))))) I discovered that this can segfault unless 'remote-disconnect' and possibly 'repository-close!' are called *after* copying the data out. I've attached a diff for this. > + > +;;; Updater > + > +(define %pre-release-words > + '("alpha" "beta" "rc" "dev" "test")) I found a few packages that use "pre" as well. > + > +(define %pre-release-rx > + (map (cut make-regexp <> regexp/icase) %pre-release-words)) > + > +(define* (version-mapping tags #:key prefix suffix delim pre-releases?) > + "Given a list of Git TAGS, return a association list where the car is the ^ an > +version corresponding to the tag, and the cdr is the name of the tag." > + (define (guess-delimiter) > + (let ((total (length tags)) > + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) > + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) > + (underscores (reduce + 0 (map (cut string-count <> #\_) tags)))) > + (cond > + ((>= dots (* total 0.35)) ".") > + ((>= dashes (* total 0.8)) "-") > + ((>= underscores (* total 0.8)) "_") > + (else "")))) > + > + (define delim-rx (regexp-quote (or delim (guess-delimiter)))) > + (define suffix-rx (string-append (or suffix "") "$")) > + > + > + (define prefix-rx (string-append "^" (or prefix "[^[:digit:]]*"))) > + (define pre-release-rx > + (if pre-releases? > + (string-append ".*(" (string-join %pre-release-words "|") ").*") > + "")) > + > + (define tag-rx > + (string-append prefix-rx "([[:digit:]][^" delim-rx "[:punct:]]*" > + "(" delim-rx "[^[:punct:]" delim-rx "]+)" > + ;; If there is are no delimiters, it could mean that the ^ no "is" > + ;; version just contains one number (e.g., "2"), thus, use > + ;; "*" instead of "+" to match zero or more numbers. > + (if (string=? delim-rx "") "*" "+") Good catch. > + pre-release-rx ")" suffix-rx)) > + > + (define (get-version tag) > + (let ((tag-match (regexp-exec (make-regexp tag-rx) tag))) > + (and tag-match > + (regexp-substitute/global > + #f delim-rx (match:substring tag-match 1) > + ;; Don't insert "." if there aren't any delimiters in the first Nit: "if there were no delimiters", to be consistent with above comment. > + ;; place. > + 'pre (if (string=? delim-rx "") "" ".") 'post)))) One issue with returning a different delimiter than the package currently uses is that the automatic updater won't really work as-is. Hmmm. When things are modified so the updater gets both the version and the git-reference, it should be able to reverse-engineer things well enough there. I imagine this is really only going to be an issue with dates currently written as "2017-01-01", anyway. I'll put my comments on that in reply to the other email. > + > + (define (entry + (eq? (version-compare (car a) (car b)) '<)) > + > + (stable-sort (filter-map (lambda (tag) > + (let ((version (get-version tag))) > + (and version (cons version tag)))) > + tags) > + entry + > +(define* (latest-tag url #:key prefix suffix delim pre-releases?) > + "Return the latest tag available from the Git repository at URL." This returns two values (in preparation for the above-mentioned switch), so maybe something like "Return the latest version and corresponding tag available from..." > + (define (pre-release? tag) > + (any (cut regexp-exec <> tag) > + %pre-release-rx)) > + > + (let* ((tags (map (cut string-drop <> (string-length "refs/tags/")) Should be "cute" so string-length is only evaluated once -- though it's probably optimized like that anyway. > + (remote-refs url #:tags? #t))) > + (versions->tags > + (version-mapping (if pre-releases? > + tags > + (filter (negate pre-release?) tags)) > + #:prefix prefix > + #:suffix suffix > + #:delim delim > + #:pre-releases? pre-releases?))) > + (cond > + ((null? tags) > + (git-no-tags-error)) > + ((null? versions->tags) > + (git-no-valid-tags-error)) > + (else > + (match (last versions->tags) > + ((version . tag) > + (values version tag))))))) > + > +(define (latest-git-tag-version package) > + "Given a PACKAGE, return the latest version of it, or #f if the latest version > +could not be determined." > + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) > + (warning (or (package-field-location package 'source) > + (package-location package)) > + (G_ "~a for ~a~%") > + (condition-message c) > + (package-name package)) > + #f) > + ((eq? (exception-kind c) 'git-error) > + (warning (or (package-field-location package 'source) > + (package-location package)) > + (G_ "failed to fetch Git repository for ~a~%") > + (package-name package)) > + #f)) > + (let* ((source (package-source package)) > + (url (git-reference-url (origin-uri source))) > + (properties (package-properties package)) > + (tag-prefix (assq-ref properties 'release-tag-prefix)) > + (tag-suffix (assq-ref properties 'release-tag-suffix)) > + (tag-version-delimiter (assq-ref properties 'release-tag-version-delimiter)) > + (refresh-pre-releases? (assq-ref properties 'accept-pre-releases?))) > + (latest-tag url > + #:prefix tag-prefix > + #:suffix tag-suffix > + #:delim tag-version-delimiter > + #:pre-releases? refresh-pre-releases?)))) This is entirely a style preference, so only take this suggestion if you like it :) (let* ((source (package-source package)) (url (git-reference-url (origin-uri source))) (property (cute assq-ref (package-properties package) <>))) (latest-tag url #:prefix (property 'release-tag-prefix) #:suffix (property 'release-tag-suffix) #:delim (property 'release-tag-version-delimiter) #:pre-releases? (property 'accept-pre-releases?))))) > + > +(define (git-package? package) > + "Whether the origin of PACKAGE is a Git repostiory." "Return true if PACKAGE is..." > + (match (package-source package) > + ((? origin? origin) > + (and (eq? (origin-method origin) git-fetch) > + (git-reference? (origin-uri origin)))) > + (_ #f))) > + > +(define (latest-git-release package) > + "Return the latest release of PACKAGE." "Return an for the latest...", to match the other updaters. > + (let* ((name (package-name package)) > + (old-version (package-version package)) > + (url (git-reference-url (origin-uri (package-source package)))) > + (new-version (latest-git-tag-version package))) > + > + (and new-version > + (upstream-source > + (package name) > + (version new-version) > + (urls (list url)))))) > + > +(define %generic-git-updater > + (upstream-updater > + (name 'generic-git) > + (description "Updater for packages hosted on Git repositories") > + (pred git-package?) > + (latest latest-git-release))) I tested this updater on all packages in .scm files starting with f through z, and I found the following packages with possibly bogus updates: --8<---------------cut here---------------start------------->8--- javaxom luakit ocproxy pitivi eid-mw libhomfly gnuradio welle-io racket-minimal milkytracker cl-portal kodi-cli openjdk java-bouncycastle hurd opencsg povray gpsbabel go stepmania ocaml-mcl many minetest packages (minetest will have its own updater, though) ocaml4.07-core-kernel, ocamlbuild and many other ocaml packages (they seem to be covered by the github updater) --8<---------------cut here---------------end--------------->8--- The following packages suggest a version -> date update, which may or may not be bogus: --8<---------------cut here---------------start------------->8--- cataclysm-dda autotrace lbalgtk nheko libqalculate cl-antik cl-antik-base cl-hu.dwim.stefil cl-stefil cl-gsll sbcl-cl-gserver --8<---------------cut here---------------end--------------->8--- -- Sarah