From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id /MKoL/cHNmEbOAAAgWs5BA (envelope-from ) for ; Mon, 06 Sep 2021 14:22:15 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id KBwOK/cHNmElLQAA1q6Kng (envelope-from ) for ; Mon, 06 Sep 2021 12:22:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8C6A1140F3 for ; Mon, 6 Sep 2021 14:22:15 +0200 (CEST) Received: from localhost ([::1]:50940 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mNDde-0006z5-8K for larch@yhetil.org; Mon, 06 Sep 2021 08:22:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45824) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNDcU-0005YH-30 for guix-patches@gnu.org; Mon, 06 Sep 2021 08:21:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:40734) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNDcT-00066d-QA for guix-patches@gnu.org; Mon, 06 Sep 2021 08:21:01 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mNDcT-0007lL-Li for guix-patches@gnu.org; Mon, 06 Sep 2021 08:21:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50359] [PATCH] import: Add 'generic-git' updater. Resent-From: Xinglu Chen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 06 Sep 2021 12:21:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50359 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Sarah Morgensen Cc: 50359@debbugs.gnu.org Received: via spool by 50359-submit@debbugs.gnu.org id=B50359.163093085329818 (code B ref 50359); Mon, 06 Sep 2021 12:21:01 +0000 Received: (at 50359) by debbugs.gnu.org; 6 Sep 2021 12:20:53 +0000 Received: from localhost ([127.0.0.1]:52280 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNDcK-0007kr-Kz for submit@debbugs.gnu.org; Mon, 06 Sep 2021 08:20:53 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:37942 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNDcI-0007ka-Ag for 50359@debbugs.gnu.org; Mon, 06 Sep 2021 08:20:51 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1630930837; bh=rTeAfqqkrZ8uCrLD42TmYnmw6Zfc0mWDeJW9zRplpEc=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=MYYVUOoe+DrZ96dkt00vEYMFEyxSINEtCD0m6tE9lhiRsm8VlNqbqiwK2gv9ooEVL OTFRKDlomoQ8fgVPqa4Aa/k+tTnu2tQFbO4GngHrLonWRvNWg20F1hGOi7bGdIBzS8 hTGdJsdFudoC2F/mSQqDu+WCMDT9WzKcjQD9cP/g= In-Reply-To: <86y28ai7ns.fsf@mgsn.dev> References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> Date: Mon, 06 Sep 2021 14:20:34 +0200 Message-ID: <87y28928vh.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -4.00 Authentication-Results: aspmx1.migadu.com; none X-Migadu-Queue-Id: 8C6A1140F3 X-Spam-Score: -4.00 X-Migadu-Scanner: scn1.migadu.com X-TUID: nRb5bvJE8gbE --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Sun, Sep 05 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >>> There are about 50-60 packages like this. >>> >>> I'm not sure how much effort should be spent including them, and for >>> some of them I'm not sure what our ideal behavior *is*. Even if we >>> could reliably detect them, should "alpha" or "dev" packages be returned >>> by the updater? >> >> I don=E2=80=99t think we usually include alpha or rc releases, so updater >> probably shouldn=E2=80=99t return them either. Not sure how we would tr= y to >> detect alpha/beta/rc releases, though, besides running something like >> >> (string-match? "(alpha|beta|rc|dev)" TAG) >> >> On each tag. > > That heuristic is pretty good. (It might miss a few, but I'd rather > accidentally include some alpha/beta/rc releases than risk excluding > real ones.) > > We could then safely sort tags with just the prefix removed -- this takes= care > of "1.1f" coming after "1.1", and so on. > > Actually, it looks like there's only a few packages with a suffix; > instead, we can probably just only use a suffix if they provide > 'tag-suffix. This is all of them (and the "dev" ones probably shouldn't > be suffixes, just part of the version): Yeah, I think that sounds like a reasonable thing to do. > (commit (string-append "v" version "-stable")))) > (commit (string-append version "-stable")))) > (commit (string-append version "-Leia")))) > (commit (string-append "haddock-" version "-release")))) > (commit (string-append "v" version "-8.13")))) > (commit (string-append "v" version "-oss")))) > (commit (string-append "v" version "-stable")))) > (commit (string-append "ddskk-" version "_" code-name)))) > (commit (string-append version "-freebsdport")))) > (commit (string-append version "-dev")))) > (commit (string-append version "-release-20210531143054")))) > (commit (string-append version "-release-20210412001032")))) > (commit (string-append "v" version "-debian")))) > (commit (string-append version "dev")))) > (commit (string-append version "_Linux")) > (commit (string-append version "R")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append version "-opt")))) > (commit (string-append "1.1-" version "-RELEASE")))) > > Additionally, these are all the weird version strings I could find that > are actually used as the commit: > 1.2.2.rc2 > 12-068oasis4 > 1.2-2 > 0.F-2 > 0.2.0-alpha > 5.1.0-b2 > 1.5-11 > 1.9.14-20210407 > 0.9.3b > 2.8-fix-2 > 2020-05-19 > 10-11.0.0 > 1.0.12-2 > 0.9.3+16.04.20160218-0ubuntu1 > 2.12.c > 1.1+11 > 5.1+4.06.0 > 4.2-411 > 1.4.0rc1-450-g2725ef99d > 2.0.0-alpha14 > 60.2.3-2 > 1.21c > 1.0.0rc4 > 2.1.0b1 > 1.02r6 > 0.32-14-gcdfe14e > 3.0.0a3 > 2.00a2.3 > 1.0beta.18 > 2.1b > 2.7.8a > 2.7.3a > 1.1.alpha19 > 1.0.2-rc4 > 0.5.3+git20200502 > 1.0.3-rc3 > 4.0.0.dev8 > 3.0.0beta1-24-g024cc9fa2 > 0.16-2-ge145396 > 2.0b6 > 2.0M10 > 1.0.7+0 > 1.16.0+5 > 0.4.0+1 > 2.2.10+0 > 4.3.1+2 > 2.13.93+0 > 2.10.4+0 > 1.0.5+5 > 0.21.0+0 > 3.3.4+0 > 2.68.1+0 > 0.10.1+1 > 6.9.10-12+3 > 2.0.1+2 > 3.100.0+1 > 0.14.0+2 > 0.1.6+2 > 3.3.0+0 > 1.8.7+0 > 1.3.0+2 > 1.42.0+0 > 1.16.1+0 > 2.35.0+0 > 1.6.37+5 > 4.1.0+1 > 2.36.0+0 > 1.3.6+4 > 2.10.1+0 > 2.26.0+0 > 1.3.4+0 > 1.1.1+2 > 1.3.1+1 > 8.44.0+0 > 0.40.1+0 > 5.15.2+0 > 1.17.0+3 > 1.18.0+3 > 2020.7.14+0 > 3.0.0+1 > 0.9.1+3 > 2.9.12+0 > 0.1.0+2 > 1.6.9+2 > 1.0.9+3 > 1.13.0+2 > 1.2.0+3 > 1.1.3+3 > 1.3.4+2 > 5.0.3+3 > 1.7.10+3 > 1.1.4+3 > 1.1.0+3 > 1.5.2+3 > 0.9.10+3 > 0.4.0+1 > 0.4.0+1 > 0.4.0+1 > 0.3.9+1 > 0.4.1+1 > 1.4.2+3 > 2.27.0+3 > 1.4.0+2 > 1.1.34+0 > 1.2.12+1 > 1.5.0+0 > 2021-06-07 > 1.2.2-5-g20dc8ed > 2.1-20201229 > 3.0-rc1 > 1.32.0-0 > 20200701.154658.b0d6223 > 3.9-0 > R63-10032.B > 0.58.2.a > 0.0.0+git20200527 > 3.028R > 3.001R > 2.57b > 1.0.0-20201130134442-10cb98267c6c > 0.0.0-20161123171359-e6a2ba005892 > 0.0.0-20210615171337-6886f2dfbf5b > 2020-11-10 > 5.2.0-alpha > 2021-01-01 > 2021-02-28 > 2020-11-01 > 4.4-git.1 > 1.217-2 > 3.3.06-1 > 0.2.0-alpha-199-g3e7a475 > 1.0.0-beta.0 > 1.20190621-4 > 1.9.0-147-g61edec1ef > 0.0.9.4f > > Yep, it looks like the above would work for the majority of these. > That's probably Good Enough^tm. Any trick you used to find all of there weird version numbers? :-) >>>> + (define no-delims >>>> + (if (string-match delim-regexp no-suffix) >>>> + (string-split* no-suffix delim*) >>>> + (git-tag-error 'tag-version-delimiter))) >>> >>> This throws an error if the version doesn't have any delimiter. >> >> Setting the =E2=80=98tag-version-delimiter=E2=80=99 prefix to an empty s= tring would >> solve this, right? Or, maybe we should just get rid of the delimiter >> thing since only a few packages use a different delimiter. > > IMO, just get rid of the delimiter. If we wanted to be *that* flexible, > we could make it so they provide a tag->version proc instead of (prefix, > suffix, delimiter). a =E2=80=98tag->version=E2=80=99 procedure would probably make things a bit= too complicated for the people writing package definitions. For example, having a delimiter would make it easy to match a tag like =E2=80=9C2021-01-01-release=E2=80=9D Delimiter is =E2=80=9C.=E2=80=9D (sorry if this hurts your eyes ;-)) scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:punct:= ]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1) $28 =3D "2021" Delimiter is =E2=80=9C-=E2=80=9D scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punct:]]= +(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1) $29 =3D "2021-01-01-release" And then, setting the suffix to =E2=80=9C-release=E2=80=9D would match just= the version part. On Sun, Sep 05 2021, Sarah Morgensen wrote (again): >>> And this lets us just do something like (untested): >>> >>> (define* (get-version tag #:key prefix suffix delim) >>> (define delim-rx (regexp-quote (or delim "."))) >>> (define prefix-rx (or prefix "[^[:digit:]]*")) >>> (define suffix-rx (or suffix ".*")) >>> (define version-char-rx >>> (string-append "[^" delim-rx "[:punct:]]")) >>> >>> (define tag-rx >>> (string-append "^" prefix "(" version-char-rx "+(" >>> delim-rx version-char-rx ")*)" suffix-rx "$")) >> >> This wouldn=E2=80=99t match anything if the version is just a plain numb= er, >> e.g., 1 or 09. > > It does, Oh, I missed the extra pair of parens, sorry about that. > but I had many errors in the definition. Again, apologies. I > shouldn't send emails that late, haha. This method should read: > > --8<---------------cut here---------------start------------->8--- > (define* (get-version tag #:key prefix suffix delim) > (define delim-rx (regexp-quote (or delim "."))) > (define prefix-rx (or prefix "[^[:digit:]]*")) > (define suffix-rx (or suffix ".*")) > (define version-char-rx > (string-append "[^" delim-rx "[:punct:]]")) > > (define tag-rx > (string-append "^" prefix-rx "(" version-char-rx "+(" > delim-rx version-char-rx "+)*)" suffix-rx "$")) > (and=3D> (string-match tag-rx tag) > (cut match:substring <> 1))) > --8<---------------cut here---------------end--------------->8--- > >> >> With this, something like =E2=80=9C1.4.0rc1-450-g2725ef99d=E2=80=9D will= result in >> =E2=80=9C1.4.0=E2=80=9D being returned, which is incorrect. Changing (c= ut >> match:substring <> 1) to just =E2=80=98match:substring=E2=80=99 would so= lve the issue, >> but then pre-release tags, which we usually don=E2=80=99t want, would a= lso get >> matched. Not sure what the best option would be in this case. >> > > With the fixed method above: > > scheme@(emacs-guix)> (get-version "8") > $16 =3D "8" > scheme@(emacs-guix)> (get-version "1.4.0rc1-450-g2725ef99d") > $17 =3D "1.4.0rc1" > > But, we still get: > > scheme@(emacs-guix)> (get-version "1.4.0-rc1") > $18 =3D "1.4.0" > > which leads us to what you talked about in your other message. Hmm, maybe we could check that if (string-append "1.4.0" suffix) is a suffix on of the tag, or is this too much overhead? This would only work if we set the suffix to the empty string by default. Since (string-match "1.4.0$" "1.4.0-rc1") returns #f, this tag would get filtered out. But then a suffix would have to be specified to match "1.4.0rc1-450-g2725ef99d". Not sure what the best option is here. >>> +(define* (ls-remote-refs url #:key tags?) >>> + "Return the list of references advertised at Git repository URL. If= TAGS? >>> +is true, limit to only refs/tags." >>> + (define (ref? ref) >>> + ;; Like `git ls-remote --refs', only show actual references. >>> + (and (string-prefix? "refs/" ref) >>> + (not (string-suffix? "^{}" ref)))) >>> + >>> + (define (tag? ref) >>> + (string-prefix? "refs/tags/" ref)) >>> + >>> + (define (include? ref) >>> + (and ref? > > This should be: > (and (ref? ref) Ah, problem solved! :-) >>> + (or (not tags?) (tag? ref)))) >>> + >>> + (with-libgit2 >>> + (with-temporary-directory >>> + (lambda (cache-directory) >>> + (let* ((repository (repository-init cache-directory)) >>> + ;; Create an in-memory remote so we don't touch disk. >>> + (remote (remote-create-anonymous repository url))) >>> + (remote-connect remote) >>> + (remote-disconnect remote) >>> + (repository-close! repository) >>> + >>> + (filter include? (map remote-head-name (remote-ls remote))))))= )) >>> >> >> For some reason it seems to include refs that do and don=E2=80=99t end w= ith >> =E2=80=9C^{}=E2=80=9D > > Sorry, another typo I missed. See above. No worries, thanks for taking the time to review this! --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE2B5IVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5BgIQAIZHtTauloSjYVC1T6cs8epK0r8B gUw3Dk3ZdBbjd9TfpfjsUnaGN2o6KwBKTnpj5atLPiM2Vo5an8BLmSOTlJQ9+O8p DBiKYY0kKdknJYhboSIKj/ejCRIkdLQy6Wj6ucWmYmxipqk+muBO0c5qoMsIYHFp iLUH0rrT7R0PMkBOSGrGgLi5Y4RRBlM9O4QqMp4RZ5Vw+brn6o3Z/+ja9/wlPSl9 md9W0g3D8d7eTmu/ATW71GDaI3UndGOUeCW4X3uTV0SqPUkyFuLKFFsqiQMxZHe1 iQ+uu1vj6shpxa+tXvpzqT3CdPDRzU8UyCtPNCUc8TaoLLj+SRlsdGv/xe1DSRYa lZhX1kxkUdJxpAjKuyU4i9N2b0Wftbmsfg4PATQJgP/XbY9RiSp6pXNY0uKNIhLF CBEoC5cq3MDjmLMswaipdlcVG4a5gxzhGI37aMBpE/of9hCDlY+rxth02m4nr4Rf 6SMeXcwYlcLNTU8FRw90d6aE+8NJxH4XR5VOW/CvZw6aArzYE3lwUvyYmFD78xfu 9LfehCH0DR3q98GCabyswVxpMrNdv8Z+wVNlpF+kDnCDzcD9vIhjKU6RTkzjp1UN zc5hQv4VjEqTs2hVSjf04X9L0s6gO2TRjpFJtquPq+GPqnozLJBuzIt+v0MRj/AH DF2EcH0wmD9yKpdI =l3fr -----END PGP SIGNATURE----- --=-=-=--