From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id kBNIGR2NG2HOIQAAgWs5BA (envelope-from ) for ; Tue, 17 Aug 2021 12:19:09 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 6IUTFR2NG2GVYQAAB5/wlQ (envelope-from ) for ; Tue, 17 Aug 2021 10:19:09 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1B43F25C1 for ; Tue, 17 Aug 2021 12:19:09 +0200 (CEST) Received: from localhost ([::1]:54208 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFwBY-0000G0-6z for larch@yhetil.org; Tue, 17 Aug 2021 06:19:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51206) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFwBS-0000Fm-Dd for guix-patches@gnu.org; Tue, 17 Aug 2021 06:19:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:40210) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mFwBS-0000rp-53 for guix-patches@gnu.org; Tue, 17 Aug 2021 06:19:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mFwBR-0003ti-IS for guix-patches@gnu.org; Tue, 17 Aug 2021 06:19:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50072] [PATCH WIP 0/4] Add upstream updater for git-fetch origins. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 17 Aug 2021 10:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50072 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Sarah Morgensen Cc: Xinglu Chen , 50072@debbugs.gnu.org Received: via spool by 50072-submit@debbugs.gnu.org id=B50072.162919552014956 (code B ref 50072); Tue, 17 Aug 2021 10:19:01 +0000 Received: (at 50072) by debbugs.gnu.org; 17 Aug 2021 10:18:40 +0000 Received: from localhost ([127.0.0.1]:51756 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mFwB6-0003tA-26 for submit@debbugs.gnu.org; Tue, 17 Aug 2021 06:18:40 -0400 Received: from andre.telenet-ops.be ([195.130.132.53]:52772) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mFwB2-0003sz-Ia for 50072@debbugs.gnu.org; Tue, 17 Aug 2021 06:18:38 -0400 Received: from ptr-bvsjgyjmffd7q9timvx.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:aaf1:9810:a0b8:a55d]) by andre.telenet-ops.be with bizsmtp id iaJa2500Q0mfAB401aJarM; Tue, 17 Aug 2021 12:18:35 +0200 Message-ID: <7986923ce7712dc341e859e62675abee12072922.camel@telenet.be> From: Maxime Devos Date: Tue, 17 Aug 2021 12:18:28 +0200 In-Reply-To: <86fsv9jh8h.fsf_-_@mgsn.dev> References: <8d1ae518b23fac5b15812a30b11df1c360ab3fbf.1629068119.git.iskarian@mgsn.dev> <86fsv9jh8h.fsf_-_@mgsn.dev> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-AH/+dNebwLIhoKM5nARb" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r21; t=1629195515; bh=jp/2pHV0Tj+XdZ8epX9Lk/Ju5XEcEvgfdciAka6dwoU=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=DEJu7rOhh2GLqJb3l5lCTbNpl9SmQOrPT0yc6Z8ssoxV3lMjlsPxijxAAH0fuVxfs c1O31sv+9JHVaDfjrV2pLAcpiXzW5P4foK4Unth8V4bzbPXPGw5cvLQaWck6giBv/u UMisq9UdhMgFogonJ737Tpp0Zm3+5KBUvaojYkLYkbTZvWdJWXzAqLvEBAenBcEx/W qA5s5MZPvbQZ+EiL/HZVPcJWv57cXI/nYEYcfYesdPywFi7zl7hAc45mccTCBFAQ2L J9GilvNOpVVUb0Gx+FJ4HvJI5RLIsMiD3lhCkVtC+47LGJ+4fV+RFlXzlVYRXtAhdo radGMTVcyaX7w== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -4.00 Authentication-Results: aspmx1.migadu.com; none X-Migadu-Queue-Id: 1B43F25C1 X-Spam-Score: -4.00 X-Migadu-Scanner: scn1.migadu.com X-TUID: UkdpWZ+djNHH --=-AH/+dNebwLIhoKM5nARb Content-Type: multipart/mixed; boundary="=-yiIn7AhCV0N/+pc8ogPA" --=-yiIn7AhCV0N/+pc8ogPA Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sarah Morgensen schreef op ma 16-08-2021 om 12:56 [-0700]: > Hi Maxime, >=20 > Thanks for taking a look at this. :) >=20 > Maxime Devos writes: >=20 > > Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]: > > > * guix/git-download.scm (checkout-to-store): New procedure. > > > * guix/upstream.scm (guess-version-transform) > > > (package-update/git-fetch): New procedures. > > > (%method-updates): Add GIT-FETCH mapping. > >=20 > > Does it support packages defined like (a) > >=20 > > (define-public gnash > > (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4") > > (revision "0")) > > (package > > (name "gnash") > > (version (git-version "0.8.11" revision commit)) > > (source (git-reference > > (url "https://example.org") > > (commit commit))) > > [...]))) >=20 > No, it doesn't. Since the commit definition isn't part of the actual > package definition, the current code has no way of updating it. It > would require a rewrite of the edit-in-place logic with probably a lot > of special-casing. Perhaps a 'surrounding-expression-location' procedure can be defined? (define (surrounding-expression-location inner-location) "Determine the location of the S-expression that surrounds the S-expressi= on at INNER-LOCATION, or #false if the inner S-expression is at the top-level.= " ??? Something like 'read', but in reverse, maybe? Doesn't need to support every construct, just "string without escapes" an= d (parentheses other-things) might be good enough in practice for now) Seems tricky to implement, but it would be more robust than relying on conventions like =E2=80=98the surrounding 'let' can be found by moving t= wo columns and two lines backwards=E2=80=99. Or see another method (let&) below that = is actually implemented ... > There are currently ~1250 package which use this format, though, so it > could be worth it... Perhaps what we actually need is a better idiom to > express this situation. Package properties ('git-commit)? A 'git-versio= n*'? >=20 > --8<---------------cut here---------------start------------->8--- > (define (git-version* version revision) > (let* ((source (package-source this-package)) > (commit (git-reference-commit (origin-uri source)))) > (git-version version revision commit))) > --8<---------------cut here---------------end--------------->8--- >=20 > I'm not sure if binding order would be an issue with that. The 'file-name' field of 'origin' is not thunked, and refers to the 'versio= n' field of the 'package' (also not thunked). If 'version' would use the 'git= -version*' from above, then there would be a loop (I'm having the 'gnash' package in m= ind, see "guix edit gnash"). And git-version* cannot be a procedure, it must be= a macro, as it used 'this-package', which can only be expanded inside a package defi= nition. Alternatively, what do you think of a let& macro, that adjusts the inner ex= pression to have the source location of the 'let&' form: (define-syntax with-source-location (lambda (s) (syntax-case s () ((_ (exp . exp*) source) "Expand to (EXP . EXP*), but with the source location replaced by the source location of SOURCE." (datum->syntax s (cons #'exp #'exp*) #:source (syntax-source #'sourc= e)))))) (define-syntax let& (lambda (s) "Like 'let', but let the inner expression have the location of the 'let&' form when it is expanded. Only a single inner expression is allowed." (syntax-case s () ((_ bindings exp) #'(let bindings (with-source-location exp s)))))) That way, 'update-package-source' doesn't need to know about the surroundin= g 'let' form; it would simply use 'edit-expression' as usual (though somethin= g like ,@(if (and old-commit new-commit) `((,old-commit . ,new-commit)) '()) would need to be added, and something to replace =E2=80=98(revision "N")=E2= =80=99 with =E2=80=98(revision "N+1")=E2=80=99.) A complete example is attached (a.scm). The previous usages of (let ((commit ...) (revision ...)) ...) would need to be adjusted to use let& instead (build-aux/update-guix-package.scm needs to be adjusted as well). Personally, I'd go with the 'let&' form > > and (b) > >=20 > > (define-public gnash > > (package > > (name "gnash") > > (version "0.8.11") > > (source (git-reference > > (url "https://example.org") > > (commit commit)) > > [...])) > > ? >=20 > Is this missing a definition for commit? If it's like above, the same > applies. Or if you mean >=20 > --8<---------------cut here---------------start------------->8--- > (source (git-reference > (url "https://example.org") > (commit "583ccbc1275c7701dc4843ec12142ff86bb305b")) > --8<---------------cut here---------------end--------------->8--- The latter. > Then that wouldn't be too hard to support. There seem to be ~136 > packages with this idiom. FWIW, the patch I sent modified 'update-package-source' to replace the commit in this case (b) (but not case (a)). > > [the patch Maxime sent] > >=20 > > upstream-source? > > (package upstream-source-package) ;string > > (version upstream-source-version) ;string > > - (urls upstream-source-urls) ;list of strings > > + ; list of strings or a > > + (urls upstream-source-urls) >=20 > Is it possible for an updater to want to return a list of > ? No, 'git-fetch' from (guix git-download) only accepts a single object, it doesn't support lists of . It will throw a type error if a list is passed. Compare with 'url-fetch*', which does accept a = list of URLs (in which case it will fall-back to the second, the third, the four= th ... entry when the first entry gives a 404 or something). > I'm still not sure what the purpose of multiple urls > is, since nearly everthing seems to just take (first urls)... As I understand it, the second, third, fourth ... URL (when using url-fetch= ) are fall-backs. Also, (guix upstream) sometimes distinguishes between the different URLs, see e.g. package-update/url-fetch, which will try to choose= a tarball with the same kind of extension (.zip, .tar.gz, .tar.xz, ...) as th= e original URI. > > (signature-urls upstream-source-signature-urls ;#f | list of string= s > > (default #f)) > > (input-changes upstream-source-input-changes > > @@ -361,6 +368,11 @@ values: 'interactive' (default), 'always', and 'ne= ver'." > > system target) > > "Download SOURCE from its first URL and lower it as a fixed-output > > derivation that would fetch it." > > + (define url > > + (match (upstream-source-urls source) > > + ((first . _) first) > > + (_ (raise (formatted-message > > + (G_ "git origins are unsupported by --with-latest")))= ))) > > (mlet* %store-monad ((url -> (first (upstream-source-urls source))) > > (signature > > -> (and=3D> (upstream-source-signature-urls so= urce) > > @@ -430,9 +442,23 @@ SOURCE, an ." > > #:key-download key-download))) > > (values version tarball source)))))) >=20 > What is this 'upstream-source-compiler' actually used for? I couldn't > figure that out, so I just left it untouched. It is used to =E2=80=98lower=E2=80=99 objects. More spec= ifically, transform-package-latest from (guix transformations) will sometimes replace the 'source' of a package with a object, and 'upstream-source-compiler' is used to turn the into a (fixed-output) derivation that can be built into a /gnu/store/...-checkout or /gnu/store/...-version.tar.gz file in the store. Greetings, Maxime --=-yiIn7AhCV0N/+pc8ogPA Content-Disposition: inline; filename="a.scm" Content-Type: text/x-scheme; name="a.scm"; charset="UTF-8" Content-Transfer-Encoding: base64 Ozs7IEdOVSBHdWl4IC0tLSBGdW5jdGlvbmFsIHBhY2thZ2UgbWFuYWdlbWVudCBmb3IgR05VCjs7 OyBDb3B5cmlnaHQgwqkgMjAyMSBNYXhpbWUgRGV2b3MgPG1heGltZWRldm9zQHRlbGVuZXQuYmU+ Cjs7Owo7OzsgVGhpcyBmaWxlIGlzIHBhcnQgb2YgR05VIEd1aXguCjs7Owo7OzsgR05VIEd1aXgg aXMgZnJlZSBzb2Z0d2FyZTsgeW91IGNhbiByZWRpc3RyaWJ1dGUgaXQgYW5kL29yIG1vZGlmeSBp dAo7OzsgdW5kZXIgdGhlIHRlcm1zIG9mIHRoZSBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZSBh cyBwdWJsaXNoZWQgYnkKOzs7IHRoZSBGcmVlIFNvZnR3YXJlIEZvdW5kYXRpb247IGVpdGhlciB2 ZXJzaW9uIDMgb2YgdGhlIExpY2Vuc2UsIG9yIChhdAo7OzsgeW91ciBvcHRpb24pIGFueSBsYXRl ciB2ZXJzaW9uLgo7OzsKOzs7IEdOVSBHdWl4IGlzIGRpc3RyaWJ1dGVkIGluIHRoZSBob3BlIHRo YXQgaXQgd2lsbCBiZSB1c2VmdWwsIGJ1dAo7OzsgV0lUSE9VVCBBTlkgV0FSUkFOVFk7IHdpdGhv dXQgZXZlbiB0aGUgaW1wbGllZCB3YXJyYW50eSBvZgo7OzsgTUVSQ0hBTlRBQklMSVRZIG9yIEZJ VE5FU1MgRk9SIEEgUEFSVElDVUxBUiBQVVJQT1NFLiAgU2VlIHRoZQo7OzsgR05VIEdlbmVyYWwg UHVibGljIExpY2Vuc2UgZm9yIG1vcmUgZGV0YWlscy4KOzs7Cjs7OyBZb3Ugc2hvdWxkIGhhdmUg cmVjZWl2ZWQgYSBjb3B5IG9mIHRoZSBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZQo7OzsgYWxv bmcgd2l0aCBHTlUgR3VpeC4gIElmIG5vdCwgc2VlIDxodHRwOi8vd3d3LmdudS5vcmcvbGljZW5z ZXMvPi4KCih1c2UtbW9kdWxlcyAoZ3VpeCBwYWNrYWdlcykKICAgICAgICAgICAgIChnbnUgcGFj a2FnZXMgYW5pbWF0aW9uKQogICAgICAgICAgICAgKGd1aXggZ2l0LWRvd25sb2FkKSkKCihkZWZp bmUtc3ludGF4IHdpdGgtc291cmNlLWxvY2F0aW9uCiAgKGxhbWJkYSAocykKICAgIChzeW50YXgt Y2FzZSBzICgpCiAgICAgICgoXyAoZXhwIC4gZXhwKikgc291cmNlKQogICAgICAgIkV4cGFuZCB0 byAoRVhQIC4gRVhQKiksIGJ1dCB3aXRoIHRoZSBzb3VyY2UgbG9jYXRpb24gcmVwbGFjZWQKYnkg dGhlIHNvdXJjZSBsb2NhdGlvbiBvZiBTT1VSQ0UuIgogICAgICAgKGRhdHVtLT5zeW50YXggcyAo Y29ucyAjJ2V4cCAjJ2V4cCopICM6c291cmNlIChzeW50YXgtc291cmNlICMnc291cmNlKSkpKSkp CgooZGVmaW5lLXN5bnRheCBsZXQmCiAgKGxhbWJkYSAocykKICAgICJMaWtlICdsZXQnLCBidXQg bGV0IHRoZSBpbm5lciBleHByZXNzaW9uIGhhdmUgdGhlIGxvY2F0aW9uCm9mIHRoZSAnbGV0Jicg Zm9ybSB3aGVuIGl0IGlzIGV4cGFuZGVkLiAgT25seSBhIHNpbmdsZSBpbm5lcgpleHByZXNzaW9u IGlzIGFsbG93ZWQuIgogICAgKHN5bnRheC1jYXNlIHMgKCkKICAgICAgKChfIGJpbmRpbmdzIGV4 cCkKICAgICAgICMnKGxldCBiaW5kaW5ncwogICAgICAgICAgICh3aXRoLXNvdXJjZS1sb2NhdGlv biBleHAgcykpKSkpKQoKCihkZWZpbmUtcHVibGljIGduYXNoMgogIDs7IFRoZSBsYXN0IHRhZ2dl ZCByZWxlYXNlIG9mIEduYXNoIHdhcyBpbiAyMDEzLgogIChsZXQmICgoY29tbWl0ICI1ODNjY2Jj MTI3NWM3NzAxZGM0ODQzZWMxMjE0MmZmODZiYjMwNWI0IikKICAgICAgICAgKHJldmlzaW9uICIw IikpCiAgICAocGFja2FnZQogICAgICAoaW5oZXJpdCBnbmFzaCkKICAgICAgKG5hbWUgImduYXNo MiIpCiAgICAgICh2ZXJzaW9uIChnaXQtdmVyc2lvbiAiMC44LjExIiByZXZpc2lvbiBjb21taXQp KQogICAgICAoc291cmNlCiAgICAgICAob3JpZ2luCiAgICAgICAgIChtZXRob2QgZ2l0LWZldGNo KQogICAgICAgICAodXJpIChnaXQtcmVmZXJlbmNlCiAgICAgICAgICAgICAgICh1cmwgImh0dHBz Oi8vZ2l0LnNhdmFubmFoLmdudS5vcmcvZ2l0L2duYXNoLmdpdC8iKQogICAgICAgICAgICAgICAo Y29tbWl0IGNvbW1pdCkpKQogICAgICAgICAoZmlsZS1uYW1lIChnaXQtZmlsZS1uYW1lIG5hbWUg dmVyc2lvbikpCiAgICAgICAgIChwYXRjaGVzIChzZWFyY2gtcGF0Y2hlcyAiZ25hc2gtZml4LWdp ZmxpYi12ZXJzaW9uLnBhdGNoIikpCiAgICAgICAgIChzaGEyNTYKICAgICAgICAgIChiYXNlMzIg IjBmaDBibGpuMGk2eXB5aDZsOTlhZmk4NTVwN2tpN2xtODY5bnExcWo2azhocnJ3aG1mcnkiKSkp KSkpKQoKKGZvcm1hdCAjdCAib2xkOiB+YX4lIiAocGFja2FnZS1sb2NhdGlvbiBnbmFzaCkpCihm b3JtYXQgI3QgIm5ldzogfmF+JSIgKHBhY2thZ2UtbG9jYXRpb24gZ25hc2gyKSkKOzsgXiBpdCBz YXlzIGNvbHVtbiAyLCB3aGljaCBpcyB0aGUgY29sdW1uIG9mIHRoZSBsZXQmIGZvcm0uCg== --=-yiIn7AhCV0N/+pc8ogPA-- --=-AH/+dNebwLIhoKM5nARb Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYRuM9BccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7sCHAP9xBj75KhXqSDRyCSSQFyATroZ3 K22ko8hK0JHF1DtoFAD/U7vAsQq8rQXjBlwMe2f+W0O55HD6OTDRwZ99pKRdwwo= =BfS0 -----END PGP SIGNATURE----- --=-AH/+dNebwLIhoKM5nARb--