From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id eAiCLxycamRLAAAASxT56A (envelope-from ) for ; Mon, 22 May 2023 00:33:00 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id UIdnLxycamTgFgEA9RJhRA (envelope-from ) for ; Mon, 22 May 2023 00:33:00 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8714D39241 for ; Mon, 22 May 2023 00:33:00 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q0r16-0002id-Ng; Sun, 21 May 2023 17:55:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q0r14-0002iD-82 for guix-patches@gnu.org; Sun, 21 May 2023 17:55:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1q0r14-0007Lz-0L for guix-patches@gnu.org; Sun, 21 May 2023 17:55:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1q0r13-0002LQ-P3 for guix-patches@gnu.org; Sun, 21 May 2023 17:55:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#63631] [PATCH] import: go: Handle subpackage versioning correctly. Resent-From: wolf Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 21 May 2023 21:55:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 63631 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Timo Wilken Cc: 54097@debbugs.gnu.org, 63631@debbugs.gnu.org X-Debbugs-Original-Cc: 54097@debbugs.gnu.org, guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16847060868898 (code B ref -1); Sun, 21 May 2023 21:55:01 +0000 Received: (at submit) by debbugs.gnu.org; 21 May 2023 21:54:46 +0000 Received: from localhost ([127.0.0.1]:32959 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q0r0n-0002JO-6Y for submit@debbugs.gnu.org; Sun, 21 May 2023 17:54:45 -0400 Received: from lists.gnu.org ([209.51.188.17]:48934) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q0r0l-0002J2-0I for submit@debbugs.gnu.org; Sun, 21 May 2023 17:54:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q0r0k-0002hV-MF for guix-patches@gnu.org; Sun, 21 May 2023 17:54:42 -0400 Received: from wolfsden.cz ([37.205.8.62]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q0r0i-0007Hu-Ok for guix-patches@gnu.org; Sun, 21 May 2023 17:54:42 -0400 Received: by wolfsden.cz (Postfix, from userid 104) id EBF9225A96C; Sun, 21 May 2023 21:54:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=wolfsden.cz; s=mail; t=1684706074; bh=7jZSQ6kPl4uZBrcrCMjCjeDmk7WiY/3V+1MatjIbtig=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=Av1Y3shEepVs1TcQ3YMvzcRZ9vOZQTy4Ib2LT8Vg2a4XBAhSvAJEFaDo+Orga4nbf 2hEzsxlKbC1UfDirH7Kry2VfN8NrqaFTlY8ujSbp/zhgJiqffKE2ZjLXubOKCVZCW9 yGFYIKIhI5WpHiAUKNzpKnTTAQpSNycOYKdaZpb3Yao6LuGEi3virJkDDMK06idZMP 7u0ZEN9fKycaIHnRXF6J2DPbYjMzka/yLyn7tW3wPHhguxImnyp3KU6nzFmEfZSxjv y1lhVWkPFEWFwN07HBJuLylQqsUEEey7pyKupxGk0WLFTOwVkBHlCWnxpdIqfM1LHB eEmH/7T8nUxR2vmdYrHMLvUnePRWF52NpQjFLoEGaF6FmIDYCX6ODnbR14T7SqDmIe ycsUjJK/9spVcP+iJrD3t6G9eJvlNr6VZrpnBUqOYG4VyqDtx1WP3x60zG/0kE7iiq Gptw8OBUVZgKVYJi7I/VhL7IP91Y+daMaRW/DI7K56ElPTf7X+mem+hmfbygp0Wg/P ipblJjidqrbYsmHRpNZ0o6TEWcNHuy+EnO/PHPVENVBmZKEu0t/c9LGbch7u9UC34e 8e4Vxc6cN55W7y+VFV6v/2cfTen/aj7JlP7VHcmXw6cN+IlaqspX4qnZayrgKspmw7 r3TwtDRDdtUzIzVuuTpWsv+c= Received: from localhost (unknown [128.0.188.242]) by wolfsden.cz (Postfix) with ESMTPSA id 0EFE125A9EB; Sun, 21 May 2023 21:54:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=wolfsden.cz; s=mail; t=1684706074; bh=7jZSQ6kPl4uZBrcrCMjCjeDmk7WiY/3V+1MatjIbtig=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=Av1Y3shEepVs1TcQ3YMvzcRZ9vOZQTy4Ib2LT8Vg2a4XBAhSvAJEFaDo+Orga4nbf 2hEzsxlKbC1UfDirH7Kry2VfN8NrqaFTlY8ujSbp/zhgJiqffKE2ZjLXubOKCVZCW9 yGFYIKIhI5WpHiAUKNzpKnTTAQpSNycOYKdaZpb3Yao6LuGEi3virJkDDMK06idZMP 7u0ZEN9fKycaIHnRXF6J2DPbYjMzka/yLyn7tW3wPHhguxImnyp3KU6nzFmEfZSxjv y1lhVWkPFEWFwN07HBJuLylQqsUEEey7pyKupxGk0WLFTOwVkBHlCWnxpdIqfM1LHB eEmH/7T8nUxR2vmdYrHMLvUnePRWF52NpQjFLoEGaF6FmIDYCX6ODnbR14T7SqDmIe ycsUjJK/9spVcP+iJrD3t6G9eJvlNr6VZrpnBUqOYG4VyqDtx1WP3x60zG/0kE7iiq Gptw8OBUVZgKVYJi7I/VhL7IP91Y+daMaRW/DI7K56ElPTf7X+mem+hmfbygp0Wg/P ipblJjidqrbYsmHRpNZ0o6TEWcNHuy+EnO/PHPVENVBmZKEu0t/c9LGbch7u9UC34e 8e4Vxc6cN55W7y+VFV6v/2cfTen/aj7JlP7VHcmXw6cN+IlaqspX4qnZayrgKspmw7 r3TwtDRDdtUzIzVuuTpWsv+c= Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id b10323c4; Sun, 21 May 2023 21:54:33 +0000 (UTC) Date: Sun, 21 May 2023 23:54:33 +0200 From: wolf Message-ID: References: <6dd1de3dd4d968876fa55f5126056834c77b0244.1684703258.git.guix@twilken.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="iw7hpULjExUreuIc" Content-Disposition: inline In-Reply-To: <6dd1de3dd4d968876fa55f5126056834c77b0244.1684703258.git.guix@twilken.net> Received-SPF: none client-ip=37.205.8.62; envelope-from=ws@wolfsnet.cz; helo=wolfsden.cz X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.25, SPF_HELO_PASS=-0.001, SPF_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01, UNPARSEABLE_RELAY=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1684708380; a=rsa-sha256; cv=none; b=kNdFAdinTNOsoDCiLKAfgh863COZFNFBzypGYApEWbzFZ0+do64zX6Ng8zgwI80Vkz02vz Dp8IQ2eeUoCEZ+kyX8aGyLORvVMCdeb8gEgapz/P4svqmI2iVQuVOgZ7M08lWxIR3ZZ1hc rdlfgtHrTFMMYKdXZMMlDfCvEcPR+Kl767Vbqdbe5aMaQTrgpHMBVEP1zc0O1qzjzTMB0k TKaTdKaALHWin9vD9tIElVYw+DZWLKrqOXyuZtFlgmtNHkE4M9h89PN4zjU5/9WVqTYPdB fpHcs+Gk3S2UwYG0uv+bNHtiOx7qsImplcteHAchziB2taN+E9L6OHLl7TG+pg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=wolfsden.cz header.s=mail header.b=Av1Y3shE; dkim=fail ("headers rsa verify failed") header.d=wolfsden.cz header.s=mail header.b=Av1Y3shE; dmarc=fail reason="SPF not aligned (relaxed)" header.from=wolfsden.cz (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1684708380; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=IiEC2XB13CIuuou4WQAjPaIYueWBiBivBpdu5FAktuA=; b=it6Zqmr8NMeHgt75N+3+tBPBck1oy4rc4aseAzlSXTz6sXhzMMutEL00BO8JmyJsrKBFJ2 3HOlLfzct0V4od0hCPyAhPjbMjQu1MTa7vFytBpuWubCMJQkf3ioSDJiRWtzPmGI3pHt27 tOVRn7mpbZjjXX5Ri/CqCzlZeZpv4j1nycTRtXGEqWHs43qtRHi0QcEGXRQMXD4B9i+Qer rtlRQ8Ssqd+1SP8OB6jaIl4lHuaHvuspRe+6PRW6SMYEoskmSpm+m7tdFGQDh1VwStm6i8 uzwFJAWkMucxdsUTGH7DI2nJ7XtQN64myuD1weVpR0eGo2LbPawmbk2xDyGviA== X-Migadu-Spam-Score: -3.41 X-Spam-Score: -3.41 X-Migadu-Queue-Id: 8714D39241 X-Migadu-Scanner: scn0.migadu.com Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=wolfsden.cz header.s=mail header.b=Av1Y3shE; dkim=fail ("headers rsa verify failed") header.d=wolfsden.cz header.s=mail header.b=Av1Y3shE; dmarc=fail reason="SPF not aligned (relaxed)" header.from=wolfsden.cz (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-TUID: RpFcDsqQyPNT --iw7hpULjExUreuIc Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, What a coincidence, this week I happened to take a look at this same issue = (in my case I wanted to build terraform). I failed to get it properly working,= so I'm happy someone else took a look at it as well. On 2023-05-21 23:18:08 +0200, Timo Wilken wrote: > Some Go source repositories (notably the Google Cloud SDK) contain multip= le > submodules and use a `refs/tags//' tagging scheme. >=20 > Fixes . >=20 > * guix/import/go.scm (vcs->origin): Accept a module-path-suffix. > (go-module->guix-package): Use the new parameter. > --- > Here's a patch that fixes the reported issue (bug#54097) for me. I've only > tested this on the github.com/googleapis/google-cloud-go/compute package = so > far, though it seems to work there. Perhaps others have more testcases? Please give the github.com/Azure/go-autorest/tracing@v0.6.0 a go. My code failed on it, and (assuming I applied the patch correctly) your does as wel= l. Here are reproduction steps to make it easier for you (please tell me if I = did something wrong): $ echo '(use-modules (guix packages) (guix git-download) (guix build-sy= stem go) ((guix licenses) #:prefix license:))' >/tmp/x.scm $ ./pre-inst-env guix import go -r github.com/Azure/go-autorest/tracing= @v0.6.0 >>/tmp/x.scm $ echo go-github-com-azure-go-autorest-tracing >>/tmp/x.scm $ guix build -f /tmp/x.scm [..] starting phase `unpack' `/gnu/store/857z63cfgclsh6g52vj9xnm7iv97yz97-go-github-com-azure-go-aut= orest-tracing-0.6.0-checkout/.gitignore' -> `/tmp/guix-build-go-github-com-= azure-go-autorest-tracing-0.6.0.drv-0/src/github.com/Azure/go-autorest/.git= ignore' error: in phase 'unpack': uncaught exception: system-error "copy-file" "~A" ("Permission denied") (13)=20 phase `unpack' failed after 0.0 seconds [..] I will not pretend to have a full grasp on how (guix build-system go) works, however my debugging lead me to the observation that it tries to unpack two dependencies into one file system tree overlayed on top of each other. I t= hink the current way (GO111MODULE=3Doff) of building of golang packages does not= play very well with well, go modules. Either the build system needs to be smarter about unpacking dependencies (a= nd doing it in a correct order), or we should start using go modules for the b= uilds (it can still be down offline, just the dependencies are in different paths= ). The second approach is what I wanted to explore, but did not get to it yet = (and likely will not for a month or two). >=20 > I don't know enough about Go tooling to use it, so I've just patched the = Guile > logic of the importer. (I don't write Go, I just want to package stuff wr= itten > in it.) In terms of performance, at least the repo contents are apparently > cached by the first `git-checkout-hash' call, even if it fails, so the se= cond > call doesn't have to redownload them. >=20 > guix/import/go.scm | 56 +++++++++++++++++++++++++++++++++++----------- > 1 file changed, 43 insertions(+), 13 deletions(-) >=20 > diff --git a/guix/import/go.scm b/guix/import/go.scm > index 0357e6a1eb..652ac58b6f 100644 > --- a/guix/import/go.scm > +++ b/guix/import/go.scm > @@ -7,6 +7,7 @@ > ;;; Copyright =A9 2021 Xinglu Chen > ;;; Copyright =A9 2021 Sarah Morgensen > ;;; Copyright =A9 2021 Simon Tournier > +;;; Copyright =A9 2023 Timo Wilken > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -89,6 +90,7 @@ (define-module (guix import go) > =20 > ;;; TODO list > ;;; - get correct hash in vcs->origin for Mercurial and Subversion > +;;; - handle subdir/vX.Y versioning in vcs->origin for Mercurial and Sub= version > =20 > ;;; Code: > =20 > @@ -513,29 +515,54 @@ (define* (git-checkout-hash url reference algorithm) > `(tag-or-commit . ,reference))= ))) > (file-hash* checkout #:algorithm algorithm #:recursive? #true))) > =20 > -(define (vcs->origin vcs-type vcs-repo-url version) > +(define (vcs->origin vcs-type vcs-repo-url module-path-suffix version) > "Generate the `origin' block of a package depending on what type of so= urce > control system is being used." > (case vcs-type > ((git) > - (let ((plain-version? (string=3D? version (go-version->git-ref vers= ion))) > - (v-prefixed? (string-prefix? "v" version))) > + (let ((v-prefixed? (string-prefix? "v" version)) > + (path-prefixed? #f) > + (trimmed-path-suffix (string-trim-both module-path-suffix #\/= )) > + (checkout-hash (false-if-git-not-found > + (git-checkout-hash > + vcs-repo-url > + (go-version->git-ref version) > + (hash-algorithm sha256))))) > + ;; If `checkout-hash' is false, that must mean that a tag named a= fter > + ;; the version doesn't exist. Some repos that contain submodules= use a > + ;; / tagging scheme instead, so try that. > + (unless checkout-hash > + (when (string=3D? "" trimmed-path-suffix) > + ;; If this isn't a submodule, / tagging m= akes no sense. > + ;; Tell the user we couldn't find the original version. > + (raise > + (formatted-message (G_ "could not find git reference '~a' in= repository '~a'") > + (go-version->git-ref version) vcs-repo-ur= l))) > + (set! path-prefixed? #t) > + (set! checkout-hash (git-checkout-hash > + vcs-repo-url > + (go-version->git-ref > + (string-append trimmed-path-suffix "/" ve= rsion)) > + (hash-algorithm sha256)))) > `(origin > (method git-fetch) > (uri (git-reference > (url ,vcs-repo-url) > - ;; This is done because the version field of the package, > - ;; which the generated quoted expression refers to, has = been > - ;; stripped of any 'v' prefixed. > - (commit ,(if (and plain-version? v-prefixed?) > - '(string-append "v" version) > - '(go-version->git-ref version))))) > + ;; The 'v' is prepended again because the version field = of > + ;; the package, which the generated quoted expression re= fers > + ;; to, has been stripped of any 'v' prefixed. > + (commit (go-version->git-ref > + ,(cond > + (path-prefixed? > + `(string-append > + ,trimmed-path-suffix "/" > + ,@(if v-prefixed? '("v" version) '(version= )))) > + (v-prefixed? '(string-append "v" version)) > + (else 'version)))))) > (file-name (git-file-name name version)) > (sha256 > (base32 > - ,(bytevector->nix-base32-string > - (git-checkout-hash vcs-repo-url (go-version->git-ref versi= on) > - (hash-algorithm sha256)))))))) > + ,(bytevector->nix-base32-string checkout-hash)))))) > ((hg) > `(origin > (method hg-fetch) > @@ -614,6 +641,9 @@ (define* (go-module->guix-package module-path #:key > (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path))) > (guix-name (go-module->guix-package-name module-path)) > (root-module-path (module-path->repository-root module-path)) > + (module-path-suffix ; subdirectory inside the source repo > + (substring module-path-sans-suffix > + (string-prefix-length root-module-path module-path-= sans-suffix))) > ;; The VCS type and URL are not included in goproxy information= =2E For > ;; this we need to fetch it from the official module page. > (meta-data (fetch-module-meta-data root-module-path)) > @@ -627,7 +657,7 @@ (define* (go-module->guix-package module-path #:key > (name ,guix-name) > (version ,(strip-v-prefix version*)) > (source > - ,(vcs->origin vcs-type vcs-repo-url version*)) > + ,(vcs->origin vcs-type vcs-repo-url module-path-suffix version*= )) > (build-system go-build-system) > (arguments > '(#:import-path ,module-path >=20 > base-commit: e499cb2c12d7f1c6d2f004364c9cc7bdb7e38cd5 > --=20 > 2.40.1 > I did not really take a look at the scheme code, I'm still Guix and Scheme beginner, so I'm very much not up to the task of doing actual code review. Nevertheless, I hope my mail helps at least a bit. Have a nice day, W. --=20 There are only two hard things in Computer Science: cache invalidation, naming things and off-by-one errors. --iw7hpULjExUreuIc Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEt4NJs4wUfTYpiGikL7/ufbZ/wakFAmRqkxkACgkQL7/ufbZ/ wamucg//aKKPWvermh5Y9aM+7NGoKaq2DqtFlIO+A32wnJcZQupMRWzKyRzfnT77 CXIwMuKwz53Dr6OaB2HP3oiXjREkISg1PdHUBAJ4ziJX+SAywlZihqjlh5698e1K m0izXVjx3IIMjLqEv6EMCgAKLRG/4HpdGmG7oTtsMN0KGHD2Kufo1Mz3lR0zYUqx LDciOb10EV7XyJvjeYHUKtzz8EzF4NQO4+S0gCY6kN1PYjFEx9wy9s996W+i7ngY EZWMZLL1ppbQtmcDzdqjgLebZ5+/P8ZjmuuELBmuBtCaUhtu3ViOSC7ieOkVkD+m 8OWW078xqHYDiTWc9Qz6K/+PsTQUJxFJRro3cHXzPV+Go9gGer921Vr0Ua4r1cCg K0WIuTjS2wMjIOdsnqPfPVWGiks6zVe07fQFze38OtcbfadVJe2fxCiBXHeq5p7p kwhZDrtcMrlBE0moNaFjPC061EtW4s9BrsLPy1NszeGp+gW12/NUyUeLv3BK3gbM 8DvAQo7QYGioIB2H0xPewMj4mArMoDhbFlZZ0cnhqGrd3snir3XXZu0Cw4hSJQBt uPIvS4XsTmCKWV4c11mYKW4T78iFkP7IFHV1JiIcLYP1W5E2fCZhcHMoUB+VFvcF lU2v6NhJzuQuPeMB/Ngj4Dey1IXrs5SbvFsISpPOboL3nRuRGgQ= =zjKZ -----END PGP SIGNATURE----- --iw7hpULjExUreuIc--