[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[Christina O'Donnell]

* guix/build-system/go.scm (go-version->git-ref): Add subdir keyword
parameter. This is needed because Go can have mutliple modules at different
versions in a single repo. It distinguishes their releases by using tags
with their subdirectory. See https://go.dev/ref/mod#vcs-version.

Change-Id: I68bc9e785e49877bb0b756de8458308549f4c957
---
 guix/build-system/go.scm | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 0934fded07..94c5439dd1 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -56,11 +56,12 @@ (define %go-pseudo-version-rx
                 "([0-9A-Fa-f]{12})"            ;commit hash
                 "(\\+incompatible)?$")))       ;optional +incompatible tag
 
-(define (go-version->git-ref version)
+(define* (go-version->git-ref version #:key subdir)
   "Parse VERSION, a \"pseudo-version\" as defined at
-<https://golang.org/ref/mod#pseudo-versions>, and extract the commit hash from
-it, defaulting to full VERSION (stripped from the \"+incompatible\" suffix if
-present) if a pseudo-version pattern is not recognized."
+<https://golang.org/ref/mod#pseudo-versions>, and extract the commit hash from it,
+defaulting to full VERSION (stripped from the \"+incompatible\" suffix if present) if
+a pseudo-version pattern is not recognized.  If SUBDIR is specified and this is not a
+pseudo-version, then this will prefix SUBDIR/ to the returned tag."
   ;; A module version like v1.2.3 is introduced by tagging a revision in the
   ;; underlying source repository.  Untagged revisions can be referred to
   ;; using a "pseudo-version" like v0.0.0-yyyymmddhhmmss-abcdefabcdef, where
@@ -78,7 +79,9 @@ (define (go-version->git-ref version)
          (match (regexp-exec %go-pseudo-version-rx version)))
     (if match
         (match:substring match 2)
-        version)))
+        (if subdir
+            (string-append subdir "/" version)
+            version))))
 
 (define (go-pseudo-version? version)
   "True if VERSION is a Go pseudo-version, i.e., a version string made of a

base-commit: efc0ee1d7f2b704d3fc0c8aea0ef0ad1ac2972e1
-- 
2.41.0

[bug#69827] [PATCH 2/3] import/go: Account for monorepo modules in the Go importer.

[Christina O'Donnell]

This change allows for importing of modules situated in repos where they're
not at the root directory.

* guix/import/go.scm (go-module-verstion-string): Delete
(go-module-version-info): New procedure
(module-path->repository-root): Add parameter
(vcs->origin): Add parameter
(path-diff): New procedure

Change-Id: If50f7a951ce8e527e5ea44ed24db10d6a3676ff0
---
 guix/import/go.scm | 71 +++++++++++++++++++++++++++++++++-------------
 1 file changed, 51 insertions(+), 20 deletions(-)

diff --git a/guix/import/go.scm b/guix/import/go.scm
index dd9298808d..8276797d9a 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -122,15 +122,14 @@ (define (go-path-escape path)
 (define (go.pkg.dev-info name)
   (http-fetch* (string-append "https://pkg.go.dev/" name)))
 
-(define* (go-module-version-string goproxy name #:key version)
-  "Fetch the version string of the latest version for NAME from the given
+(define* (go-module-version-info goproxy name #:key version)
+  "Fetch a JSON object encoding about the lastest version for NAME from the given
 GOPROXY server, or for VERSION when specified."
   (let ((file (if version
                   (string-append "@v/" version ".info")
                   "@latest")))
-    (assoc-ref (json-fetch* (format #f "~a/~a/~a"
-                                    goproxy (go-path-escape name) file))
-               "Version")))
+    (json-fetch* (format #f "~a/~a/~a"
+                         goproxy (go-path-escape name) file))))
 
 (define* (go-module-available-versions goproxy name)
   "Retrieve the available versions for a given module from the module proxy.
@@ -140,8 +139,12 @@ (define* (go-module-available-versions goproxy name)
          (body (http-fetch* url))
          (versions (remove string-null? (string-split body #\newline))))
     (if (null? versions)
-        (list (go-module-version-string goproxy name)) ;latest version
-        versions)))
+        (begin
+          ;; If we haven't recieved any versions, look in the version-info json
+          ;; object and return a one-element list if found.
+          (or (and=> (assoc-ref (go-module-version-info goproxy name) "Version")
+                     list))))
+        versions))
 
 (define (go-package-licenses name)
   "Retrieve the list of licenses that apply to NAME, a Go package or module
@@ -431,7 +434,7 @@ (define known-vcs
 (/[A-Za-z0-9_.\\-]+)*$"
     'git)))
 
-(define (module-path->repository-root module-path)
+(define (module-path->repository-root module-path version-info)
   "Infer the repository root from a module path.  Go modules can be
 defined at any level of a repository tree, but querying for the meta tag
 usually can only be done from the web page at the root of the repository,
@@ -452,6 +455,17 @@ (define (module-path->repository-root module-path)
              (lambda (vcs)
                (match:substring (regexp-exec (vcs-root-regex vcs)
                                              module-path) 1)))
+      (and=> (assoc-ref version-info "Origin")
+             (lambda (origin)
+               (and=> (assoc-ref origin "Subdir")
+                      (lambda (subdir)
+                        ;; If version-info contains a 'subdir' and that is a suffix,
+                        ;; then the repo-root can be found by stripping off the
+                        ;; suffix.
+                        (if (string-suffix? (string-append "/" subdir) module-path)
+                            (string-drop-right module-path
+                                               (+ 1 (string-length subdir)))
+                            #f)))))
       (vcs-qualified-module-path->root-repo-url module-path)
       module-path))
 
@@ -534,13 +548,21 @@ (define* (git-checkout-hash url reference algorithm)
                                           `(tag-or-commit . ,reference)))))
     (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
 
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define (vcs->origin vcs-type vcs-repo-url version subdir)
   "Generate the `origin' block of a package depending on what type of source
 control system is being used."
   (case vcs-type
     ((git)
-     (let ((plain-version? (string=? version (go-version->git-ref version)))
-           (v-prefixed?    (string-prefix? "v" version)))
+     (let* ((plain-version? (string=? version (go-version->git-ref version
+                                                                   #:subdir subdir)))
+            (v-prefixed?    (string-prefix? "v" version))
+            ;; This is done because the version field of the package,
+            ;; which the generated quoted expression refers to, has been
+            ;; stripped of any 'v' prefixed.
+            (version-expr   (if (and plain-version? v-prefixed?)
+                                '(string-append "v" version)
+                                `(go-version->git-ref version
+                                                      ,@(if subdir `(#:subdir ,subdir) '())))))
        `(origin
           (method git-fetch)
           (uri (git-reference
@@ -548,14 +570,13 @@ (define (vcs->origin vcs-type vcs-repo-url version)
                 ;; This is done because the version field of the package,
                 ;; which the generated quoted expression refers to, has been
                 ;; stripped of any 'v' prefixed.
-                (commit ,(if (and plain-version? v-prefixed?)
-                             '(string-append "v" version)
-                             '(go-version->git-ref version)))))
+                (commit ,version-expr)))
           (file-name (git-file-name name version))
           (sha256
            (base32
             ,(bytevector->nix-base32-string
-              (git-checkout-hash vcs-repo-url (go-version->git-ref version)
+              (git-checkout-hash vcs-repo-url (go-version->git-ref version
+                                                                   #:subdir subdir)
                                  (hash-algorithm sha256))))))))
     ((hg)
      `(origin
@@ -612,6 +633,12 @@ (define (validate-version version available-versions module-path)
                                   (map strip-v-prefix
                                        available-versions)))))))))
 
+(define (path-diff parent child)
+  (if (and (string-prefix? parent child) (not (string=? parent child)))
+      (let ((parent-len (string-length parent)))
+        (string-trim (substring child parent-len) (char-set #\/)))
+      #f))
+
 (define* (go-module->guix-package module-path #:key
                                   (goproxy "https://proxy.golang.org")
                                   version
@@ -623,9 +650,11 @@ (define* (go-module->guix-package module-path #:key
   (let* ((available-versions (go-module-available-versions goproxy module-path))
          (version* (validate-version
                     (or (and version (ensure-v-prefix version))
-                        (go-module-version-string goproxy module-path)) ;latest
+                        (assoc-ref (go-module-version-info goproxy module-path)
+                                   "Version")) ;latest
                     available-versions
                     module-path))
+         (version-info (go-module-version-info goproxy module-path #:version version*))
          (content (fetch-go.mod goproxy module-path version*))
          (min-go-version (second (go.mod-go-version (parse-go.mod content))))
          (dependencies+versions (go.mod-requirements (parse-go.mod content)))
@@ -634,11 +663,13 @@ (define* (go-module->guix-package module-path #:key
                            (map car dependencies+versions)))
          (module-path-sans-suffix
           (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path)))
-         (guix-name (go-module->guix-package-name module-path))
-         (root-module-path (module-path->repository-root module-path))
+         (guix-name (go-module->guix-package-name module-path-sans-suffix ))
+         (root-module-path (module-path->repository-root module-path-sans-suffix
+                                                         version-info))
          ;; The VCS type and URL are not included in goproxy information. For
          ;; this we need to fetch it from the official module page.
          (meta-data (fetch-module-meta-data root-module-path))
+         (subdir (path-diff root-module-path module-path-sans-suffix))
          (vcs-type (module-meta-vcs meta-data))
          (vcs-repo-url (module-meta-data-repo-url meta-data goproxy))
          (synopsis (go-package-synopsis module-path))
@@ -649,14 +680,14 @@ (define* (go-module->guix-package module-path #:key
         (name ,guix-name)
         (version ,(strip-v-prefix version*))
         (source
-         ,(vcs->origin vcs-type vcs-repo-url version*))
+         ,(vcs->origin vcs-type vcs-repo-url version* subdir))
         (build-system go-build-system)
         (arguments
          (list ,@(if (version>? min-go-version (package-version (go-package)))
                      `(#:go ,(string->number min-go-version))
                      '())
                #:import-path ,module-path
-               ,@(if (string=? module-path-sans-suffix root-module-path)
+               ,@(if (string=? module-path root-module-path)
                      '()
                      `(#:unpack-path ,root-module-path))))
         ,@(maybe-propagated-inputs
-- 
2.41.0

[bug#69827] [PATCH 3/3] import/go: Add diagnostics.

[Christina O'Donnell]

guix/import/go.scm (go-module-available-versions): Add warning when fetching
a list of versions from the proxy fails and raise exception when no version
can be found at all.
(module-path->repository-root): Warn when all attempts to find the repository
root have failed.
(fetch-module-meta-data): Raise exception when no meta element could be found.
(go-module->guix-package): Catch general exceptions and warn that the package
could not be imported.

Change-Id: I6dcdccc71f54bfec7110f6bfc5aeb8855502d1e3
---
 guix/import/go.scm | 56 ++++++++++++++++++++++++++++++++++++----------
 1 file changed, 44 insertions(+), 12 deletions(-)

diff --git a/guix/import/go.scm b/guix/import/go.scm
index 8276797d9a..5dd5b3d221 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -140,10 +140,15 @@ (define* (go-module-available-versions goproxy name)
          (versions (remove string-null? (string-split body #\newline))))
     (if (null? versions)
         (begin
+          (warning (G_ "Empty list of versions on proxy ~a for package '~a'. Using latest.~%")
+                   goproxy name)
           ;; If we haven't recieved any versions, look in the version-info json
           ;; object and return a one-element list if found.
           (or (and=> (assoc-ref (go-module-version-info goproxy name) "Version")
-                     list))))
+                     list)
+              (raise (make-compound-condition
+                      (formatted-message (G_ "No versions available for '~a' on proxy ~a.")
+                                         name goproxy))))))
         versions))
 
 (define (go-package-licenses name)
@@ -467,7 +472,10 @@ (define (module-path->repository-root module-path version-info)
                                                (+ 1 (string-length subdir)))
                             #f)))))
       (vcs-qualified-module-path->root-repo-url module-path)
-      module-path))
+      (begin
+        (warning (G_ "Unable to determine repository root of '~a'. Guessing '~a'.~%")
+                 module-path module-path)
+        module-path)))
 
 (define* (go-module->guix-package-name module-path #:optional version)
   "Converts a module's path to the canonical Guix format for Go packages.
@@ -512,14 +520,19 @@ (define (fetch-module-meta-data module-path)
          (select (sxpath `(// (meta (@ (equal? (name "go-import"))))
                               // content))))
     (match (select (html->sxml meta-data #:strict? #t))
-      (() #f)                           ;nothing selected
+      (() (raise (make-compound-condition
+                  (formatted-message (G_ "no <meta/> element in result when accessing module path '~a' using go-get")
+                                     module-path))))
       ((('content content-text) ..1)
        (or
         (find (lambda (meta)
                 (string-prefix? (module-meta-import-prefix meta) module-path))
               (map go-import->module-meta content-text))
         ;; Fallback to the first meta if no import prefixes match.
-        (go-import->module-meta (first content-text)))))))
+        (go-import->module-meta (first content-text))
+        (raise (make-compound-condition
+                (formatted-message (G_ "unable to parse <meta/> when accessing module path '~a' using go-get")
+                                   module-path))))))))
 
 (define (module-meta-data-repo-url meta-data goproxy)
   "Return the URL where the fetcher which will be used can download the
@@ -716,16 +729,35 @@ (define go-module->guix-package*
     ;; consistently.
     (setvbuf (current-error-port) 'none)
     (let ((package-name (match args ((name _ ...) name))))
-      (guard (c ((http-get-error? c)
-                 (warning (G_ "Failed to import package ~s.
+      (begin
+        (info (G_ "Importing package ~s...~%") package-name)
+        (guard (c ((http-get-error? c)
+                        (warning (G_ "Failed to import package ~s.
 reason: ~s could not be fetched: HTTP error ~a (~s).
 This package and its dependencies won't be imported.~%")
-                          package-name
-                          (uri->string (http-get-error-uri c))
-                          (http-get-error-code c)
-                          (http-get-error-reason c))
-                 (values #f '())))
-        (apply go-module->guix-package args)))))
+                                 package-name
+                                 (uri->string (http-get-error-uri c))
+                                 (http-get-error-code c)
+                                 (http-get-error-reason c))
+
+                        (values #f '()))
+                  ((formatted-message? c)
+                   (warning (G_ "Failed to import package ~s.
+reason: ~a
+This package and its dependencies won't be imported.~%")
+                            package-name
+                            (apply format #f
+                                   (formatted-message-string c)
+                                   (formatted-message-arguments c)))
+                   (values #f '()))
+                  ((git-error? c)
+                   (warning (G_ "Failed to import package ~s.
+reason: ~a
+This package and its dependencies won't be imported.~%")
+                            package-name
+                            (git-error-message c))
+                   (values #f '())))
+               (apply go-module->guix-package args))))))
 
 (define* (go-module-recursive-import package-name
                                      #:key (goproxy "https://proxy.golang.org")
-- 
2.41.0

[bug#69827] Fixing the go importer so that it can import modules located in a monorepo

[Christina O'Donnell]

Hi,

These three patches above allow the go importer to automatically import 
modules that are located in a monorepo where the submodules are 
versioned independently. See https://go.dev/ref/mod#vcs-version:

 > If a module is defined in a subdirectory within the repository, that 
is, the module subdirectory portion of the module path is not empty, 
then each tag name must be prefixed with the module subdirectory, 
followed by a slash. For example, the module golang.org/x/tools/gopls is 
defined in the gopls subdirectory of the repository with root path 
golang.org/x/tools. The version v0.4.0 of that module must have the tag 
named gopls/v0.4.0 in that repository.

After this change, I am able to import github.com/gohugio/hugo and its 
400 odd dependencies with much fewer failures.

I'll submit those patches separately once I've tested them.

Kind regards,

Christina

PS. This is my first time sending a patch series to debbugs, so I almost 
certainly did something wrong. Feedback on that would be appreciated.

[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 287 bytes --]

Hi,

Just a clarifying question. Do we definitely need to know
 anything about the folder structure where go.mod governs the
 module definition?

There is an API https://pkg.go.dev/  where the most of go projects
 are registered, would it be easier just query it directly?

Thanks,
Oleg

[-- Attachment #2: Type: text/html, Size: 625 bytes --]

[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[Christina O'Donnell]

Hi Oleg,

Thanks for having a look at this!

I don't believe you copied me in the last email so I didn't see your 
email when it went out.

 > Just a clarifying question. Do we definitely need to know
 > anything about the folder structure where go.mod governs the
 > module definition?

 > There is an API https://pkg.go.dev/ where the most of go projects
 > are registered, would it be easier just query it directly?

pkg.go.dev gets its sources from proxy.golang.org, and pkg.go.dev 
focuses mainly on building and hosting documentation [1]. It doesn't 
expose an API for fetching version info [2].

We're already querying proxy.golang.org to get the latest version, and 
that response contains information about the sub-directory and git tag, 
so we may as well use it.

Kind regards,

Christina

[1] https://pkg.go.dev/about
[2] https://pkg.go.dev/cmd/api

[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[Sharlatan Hellseher]

[-- Attachment #1.1: Type: text/plain, Size: 661 bytes --]

Hi,

It's time to proper review your proposal :-)

I faced with obstacles of monorepo during review  process, especially where
nested path (subdir) has more than one level.

Let's have look at https://github.com/Azure/azure-sdk-for-go
which is required for restic update.

I will apply your patches and play with importing/upgrading projects like
restic to check how it feels.

Related issues:
- https://issues.guix.gnu.org/72963, requires
https://github.com/charmbracelet/x - monorepo
- https://issues.guix.gnu.org/63019, requires
https://github.com/Azure/azure-sdk-for-go - monorepo
- updating aws-sdk-go-v2 https://github.com/aws/aws-sdk-go-v2

Thanks,
Oleg

[-- Attachment #1.2: Type: text/html, Size: 1388 bytes --]

[-- Attachment #2: Screenshot_20240904-200044.png --]
[-- Type: image/png, Size: 144457 bytes --]

[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]


Hi,

Some feed back on the patches: when applied on go-team I could easily
import the full Restic dependencies tree it took about 30min to complete
as Guix missed nearly all packages for google cloud.

Mean while it was handy to use in IPFS/Kubo unbundling process where me
and Artyom in our leisure time try to pack all vendored dependencies.

The import of https://github.com/libp2p/go-libp2p recursivly produced
all 26 missing packages which I about to submit soon!

As a side note, it's not the first proposal of that kind, please see:

- <2021-12-07> guix import go error <https://issues.guix.gnu.org/52362>
  by Stephen Webber <montokapro@gmail.com>

- <2023-04-21> Go importer doesn't know MODULE/vX.Y version tags
  <https://issues.guix.gnu.org/63001> by Timo Wilken <guix@twilken.net>

- <2023-05-22> [PATCH 0/2] Fix annoyance with "guix import go"
  <https://issues.guix.gnu.org/63647> by Simon Tournier
  <zimon.toutoune@gmail.com>

- <2023-06-12> [PATH] fix a bug on importing go packages.
  <https://issues.guix.gnu.org/64035>,
  <https://issues.guix.gnu.org/64036> by Elbek


May we come to some consensus and apply some median patch including all
proposals and fixing the currently issue with import golang modules with
subdirs?

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[Simon Tournier]

Hi Oleg,

Thanks for taking care about the unmerged patches. :-)


On Sat, 07 Sep 2024 at 15:28, Sharlatan Hellseher <sharlatanus@gmail.com> wrote:

> - <2023-05-22> [PATCH 0/2] Fix annoyance with "guix import go"
>   <https://issues.guix.gnu.org/63647> by Simon Tournier
>   <zimon.toutoune@gmail.com>

[...]

> May we come to some consensus and apply some median patch including all
> proposals and fixing the currently issue with import golang modules with
> subdirs?

Well, from my perspective, patch#63647 was ready but since I am not
really using “guix import go”, I do not feel qualified to know what is
better: error out at the first Git issue and continue with a wrong hash.

Somehow, what appears to you the better answer of:

        I thought it would be more appropriate to error out as soon as we have a
        Git problem, rather than print a warning and emit an incorrect hash.

        I understand that, when using ‘--recursive’, this means aborting the
        whole process without producing anything.  But maybe that’s better that
        producing an incorrect (origin …) form?

My intuition is not to abort the whole process and instead get something
where I manually fix these incorrect hashes.  When I had to do it, it
had been far easier and faster than looping over “guix import go”.

In other words, “guix import go foo -r” and paste the output in one
file, then run “guix build foo” and fix the incorrect hash origins or
some other package tweaks.

I have not looked into the other patches; if they fix similar issues,

Cheers,
simon


1: [bug#63647] [PATCH 0/2] Fix annoyance with "guix import go"
Ludovic Courtès <ludo@gnu.org>
Tue, 06 Jun 2023 17:42:00 +0200
id:87zg5c61iv.fsf@gnu.org
https://issues.guix.gnu.org/63647
https://issues.guix.gnu.org/msgid/87zg5c61iv.fsf@gnu.org
https://yhetil.org/guix/87zg5c61iv.fsf@gnu.org

bug#69827: [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 682 bytes --]


Hi,

After a few days of testing with importing, building and updating Golang
packages I can say it feels very stable with patches proposed by
Christiana - there was no single import failure on such a long chain of
imports like for go-github-com-spf13-afero (209 packages+),
go-github-com-ipfs-boxo (231 package+).

I've used it as a current implementation to fix mentioned issues with
guix import go.

Pushed as 45cbf468d2..23dbf8d073 to go-team.

The branch contains the fresh versions of golang-build packages which
might need to rebuild most of the Goalang containing in Guix. But it
would let us to bring Prometheus soon, most of the components a also on
the branch.

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]