[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[Christina O'Donnell]

* guix/build-system/go.scm (go-version->git-ref): Add subdir keyword
parameter. This is needed because Go can have mutliple modules at different
versions in a single repo. It distinguishes their releases by using tags
with their subdirectory. See https://go.dev/ref/mod#vcs-version.

Change-Id: I68bc9e785e49877bb0b756de8458308549f4c957
---
 guix/build-system/go.scm | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 0934fded07..94c5439dd1 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -56,11 +56,12 @@ (define %go-pseudo-version-rx
                 "([0-9A-Fa-f]{12})"            ;commit hash
                 "(\\+incompatible)?$")))       ;optional +incompatible tag
 
-(define (go-version->git-ref version)
+(define* (go-version->git-ref version #:key subdir)
   "Parse VERSION, a \"pseudo-version\" as defined at
-<https://golang.org/ref/mod#pseudo-versions>, and extract the commit hash from
-it, defaulting to full VERSION (stripped from the \"+incompatible\" suffix if
-present) if a pseudo-version pattern is not recognized."
+<https://golang.org/ref/mod#pseudo-versions>, and extract the commit hash from it,
+defaulting to full VERSION (stripped from the \"+incompatible\" suffix if present) if
+a pseudo-version pattern is not recognized.  If SUBDIR is specified and this is not a
+pseudo-version, then this will prefix SUBDIR/ to the returned tag."
   ;; A module version like v1.2.3 is introduced by tagging a revision in the
   ;; underlying source repository.  Untagged revisions can be referred to
   ;; using a "pseudo-version" like v0.0.0-yyyymmddhhmmss-abcdefabcdef, where
@@ -78,7 +79,9 @@ (define (go-version->git-ref version)
          (match (regexp-exec %go-pseudo-version-rx version)))
     (if match
         (match:substring match 2)
-        version)))
+        (if subdir
+            (string-append subdir "/" version)
+            version))))
 
 (define (go-pseudo-version? version)
   "True if VERSION is a Go pseudo-version, i.e., a version string made of a

base-commit: efc0ee1d7f2b704d3fc0c8aea0ef0ad1ac2972e1
-- 
2.41.0

[bug#69827] [PATCH 2/3] import/go: Account for monorepo modules in the Go importer.

[Christina O'Donnell]

This change allows for importing of modules situated in repos where they're
not at the root directory.

* guix/import/go.scm (go-module-verstion-string): Delete
(go-module-version-info): New procedure
(module-path->repository-root): Add parameter
(vcs->origin): Add parameter
(path-diff): New procedure

Change-Id: If50f7a951ce8e527e5ea44ed24db10d6a3676ff0
---
 guix/import/go.scm | 71 +++++++++++++++++++++++++++++++++-------------
 1 file changed, 51 insertions(+), 20 deletions(-)

diff --git a/guix/import/go.scm b/guix/import/go.scm
index dd9298808d..8276797d9a 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -122,15 +122,14 @@ (define (go-path-escape path)
 (define (go.pkg.dev-info name)
   (http-fetch* (string-append "https://pkg.go.dev/" name)))
 
-(define* (go-module-version-string goproxy name #:key version)
-  "Fetch the version string of the latest version for NAME from the given
+(define* (go-module-version-info goproxy name #:key version)
+  "Fetch a JSON object encoding about the lastest version for NAME from the given
 GOPROXY server, or for VERSION when specified."
   (let ((file (if version
                   (string-append "@v/" version ".info")
                   "@latest")))
-    (assoc-ref (json-fetch* (format #f "~a/~a/~a"
-                                    goproxy (go-path-escape name) file))
-               "Version")))
+    (json-fetch* (format #f "~a/~a/~a"
+                         goproxy (go-path-escape name) file))))
 
 (define* (go-module-available-versions goproxy name)
   "Retrieve the available versions for a given module from the module proxy.
@@ -140,8 +139,12 @@ (define* (go-module-available-versions goproxy name)
          (body (http-fetch* url))
          (versions (remove string-null? (string-split body #\newline))))
     (if (null? versions)
-        (list (go-module-version-string goproxy name)) ;latest version
-        versions)))
+        (begin
+          ;; If we haven't recieved any versions, look in the version-info json
+          ;; object and return a one-element list if found.
+          (or (and=> (assoc-ref (go-module-version-info goproxy name) "Version")
+                     list))))
+        versions))
 
 (define (go-package-licenses name)
   "Retrieve the list of licenses that apply to NAME, a Go package or module
@@ -431,7 +434,7 @@ (define known-vcs
 (/[A-Za-z0-9_.\\-]+)*$"
     'git)))
 
-(define (module-path->repository-root module-path)
+(define (module-path->repository-root module-path version-info)
   "Infer the repository root from a module path.  Go modules can be
 defined at any level of a repository tree, but querying for the meta tag
 usually can only be done from the web page at the root of the repository,
@@ -452,6 +455,17 @@ (define (module-path->repository-root module-path)
              (lambda (vcs)
                (match:substring (regexp-exec (vcs-root-regex vcs)
                                              module-path) 1)))
+      (and=> (assoc-ref version-info "Origin")
+             (lambda (origin)
+               (and=> (assoc-ref origin "Subdir")
+                      (lambda (subdir)
+                        ;; If version-info contains a 'subdir' and that is a suffix,
+                        ;; then the repo-root can be found by stripping off the
+                        ;; suffix.
+                        (if (string-suffix? (string-append "/" subdir) module-path)
+                            (string-drop-right module-path
+                                               (+ 1 (string-length subdir)))
+                            #f)))))
       (vcs-qualified-module-path->root-repo-url module-path)
       module-path))
 
@@ -534,13 +548,21 @@ (define* (git-checkout-hash url reference algorithm)
                                           `(tag-or-commit . ,reference)))))
     (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
 
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define (vcs->origin vcs-type vcs-repo-url version subdir)
   "Generate the `origin' block of a package depending on what type of source
 control system is being used."
   (case vcs-type
     ((git)
-     (let ((plain-version? (string=? version (go-version->git-ref version)))
-           (v-prefixed?    (string-prefix? "v" version)))
+     (let* ((plain-version? (string=? version (go-version->git-ref version
+                                                                   #:subdir subdir)))
+            (v-prefixed?    (string-prefix? "v" version))
+            ;; This is done because the version field of the package,
+            ;; which the generated quoted expression refers to, has been
+            ;; stripped of any 'v' prefixed.
+            (version-expr   (if (and plain-version? v-prefixed?)
+                                '(string-append "v" version)
+                                `(go-version->git-ref version
+                                                      ,@(if subdir `(#:subdir ,subdir) '())))))
        `(origin
           (method git-fetch)
           (uri (git-reference
@@ -548,14 +570,13 @@ (define (vcs->origin vcs-type vcs-repo-url version)
                 ;; This is done because the version field of the package,
                 ;; which the generated quoted expression refers to, has been
                 ;; stripped of any 'v' prefixed.
-                (commit ,(if (and plain-version? v-prefixed?)
-                             '(string-append "v" version)
-                             '(go-version->git-ref version)))))
+                (commit ,version-expr)))
           (file-name (git-file-name name version))
           (sha256
            (base32
             ,(bytevector->nix-base32-string
-              (git-checkout-hash vcs-repo-url (go-version->git-ref version)
+              (git-checkout-hash vcs-repo-url (go-version->git-ref version
+                                                                   #:subdir subdir)
                                  (hash-algorithm sha256))))))))
     ((hg)
      `(origin
@@ -612,6 +633,12 @@ (define (validate-version version available-versions module-path)
                                   (map strip-v-prefix
                                        available-versions)))))))))
 
+(define (path-diff parent child)
+  (if (and (string-prefix? parent child) (not (string=? parent child)))
+      (let ((parent-len (string-length parent)))
+        (string-trim (substring child parent-len) (char-set #\/)))
+      #f))
+
 (define* (go-module->guix-package module-path #:key
                                   (goproxy "https://proxy.golang.org")
                                   version
@@ -623,9 +650,11 @@ (define* (go-module->guix-package module-path #:key
   (let* ((available-versions (go-module-available-versions goproxy module-path))
          (version* (validate-version
                     (or (and version (ensure-v-prefix version))
-                        (go-module-version-string goproxy module-path)) ;latest
+                        (assoc-ref (go-module-version-info goproxy module-path)
+                                   "Version")) ;latest
                     available-versions
                     module-path))
+         (version-info (go-module-version-info goproxy module-path #:version version*))
          (content (fetch-go.mod goproxy module-path version*))
          (min-go-version (second (go.mod-go-version (parse-go.mod content))))
          (dependencies+versions (go.mod-requirements (parse-go.mod content)))
@@ -634,11 +663,13 @@ (define* (go-module->guix-package module-path #:key
                            (map car dependencies+versions)))
          (module-path-sans-suffix
           (match:prefix (string-match "([\\./]v[0-9]+)?$" module-path)))
-         (guix-name (go-module->guix-package-name module-path))
-         (root-module-path (module-path->repository-root module-path))
+         (guix-name (go-module->guix-package-name module-path-sans-suffix ))
+         (root-module-path (module-path->repository-root module-path-sans-suffix
+                                                         version-info))
          ;; The VCS type and URL are not included in goproxy information. For
          ;; this we need to fetch it from the official module page.
          (meta-data (fetch-module-meta-data root-module-path))
+         (subdir (path-diff root-module-path module-path-sans-suffix))
          (vcs-type (module-meta-vcs meta-data))
          (vcs-repo-url (module-meta-data-repo-url meta-data goproxy))
          (synopsis (go-package-synopsis module-path))
@@ -649,14 +680,14 @@ (define* (go-module->guix-package module-path #:key
         (name ,guix-name)
         (version ,(strip-v-prefix version*))
         (source
-         ,(vcs->origin vcs-type vcs-repo-url version*))
+         ,(vcs->origin vcs-type vcs-repo-url version* subdir))
         (build-system go-build-system)
         (arguments
          (list ,@(if (version>? min-go-version (package-version (go-package)))
                      `(#:go ,(string->number min-go-version))
                      '())
                #:import-path ,module-path
-               ,@(if (string=? module-path-sans-suffix root-module-path)
+               ,@(if (string=? module-path root-module-path)
                      '()
                      `(#:unpack-path ,root-module-path))))
         ,@(maybe-propagated-inputs
-- 
2.41.0

[bug#69827] [PATCH 3/3] import/go: Add diagnostics.

[Christina O'Donnell]

guix/import/go.scm (go-module-available-versions): Add warning when fetching
a list of versions from the proxy fails and raise exception when no version
can be found at all.
(module-path->repository-root): Warn when all attempts to find the repository
root have failed.
(fetch-module-meta-data): Raise exception when no meta element could be found.
(go-module->guix-package): Catch general exceptions and warn that the package
could not be imported.

Change-Id: I6dcdccc71f54bfec7110f6bfc5aeb8855502d1e3
---
 guix/import/go.scm | 56 ++++++++++++++++++++++++++++++++++++----------
 1 file changed, 44 insertions(+), 12 deletions(-)

diff --git a/guix/import/go.scm b/guix/import/go.scm
index 8276797d9a..5dd5b3d221 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -140,10 +140,15 @@ (define* (go-module-available-versions goproxy name)
          (versions (remove string-null? (string-split body #\newline))))
     (if (null? versions)
         (begin
+          (warning (G_ "Empty list of versions on proxy ~a for package '~a'. Using latest.~%")
+                   goproxy name)
           ;; If we haven't recieved any versions, look in the version-info json
           ;; object and return a one-element list if found.
           (or (and=> (assoc-ref (go-module-version-info goproxy name) "Version")
-                     list))))
+                     list)
+              (raise (make-compound-condition
+                      (formatted-message (G_ "No versions available for '~a' on proxy ~a.")
+                                         name goproxy))))))
         versions))
 
 (define (go-package-licenses name)
@@ -467,7 +472,10 @@ (define (module-path->repository-root module-path version-info)
                                                (+ 1 (string-length subdir)))
                             #f)))))
       (vcs-qualified-module-path->root-repo-url module-path)
-      module-path))
+      (begin
+        (warning (G_ "Unable to determine repository root of '~a'. Guessing '~a'.~%")
+                 module-path module-path)
+        module-path)))
 
 (define* (go-module->guix-package-name module-path #:optional version)
   "Converts a module's path to the canonical Guix format for Go packages.
@@ -512,14 +520,19 @@ (define (fetch-module-meta-data module-path)
          (select (sxpath `(// (meta (@ (equal? (name "go-import"))))
                               // content))))
     (match (select (html->sxml meta-data #:strict? #t))
-      (() #f)                           ;nothing selected
+      (() (raise (make-compound-condition
+                  (formatted-message (G_ "no <meta/> element in result when accessing module path '~a' using go-get")
+                                     module-path))))
       ((('content content-text) ..1)
        (or
         (find (lambda (meta)
                 (string-prefix? (module-meta-import-prefix meta) module-path))
               (map go-import->module-meta content-text))
         ;; Fallback to the first meta if no import prefixes match.
-        (go-import->module-meta (first content-text)))))))
+        (go-import->module-meta (first content-text))
+        (raise (make-compound-condition
+                (formatted-message (G_ "unable to parse <meta/> when accessing module path '~a' using go-get")
+                                   module-path))))))))
 
 (define (module-meta-data-repo-url meta-data goproxy)
   "Return the URL where the fetcher which will be used can download the
@@ -716,16 +729,35 @@ (define go-module->guix-package*
     ;; consistently.
     (setvbuf (current-error-port) 'none)
     (let ((package-name (match args ((name _ ...) name))))
-      (guard (c ((http-get-error? c)
-                 (warning (G_ "Failed to import package ~s.
+      (begin
+        (info (G_ "Importing package ~s...~%") package-name)
+        (guard (c ((http-get-error? c)
+                        (warning (G_ "Failed to import package ~s.
 reason: ~s could not be fetched: HTTP error ~a (~s).
 This package and its dependencies won't be imported.~%")
-                          package-name
-                          (uri->string (http-get-error-uri c))
-                          (http-get-error-code c)
-                          (http-get-error-reason c))
-                 (values #f '())))
-        (apply go-module->guix-package args)))))
+                                 package-name
+                                 (uri->string (http-get-error-uri c))
+                                 (http-get-error-code c)
+                                 (http-get-error-reason c))
+
+                        (values #f '()))
+                  ((formatted-message? c)
+                   (warning (G_ "Failed to import package ~s.
+reason: ~a
+This package and its dependencies won't be imported.~%")
+                            package-name
+                            (apply format #f
+                                   (formatted-message-string c)
+                                   (formatted-message-arguments c)))
+                   (values #f '()))
+                  ((git-error? c)
+                   (warning (G_ "Failed to import package ~s.
+reason: ~a
+This package and its dependencies won't be imported.~%")
+                            package-name
+                            (git-error-message c))
+                   (values #f '())))
+               (apply go-module->guix-package args))))))
 
 (define* (go-module-recursive-import package-name
                                      #:key (goproxy "https://proxy.golang.org")
-- 
2.41.0

[bug#69827] Fixing the go importer so that it can import modules located in a monorepo

[Christina O'Donnell]

Hi,

These three patches above allow the go importer to automatically import 
modules that are located in a monorepo where the submodules are 
versioned independently. See https://go.dev/ref/mod#vcs-version:

 > If a module is defined in a subdirectory within the repository, that 
is, the module subdirectory portion of the module path is not empty, 
then each tag name must be prefixed with the module subdirectory, 
followed by a slash. For example, the module golang.org/x/tools/gopls is 
defined in the gopls subdirectory of the repository with root path 
golang.org/x/tools. The version v0.4.0 of that module must have the tag 
named gopls/v0.4.0 in that repository.

After this change, I am able to import github.com/gohugio/hugo and its 
400 odd dependencies with much fewer failures.

I'll submit those patches separately once I've tested them.

Kind regards,

Christina

PS. This is my first time sending a patch series to debbugs, so I almost 
certainly did something wrong. Feedback on that would be appreciated.

[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 287 bytes --]

Hi,

Just a clarifying question. Do we definitely need to know
 anything about the folder structure where go.mod governs the
 module definition?

There is an API https://pkg.go.dev/  where the most of go projects
 are registered, would it be easier just query it directly?

Thanks,
Oleg

[-- Attachment #2: Type: text/html, Size: 625 bytes --]

[bug#69827] [PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

[Christina O'Donnell]

Hi Oleg,

Thanks for having a look at this!

I don't believe you copied me in the last email so I didn't see your 
email when it went out.

 > Just a clarifying question. Do we definitely need to know
 > anything about the folder structure where go.mod governs the
 > module definition?

 > There is an API https://pkg.go.dev/ where the most of go projects
 > are registered, would it be easier just query it directly?

pkg.go.dev gets its sources from proxy.golang.org, and pkg.go.dev 
focuses mainly on building and hosting documentation [1]. It doesn't 
expose an API for fetching version info [2].

We're already querying proxy.golang.org to get the latest version, and 
that response contains information about the sub-directory and git tag, 
so we may as well use it.

Kind regards,

Christina

[1] https://pkg.go.dev/about
[2] https://pkg.go.dev/cmd/api