bug#54097: go importer does not honor multi-module repositories

bug#54097: go importer does not honor multi-module repositories

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 2474 bytes --]

Go usually has the 1 repository=1 module convention. However, it
is also allowed that one repository contains multiple go modules.

If repository "foo" contains only one module, then versions are tagged
"v1.2.3".

However, if the repository "foo" contains modules "bar" and "baz", each
in a sub-directory of "foo", the versions will be tagged with their
respective prefix, i.e.:

foo/v1.2.3
bar/v4.5.6

See here:

https://github.com/golang/go/wiki/Modules#publishing-a-release
https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories
https://stackoverflow.com/questions/64701064/golang-separate-versioning-of-multiple-modules

However, our go-importer does not honor this. The Google Cloud API
modules are structured into sub-modules, but our importer searches the
wrong tag and raises an exception:

$ ./pre-inst-env guix import go cloud.google.com/go/storage
URL FOR VERSIONS: https://proxy.golang.org/cloud.google.com/go/storage/@v/list
FETCH_GO_MOD: https://proxy.golang.org/cloud.google.com/go/storage/@v/v1.21.0.mod
Backtrace:
In ice-9/boot-9.scm:
  1752:10 17 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          16 (apply-smob/0 #<thunk 7f5b3d24f0c0>)
In ice-9/boot-9.scm:
    724:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle?>)
In ice-9/eval.scm:
    619:8 14 (_ #(#(#<directory (guile-user) 7f5b3d255c80>)))
In guix/ui.scm:
   2209:7 13 (run-guix . _)
  2172:10 12 (run-guix-command _ . _)
In guix/scripts/import.scm:
   124:11 11 (guix-import . _)
In ice-9/boot-9.scm:
  1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/import/go.scm:
   116:29  9 (_)
In ice-9/exceptions.scm:
   406:15  8 (go-module->guix-package* . _)
In ice-9/boot-9.scm:
  1752:10  7 (with-exception-handler _ _ #:unwind? _ # _)
In guix/import/go.scm:
   525:18  6 (go-module->guix-package "cloud.google.com/go/storage" # ?)
In guix/git.scm:
    277:4  5 (update-cached-checkout _ #:ref _ #:recursive? _ # _ # _ ?)
   266:18  4 (resolve _)
In git/reference.scm:
     60:8  3 (_ _ _)
In git/bindings.scm:
     77:2  2 (raise-git-error _)
In ice-9/boot-9.scm:
  1685:16  1 (raise-exception _ #:continuable? _)
  1683:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1683:16: In procedure raise-exception:
Git error: reference 'refs/tags/v1.21.0' not found


The correct git reference to look for is:

refs/tags/storage/v1.21.0

Björn


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#54097: go importer does not honor multi-module repositories

[raingloom]

On Mon, 21 Feb 2022 23:43:54 +0100
Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

> Go usually has the 1 repository=1 module convention. However, it
> is also allowed that one repository contains multiple go modules.
> 
> If repository "foo" contains only one module, then versions are tagged
> "v1.2.3".
> 
> However, if the repository "foo" contains modules "bar" and "baz",
> each in a sub-directory of "foo", the versions will be tagged with
> their respective prefix, i.e.:
> 
> foo/v1.2.3
> bar/v4.5.6
> 
> See here:
> 
> https://github.com/golang/go/wiki/Modules#publishing-a-release
> https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories
> https://stackoverflow.com/questions/64701064/golang-separate-versioning-of-multiple-modules
> 
> However, our go-importer does not honor this. The Google Cloud API
> modules are structured into sub-modules, but our importer searches the
> wrong tag and raises an exception:
> 
> $ ./pre-inst-env guix import go cloud.google.com/go/storage
> URL FOR VERSIONS:
> https://proxy.golang.org/cloud.google.com/go/storage/@v/list
> FETCH_GO_MOD:
> https://proxy.golang.org/cloud.google.com/go/storage/@v/v1.21.0.mod
> Backtrace: In ice-9/boot-9.scm: 1752:10 17 (with-exception-handler _
> _ #:unwind? _ # _) In unknown file:
>           16 (apply-smob/0 #<thunk 7f5b3d24f0c0>)
> In ice-9/boot-9.scm:
>     724:2 15 (call-with-prompt _ _ #<procedure
> default-prompt-handle?>) In ice-9/eval.scm:
>     619:8 14 (_ #(#(#<directory (guile-user) 7f5b3d255c80>)))
> In guix/ui.scm:
>    2209:7 13 (run-guix . _)
>   2172:10 12 (run-guix-command _ . _)
> In guix/scripts/import.scm:
>    124:11 11 (guix-import . _)
> In ice-9/boot-9.scm:
>   1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
> In guix/scripts/import/go.scm:
>    116:29  9 (_)
> In ice-9/exceptions.scm:
>    406:15  8 (go-module->guix-package* . _)
> In ice-9/boot-9.scm:
>   1752:10  7 (with-exception-handler _ _ #:unwind? _ # _)
> In guix/import/go.scm:
>    525:18  6 (go-module->guix-package "cloud.google.com/go/storage" #
> ?) In guix/git.scm:
>     277:4  5 (update-cached-checkout _ #:ref _ #:recursive? _ # _ # _
> ?) 266:18  4 (resolve _)
> In git/reference.scm:
>      60:8  3 (_ _ _)
> In git/bindings.scm:
>      77:2  2 (raise-git-error _)
> In ice-9/boot-9.scm:
>   1685:16  1 (raise-exception _ #:continuable? _)
>   1683:16  0 (raise-exception _ #:continuable? _)
> 
> ice-9/boot-9.scm:1683:16: In procedure raise-exception:
> Git error: reference 'refs/tags/v1.21.0' not found
> 
> 
> The correct git reference to look for is:
> 
> refs/tags/storage/v1.21.0
> 
> Björn
> 

I think this has been mentioned before, but I really think we should
use Go's built-in tooling to extract this info, instead of
reimplementing it in Guile. There is already some massive ineffeciency
introduced by the importer cloning whole repos, whereas `go get` is
smart enough to only download `go.mod` files.
Just my 2c.

bug#54097: go importer does not honor multi-module repositories

[Blake Shaw via Bug reports for GNU Guix]

great investigation, thanks for sharing!
-- 
“In girum imus nocte et consumimur igni”