Go importer and packages with version flags

Go importer and packages with version flags

[Jack Hill]

Hi Guix,

While I was was working with the go importer today, it suggested I package 
go-github-com-russross-blackfriday-v2. Fair enough, except we already have 
a package for go-github-com-russross-blackfriday. The packages differ in 
their version (2.1.0 verses 2.0.1), their imputs (the imported package 
definition is missing the inputs and native-inputs that we have in the 
Guix package), and the presense of the v2 versioning disambiguation tag.

I was puzzled by this last bit since both packages seem to compatible with 
the v2 API. If I replace the exising package definition with the one 
suggested by the importer (with one small tweak, I needed to add a /v2 to 
the #:unpack-path) all the packages reported by `guix refresh -l 
go-github-com-russross-blackfriday` build sucessfully.

What's the path forward here? I'm thinking I should update our blackfriday 
backage to the 2.1.0 release, but leave the inputs and native inputs. I'm 
less sure about what to do about the v2 tag. Should I switch over the 
package and varialbe name? What do we usually do?

Best,
Jack

The package the importer suggested:

```
(define-public go-github-com-russross-blackfriday-v2
   (package
     (name "go-github-com-russross-blackfriday-v2")
     (version "2.1.0")
     (source
       (origin
         (method git-fetch)
         (uri (git-reference
                (url "https://github.com/russross/blackfriday")
                (commit (string-append "v" version))))
         (file-name (git-file-name name version))
         (sha256
           (base32 "0d1rg1drrfmabilqjjayklsz5d0n3hkf979sr3wsrw92bfbkivs7"))))
     (build-system go-build-system)
     (arguments
       '(#:import-path
         "github.com/russross/blackfriday/v2"
         #:unpack-path
         "github.com/russross/blackfriday"))
     (home-page "https://github.com/russross/blackfriday")
     (synopsis "Blackfriday")
     (description "Package blackfriday is a markdown processor.
")
     (license license:bsd-2)))
```

Our existing package:

```
(define-public go-github-com-russross-blackfriday
   (package
     (name "go-github-com-russross-blackfriday")
     (version "2.0.1")
     (source
      (origin
        (method git-fetch)
        (uri (git-reference
              (url "https://github.com/russross/blackfriday")
              (commit (string-append "v" version))))
        (file-name (git-file-name name version))
        (sha256
         (base32
          "0nlz7isdd4rgnwzs68499hlwicxz34j2k2a0b8jy0y7ycd2bcr5j"))))
     (build-system go-build-system)
     (arguments
      '(#:import-path "github.com/russross/blackfriday"))
     (propagated-inputs
      `(("go-github-com-shurcool-sanitized-anchor-name"
         ,go-github-com-shurcool-sanitized-anchor-name)))
     (native-inputs
      `(("go-github-com-pmezard-go-difflib" ,go-github-com-pmezard-go-difflib)))
     (home-page "https://github.com/russross/blackfriday")
     (synopsis "Markdown processor in Go")
     (description "Blackfriday is a Markdown processor in Go.")
     (license license:bsd-2)))
```

Re: Go importer and packages with version flags

[Katherine Cox-Buday]

Jack Hill <jackhill@jackhill.us> writes:

> Hi Guix,

Hey, Jack, a few thoughts.

> While I was was working with the go importer today, it suggested I package
> go-github-com-russross-blackfriday-v2. Fair enough, except we already have a
> package for go-github-com-russross-blackfriday.

I was poking around a rust code-base the other day and I noticed our crate importer, and thus a lot of crate packages, have major-version suffixes. I think one of the unique benefits of Guix is that it can simultaneously have multiple versions of libraries installed, and I think we should allow this for library packages.

I know that leads to dependency graph explosion, but perhaps we only commit to substitutes for the latest version, and thus any packages using old versions. It should converge over time unless packages go unmaintained.

I thought our current stance was to only allow one version at a time, but the crate packages made me question this. I'd like clarity too.

> The packages differ in their version (2.1.0 verses 2.0.1), their imputs (the
> imported package definition is missing the inputs and native-inputs that we
> have in the Guix package), and the presense of the v2 versioning
> disambiguation tag.

The importer suggested package is correct. It looks like the listed dependencies dropped >= v1.5. See:

- https://pkg.go.dev/github.com/russross/blackfriday@v2.0.0+incompatible?tab=imports
vs
- https://pkg.go.dev/github.com/russross/blackfriday/v2?tab=imports

I wasn't sure if pkg.go listed test dependencies, so I ran this command to be sure:

#+begin_example
$ go list -deps -test -f '{{.ImportPath}} {{.Standard}}' |awk '{if ($2 == "false")  print $1}'
github.com/russross/blackfriday/v2
github.com/russross/blackfriday/v2.test
#+end_example

I think those were just dependencies incorrectly carried over as the package changed.

I hope that helps.

-- 
Katherine

Re: Go importer and packages with version flags

[Sarah Morgensen]

Hi Katherine, Jack,

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> Jack Hill <jackhill@jackhill.us> writes:
>
>> Hi Guix,
>
> Hey, Jack, a few thoughts.
>
>> While I was was working with the go importer today, it suggested I package
>> go-github-com-russross-blackfriday-v2. Fair enough, except we already have a
>> package for go-github-com-russross-blackfriday.
>
> I was poking around a rust code-base the other day and I noticed our crate
> importer, and thus a lot of crate packages, have major-version suffixes. I
> think one of the unique benefits of Guix is that it can simultaneously have
> multiple versions of libraries installed, and I think we should allow this
> for library packages.
>
> I know that leads to dependency graph explosion, but perhaps we only commit
> to substitutes for the latest version, and thus any packages using old
> versions. It should converge over time unless packages go unmaintained.
>
> I thought our current stance was to only allow one version at a time, but
> the crate packages made me question this. I'd like clarity too.

I think there's a bit of a difference between (our packages for) the Rust and
Go ecosystems here.

In the Go ecosystem, a module is uniquely identified by its module path,
e.g. "github.com/russross/blackfriday/v2".  According to Go's major version
suffix rules [0], "[s]tarting with major version 2, module paths must have a
major version suffix like /v2 that matches the major version."  Therefore,
each major version is logically a different module according to Go, and so I
think we should treat them as separate packages as well.  (Note that in many
cases we can use 'inherit' for brevity.)

Additionally, the major version suffix rules dictate that "[i]f an old package
and a new package have the same import path, the new package must be backwards
compatible with the old package."  Assuming upstream sources follow the rules,
we should be able to update each Go package within each major version without
breaking dependencies.

(A corollary to that is that packages often break if you try to use a v2 when
it is expecting a v1.)

I think this differs from Rust, where we have, for example, package-0.1 and
package-0.2 coexisting.  This difference should prevent dependency graph
explosion for Go.

There are some caveats with "major version suffixes":

* Major versions 0 and 1 don't get a version suffix (so no /v1)...

    * ...except for module paths starting with "gopkg.in/", which always have
      a major version suffix, but theirs is styled ".v1" rather than "/v1".

* Modules may either be located in the repository root, or in a "/v2"
  subdirectory (for major version 2).  This makes things difficult for our
  importer, because we can't know whether the unpack path should include "/v2"
  without looking at the repository contents.

This is why Jack had to manually add "/v2" to the unpack path.  I recently
made some changes to the importer to better handle, for example,
"github.com/example/repository/subproject", but it doesn't yet discriminate
between "/subproject" and "/v2", so it treated "/v2" like a subdirectory of
the repository.  (Until we fix this properly, the importer should probably not
count major version suffixes when calculating the unpack path, since most
projects don't use a "/v2" subdirectory.)


All that to say... I think we should definitely maintain coexisting Go v2, v3,
etc. package definitions.  We should probably go the way of Rust though, so we
have them all in the same package, at different versions:

(define-public go-github-com-russross-blackfriday-v2
  (package
    (name "go-github-com-russross-blackfriday")
    (version "2.1.0")

instead of as different packages:

(define-public "go-github-com-russross-blackfriday-v2"
  (package
    (name "go-github-com-russross-blackfriday-v2")
    (version "2.1.0")

And of course, it should be policy to remove dependency packages with no
dependents.  (Perhaps we could write a new linter to warn if a "go-" package
has no inheriters and no dependents?)

Does that sound reasonable?

--
Sarah

Re: Go importer and packages with version flags

[Katherine Cox-Buday]

Sarah Morgensen <iskarian@mgsn.dev> writes:

> Hi Katherine, Jack,
>
> Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:
>
>> Jack Hill <jackhill@jackhill.us> writes:
>>
>>> Hi Guix,
>>
>> Hey, Jack, a few thoughts.
>>
>>> While I was was working with the go importer today, it suggested I package
>>> go-github-com-russross-blackfriday-v2. Fair enough, except we already have a
>>> package for go-github-com-russross-blackfriday.
>>
>> I was poking around a rust code-base the other day and I noticed our crate
>> importer, and thus a lot of crate packages, have major-version suffixes. I
>> think one of the unique benefits of Guix is that it can simultaneously have
>> multiple versions of libraries installed, and I think we should allow this
>> for library packages.
>>
>> I know that leads to dependency graph explosion, but perhaps we only commit
>> to substitutes for the latest version, and thus any packages using old
>> versions. It should converge over time unless packages go unmaintained.
>>
>> I thought our current stance was to only allow one version at a time, but
>> the crate packages made me question this. I'd like clarity too.
>
> I think there's a bit of a difference between (our packages for) the Rust and
> Go ecosystems here.
>
> In the Go ecosystem, a module is uniquely identified by its module path,
> e.g. "github.com/russross/blackfriday/v2".  According to Go's major version
> suffix rules [0], "[s]tarting with major version 2, module paths must have a
> major version suffix like /v2 that matches the major version."  Therefore,
> each major version is logically a different module according to Go, and so I
> think we should treat them as separate packages as well.  (Note that in many
> cases we can use 'inherit' for brevity.)

That's a great point! I hadn't considered that we could leverage this to consider major versioned Go modules as separate packages. That's great!

> Additionally, the major version suffix rules dictate that "[i]f an old package
> and a new package have the same import path, the new package must be backwards
> compatible with the old package."  Assuming upstream sources follow the rules,
> we should be able to update each Go package within each major version without
> breaking dependencies.
>
> (A corollary to that is that packages often break if you try to use a v2 when
> it is expecting a v1.)
>
> I think this differs from Rust, where we have, for example, package-0.1 and
> package-0.2 coexisting.  This difference should prevent dependency graph
> explosion for Go.

It's nice that our Rust packages are enjoying the same stance, but I'm still not clear on why? Does Rust have the same guarantees?

> There are some caveats with "major version suffixes":
>
> * Major versions 0 and 1 don't get a version suffix (so no /v1)...
>
>     * ...except for module paths starting with "gopkg.in/", which always have
>       a major version suffix, but theirs is styled ".v1" rather than "/v1".
>
> * Modules may either be located in the repository root, or in a "/v2"
>   subdirectory (for major version 2).  This makes things difficult for our
>   importer, because we can't know whether the unpack path should include "/v2"
>   without looking at the repository contents.
>
> This is why Jack had to manually add "/v2" to the unpack path.  I recently
> made some changes to the importer to better handle, for example,
> "github.com/example/repository/subproject", but it doesn't yet discriminate
> between "/subproject" and "/v2", so it treated "/v2" like a subdirectory of
> the repository.  (Until we fix this properly, the importer should probably not
> count major version suffixes when calculating the unpack path, since most
> projects don't use a "/v2" subdirectory.)

As an aside, when I started writing the importer, I didn't know it was a possibility to just use the Go toolchain to help us generate packages. I thought "the Guix way" was to do it all in scheme. It's nice that it's in scheme, but I would love to leverage the Go toolchain instead.

IMO, module resolution and graph dependencies in Go are complicated enough that I'd much rather take the effort we put in trying to replicate and keep up with the Go toolchain's behavior, and spend that effort elsewhere.

Does anyone have opinions on this?

> All that to say... I think we should definitely maintain coexisting Go v2, v3,
> etc. package definitions.  We should probably go the way of Rust though, so we
> have them all in the same package, at different versions:
>
> (define-public go-github-com-russross-blackfriday-v2
>   (package
>     (name "go-github-com-russross-blackfriday")
>     (version "2.1.0")
>
> instead of as different packages:
>
> (define-public "go-github-com-russross-blackfriday-v2"
>   (package
>     (name "go-github-com-russross-blackfriday-v2")
>     (version "2.1.0")
>
> And of course, it should be policy to remove dependency packages with no
> dependents.  (Perhaps we could write a new linter to warn if a "go-" package
> has no inheriters and no dependents?)

I disagree with this part, only because it's possible the packages are dependent on non-public (or at least not Guix mainstream) Guix packages. We get the wealth of the commons if we maintain this package in Guix proper. However, I think this is definitely an edge case.

> Does that sound reasonable?

Reasonable? No, incredible! :)

-- 
Katherine

Re: Go importer and packages with version flags

[Sarah Morgensen]

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> Sarah Morgensen <iskarian@mgsn.dev> writes:
>
>> Hi Katherine, Jack,
>>
>> Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:
>>
>>> Jack Hill <jackhill@jackhill.us> writes:
>>>
>>>> Hi Guix,
>>>
>>> Hey, Jack, a few thoughts.
>>>
>>>> While I was was working with the go importer today, it suggested I package
>>>> go-github-com-russross-blackfriday-v2. Fair enough, except we already have a
>>>> package for go-github-com-russross-blackfriday.
>>>
>>> I was poking around a rust code-base the other day and I noticed our crate
>>> importer, and thus a lot of crate packages, have major-version suffixes. I
>>> think one of the unique benefits of Guix is that it can simultaneously have
>>> multiple versions of libraries installed, and I think we should allow this
>>> for library packages.
>>>
>>> I know that leads to dependency graph explosion, but perhaps we only commit
>>> to substitutes for the latest version, and thus any packages using old
>>> versions. It should converge over time unless packages go unmaintained.
>>>
>>> I thought our current stance was to only allow one version at a time, but
>>> the crate packages made me question this. I'd like clarity too.
>>
>> I think there's a bit of a difference between (our packages for) the Rust and
>> Go ecosystems here.
>>
>> In the Go ecosystem, a module is uniquely identified by its module path,
>> e.g. "github.com/russross/blackfriday/v2".  According to Go's major version
>> suffix rules [0], "[s]tarting with major version 2, module paths must have a
>> major version suffix like /v2 that matches the major version."  Therefore,
>> each major version is logically a different module according to Go, and so I
>> think we should treat them as separate packages as well.  (Note that in many
>> cases we can use 'inherit' for brevity.)
>
> That's a great point! I hadn't considered that we could leverage this to
> consider major versioned Go modules as separate packages. That's great!
>
>> Additionally, the major version suffix rules dictate that "[i]f an old package
>> and a new package have the same import path, the new package must be backwards
>> compatible with the old package."  Assuming upstream sources follow the rules,
>> we should be able to update each Go package within each major version without
>> breaking dependencies.
>>
>> (A corollary to that is that packages often break if you try to use a v2 when
>> it is expecting a v1.)
>>
>> I think this differs from Rust, where we have, for example, package-0.1 and
>> package-0.2 coexisting.  This difference should prevent dependency graph
>> explosion for Go.
>
> It's nice that our Rust packages are enjoying the same stance, but I'm still
> not clear on why? Does Rust have the same guarantees?

I'll leave this for someone who actually knows the Rust ecosystem to answer :)

>
>> There are some caveats with "major version suffixes":
>>
>> * Major versions 0 and 1 don't get a version suffix (so no /v1)...
>>
>>     * ...except for module paths starting with "gopkg.in/", which always have
>>       a major version suffix, but theirs is styled ".v1" rather than "/v1".
>>
>> * Modules may either be located in the repository root, or in a "/v2"
>>   subdirectory (for major version 2).  This makes things difficult for our
>>   importer, because we can't know whether the unpack path should include "/v2"
>>   without looking at the repository contents.
>>
>> This is why Jack had to manually add "/v2" to the unpack path.  I recently
>> made some changes to the importer to better handle, for example,
>> "github.com/example/repository/subproject", but it doesn't yet discriminate
>> between "/subproject" and "/v2", so it treated "/v2" like a subdirectory of
>> the repository.  (Until we fix this properly, the importer should probably not
>> count major version suffixes when calculating the unpack path, since most
>> projects don't use a "/v2" subdirectory.)
>
> As an aside, when I started writing the importer, I didn't know it was a
> possibility to just use the Go toolchain to help us generate packages. I
> thought "the Guix way" was to do it all in scheme. It's nice that it's in
> scheme, but I would love to leverage the Go toolchain instead.
>
> IMO, module resolution and graph dependencies in Go are complicated enough
> that I'd much rather take the effort we put in trying to replicate and keep
> up with the Go toolchain's behavior, and spend that effort elsewhere.
>
> Does anyone have opinions on this?

Hmmm.  Setting aside whether or not we want to use external tools in
general...

If we use the Go tool, we shift the problem domain into "translating between
`go' and Guix."

For example: when Go downloads a module, it only saves the module, not the
whole repository*, so we can't use that to generate the hash.  (* Except it
does if you used GOPROXY=direct, but that's in the module cache, and only a
bare repository.)

Or, the fact that we import the latest available version of packages (unless
--pin-versions is used), but Go uses exact versions selected with MVS [0].

You might also be interested in taking a look at Gentoo dealing with this
translation problem [1].

I'm not saying that this would necessarily be a bad tradeoff either, but it's
definitely a tradeoff.

Did you have something particular in mind as far as leveraging the Go tooling?

>
>> All that to say... I think we should definitely maintain coexisting Go v2, v3,
>> etc. package definitions.  We should probably go the way of Rust though, so we
>> have them all in the same package, at different versions:
>>
>> (define-public go-github-com-russross-blackfriday-v2
>>   (package
>>     (name "go-github-com-russross-blackfriday")
>>     (version "2.1.0")
>>
>> instead of as different packages:
>>
>> (define-public "go-github-com-russross-blackfriday-v2"
>>   (package
>>     (name "go-github-com-russross-blackfriday-v2")
>>     (version "2.1.0")
>>
>> And of course, it should be policy to remove dependency packages with no
>> dependents.  (Perhaps we could write a new linter to warn if a "go-" package
>> has no inheriters and no dependents?)
>
> I disagree with this part, only because it's possible the packages are
> dependent on non-public (or at least not Guix mainstream) Guix packages. We
> get the wealth of the commons if we maintain this package in Guix
> proper. However, I think this is definitely an edge case.

Good point.

>> Does that sound reasonable?
>
> Reasonable? No, incredible! :)

Thank you.  I think I've just spent too much time reading Go docs in the past
couple months... not that I even write Go! :)

[0] <https://golang.org/ref/mod#minimal-version-selection>

[1] <https://github.com/golang/go/issues/35922> cmd/go: allow extraction of
urls used to download dependencies

--
Sarah

Could the Go importer use the Go toolchain? (was Re: Go importer and packages with version flags)

[Katherine Cox-Buday]

Sarah Morgensen <iskarian@mgsn.dev> writes:

>> As an aside, when I started writing the importer, I didn't know it was a
>> possibility to just use the Go toolchain to help us generate packages. I
>> thought "the Guix way" was to do it all in scheme. It's nice that it's in
>> scheme, but I would love to leverage the Go toolchain instead.
>>
>> IMO, module resolution and graph dependencies in Go are complicated enough
>> that I'd much rather take the effort we put in trying to replicate and keep
>> up with the Go toolchain's behavior, and spend that effort elsewhere.
>>
>> Does anyone have opinions on this?
>
> Hmmm.  Setting aside whether or not we want to use external tools in
> general...
>
> If we use the Go tool, we shift the problem domain into "translating between
> `go' and Guix."
>
> For example: when Go downloads a module, it only saves the module, not the
> whole repository*, so we can't use that to generate the hash.  (* Except it
> does if you used GOPROXY=direct, but that's in the module cache, and only a
> bare repository.)

We could use the ~GOMODCACHE~ environment variable[0] to specify where these files are placed.

> Or, the fact that we import the latest available version of packages (unless
> --pin-versions is used), but Go uses exact versions selected with MVS [0].

Won't the importer import the import path you specify? The Go toolchain can also specify a ~@latest~ "version" to fetch the latest.

> You might also be interested in taking a look at Gentoo dealing with this
> translation problem [1].
>
> I'm not saying that this would necessarily be a bad tradeoff either, but it's
> definitely a tradeoff.

Agreed. I think the problem space of managing the translation between the toolchain and Guix is probably both smaller and less volatile.

I.e., the toolchain operations that dump dependency graph type stuff tend to support JSON output, and the download bits have options for where stuff gets stored on disk. As you know, the steps to go from an import path to a repository on-disk is complicated. If we could keep that complexity encapsulated upstream, which also happens to be the reference implementation, it could probably make the importer resilient to change and maybe even faster.

> Did you have something particular in mind as far as leveraging the Go tooling?

I haven't though too much about it, but ~GOPROXY=direct~ was my guess too. Thinking about this, would a shallow fetch into a bare repository be so bad?

It made me start wondering why our ~git-download~ type /doesn't/ have this behavior. Are we unnecessarily pulling down the entire commit history with all tags and branches of a large repository?

I don't even know what the philosophy is being doing this or not. Isn't the only commitment a Guix package makes for software at a particular version/commit that the store contains that immutable version of the source, not the complete repository?

Also, Guix supports all kinds of download types, one of which is zip files. The rust importer generates packages that fetch ~.tar.gz~'s from https://create.io. When I started writing the importer, I specifically kept the packages it generated decoupled from Google's module cache, but to make the comparison: the Go importer could do the same thing as the Rust importer: download zips from https://proxy.golang.org.

If this line of reasoning is wrong, it also looks like the toolchain respects standard VCS configuration. I wonder if there's a way to manipulate the fetches so that they're not shallow or bare.

TLDR: A shallow fetch into a bare repository might actually be /ideal/! It's small, and still directly connected to the package's actual source.

> [0] <https://golang.org/ref/mod#minimal-version-selection>
>
> [1] <https://github.com/golang/go/issues/35922> cmd/go: allow extraction of
> urls used to download dependencies
>
> --
> Sarah

[0] - https://golang.org/ref/mod#environment-variables
[1] - https://golang.org/ref/mod#private-module-repo-auth

-- 
Katherine

Re: Could the Go importer use the Go toolchain? (was Re: Go importer and packages with version flags)

[raingloom]

On Thu, 30 Sep 2021 10:31:08 -0500
Katherine Cox-Buday <cox.katherine.e@gmail.com> wrote:

> Sarah Morgensen <iskarian@mgsn.dev> writes:
> > Did you have something particular in mind as far as leveraging the
> > Go tooling?  
> 
> I haven't though too much about it, but ~GOPROXY=direct~ was my guess
> too. Thinking about this, would a shallow fetch into a bare
> repository be so bad?
> 
> It made me start wondering why our ~git-download~ type /doesn't/ have
> this behavior. Are we unnecessarily pulling down the entire commit
> history with all tags and branches of a large repository?
> 
> I don't even know what the philosophy is being doing this or not.
> Isn't the only commitment a Guix package makes for software at a
> particular version/commit that the store contains that immutable
> version of the source, not the complete repository?

AFAIK there is a long standing libgit2 bug for adding support for
shallow fetches. Or rather operations on shallow repositories.
https://github.com/libgit2/libgit2/issues/3058

So guile-git can't have shallow fetches until libgit2 gets its act
together. Or Guix could use a patched libgit2 I guess.

Re: Could the Go importer use the Go toolchain? (was Re: Go importer and packages with version flags)

[François]

Hello,

After a hiatus I am trying to package several softwares written on Golang
(I would like to have terraform and go-jsonnet for example) and I have
some problems with the current implementation so I am resurrecting this
old mail from my Draft folder.

Looking at it I think it is mostly reformulations of what Katherine and
Sarah already wrote on the thread but several months later it could be
a useful reminder.

I have not really the energy at the moment to work on this alone but I
would be interested for team work.

On Thu, Sep 30, 2021 at 10:31:08AM -0500, Katherine Cox-Buday wrote:
> Sarah Morgensen <iskarian@mgsn.dev> writes:
> 
> >> IMO, module resolution and graph dependencies in Go are complicated enough
> >> that I'd much rather take the effort we put in trying to replicate and keep
> >> up with the Go toolchain's behavior, and spend that effort elsewhere.
> >>
> >> Does anyone have opinions on this?
> >
> > Hmmm.  Setting aside whether or not we want to use external tools in
> > general...
> >
> > If we use the Go tool, we shift the problem domain into "translating between
> > `go' and Guix."

I felt uneasy when we reimplemented some inherently Go-specific pieces
of software like go.mod file parsing and dependency resolution. It seems
so brittle. So I think that for importer specifically we should be able
to use external programs like the Go toolchain to be in $PATH. And,
given the Go tool would have proper interfaces, we would indeed just
have to do some sort of translation.

But as to the exact "how" it is not easy.

Ideally we could split the tasks in severals steps.
1. identify all dependencies on form of go modules paths
2. identify source repositories (generally identified by a git-ref) for each of those modules
3. download content of source repository
4. populate a ready-to-use source code local cache
5. build

As I see it in Guix we want :
- 1 and 2 in the importer;
- 3 is the "origin" method;
- 4 and 5 in the build system.

`go mod download` does all of 1, 2 and 3. The translations problems are
how to extract data from this output to isolate dependency management on
the importer and having the necessary info for the "origin" method. Nix
bypass completely the problem by using the complete go mod cache as its
"origin" (or more recently "vendored" dependencies see this Nix change
to go mod vendor[4]) for each Go package. This solution is not viable
for us as we want to be able to reuse Go modules.

For the build system part I think we want to use the GOMODCACHE format as
output of Guix package and to be able to do union-dir of all dependencies
when we want to build a leaf package (e.g. using GOPROXY=file:/// or
something like that).

> > For example: when Go downloads a module, it only saves the module, not the
> > whole repository*, so we can't use that to generate the hash.  (* Except it
> > does if you used GOPROXY=direct, but that's in the module cache, and only a
> > bare repository.)
>
> We could use the ~GOMODCACHE~ environment variable[0] to specify where these files are placed.

Yes we want to do that but it leaves open as to how we create this
directory structure. I think it is good to use Go tooling at "guix import"
time but not at build time. In consequence the source must be downloaded
either from upstream git repo (bare or not I do not know) or as zips
from proxy.golang.org. The zip solution could just parse the directory
contents of GOMODCACHE as described in [5]. I would rather import git
repositories closer to real source but it seems more difficult to do.

Even with zip, it leaves open the problem of mapping source zips with
Guix packages and to handle the dependencies correctly.

Any interest on the matter ?

François

[4]: https://github.com/NixOS/nixpkgs/pull/86376/commits/9761128d2da7bf4d878982918242e43ae28f9b94

[5]: https://github.com/golang/go/issues/35922#issuecomment-560464243

    cd go/src/golang.org/x/tools/gopls   # or any other module
    export GOPATH=$(mktemp -d)
    go mod download
    find $GOPATH/pkg/mod/cache/download -type f | \
        grep '\.\(mod\|info\|zip\)$' | \
        sed -e "s,$GOPATH/pkg/mod/cache/download,https://proxy.golang.org,"

Re: Could the Go importer use the Go toolchain? (was Re: Go importer and packages with version flags)

[Maxim Cournoyer]

Hi François,

Good to see interest being picked up on the Go front.

François <francois-oss@avalenn.eu> writes:

> Hello,
>
> After a hiatus I am trying to package several softwares written on Golang
> (I would like to have terraform and go-jsonnet for example) and I have
> some problems with the current implementation so I am resurrecting this
> old mail from my Draft folder.
>
> Looking at it I think it is mostly reformulations of what Katherine and
> Sarah already wrote on the thread but several months later it could be
> a useful reminder.
>
> I have not really the energy at the moment to work on this alone but I
> would be interested for team work.
>
> On Thu, Sep 30, 2021 at 10:31:08AM -0500, Katherine Cox-Buday wrote:
>> Sarah Morgensen <iskarian@mgsn.dev> writes:
>> 
>> >> IMO, module resolution and graph dependencies in Go are complicated enough
>> >> that I'd much rather take the effort we put in trying to replicate and keep
>> >> up with the Go toolchain's behavior, and spend that effort elsewhere.
>> >>
>> >> Does anyone have opinions on this?
>> >
>> > Hmmm.  Setting aside whether or not we want to use external tools in
>> > general...
>> >
>> > If we use the Go tool, we shift the problem domain into "translating between
>> > `go' and Guix."
>
> I felt uneasy when we reimplemented some inherently Go-specific pieces
> of software like go.mod file parsing and dependency resolution. It seems
> so brittle. So I think that for importer specifically we should be able
> to use external programs like the Go toolchain to be in $PATH. And,
> given the Go tool would have proper interfaces, we would indeed just
> have to do some sort of translation.

It's true, that we could perhaps make use of Go for the importer, if
that simplify things and make them more robust in the long term.  I do
not expect these file formats (go.mod) to change too much though, so the
brittleness concern may be exaggerated.  In general, there's a tendency
in Guix to do everything in Guile, for better and worst.

> But as to the exact "how" it is not easy.
>
> Ideally we could split the tasks in severals steps.
> 1. identify all dependencies on form of go modules paths
> 2. identify source repositories (generally identified by a git-ref) for each of those modules
> 3. download content of source repository
> 4. populate a ready-to-use source code local cache
> 5. build
>
> As I see it in Guix we want :
> - 1 and 2 in the importer;
> - 3 is the "origin" method;
> - 4 and 5 in the build system.
>
> `go mod download` does all of 1, 2 and 3. The translations problems are
> how to extract data from this output to isolate dependency management on
> the importer and having the necessary info for the "origin" method. Nix
> bypass completely the problem by using the complete go mod cache as its
> "origin" (or more recently "vendored" dependencies see this Nix change
> to go mod vendor[4]) for each Go package. This solution is not viable
> for us as we want to be able to reuse Go modules.
>
> For the build system part I think we want to use the GOMODCACHE format as
> output of Guix package and to be able to do union-dir of all dependencies
> when we want to build a leaf package (e.g. using GOPROXY=file:/// or
> something like that).

I agree this seems the way to go.

[...]

> Any interest on the matter ?

I think supporting GO modules properly is going to be a must have as Go
moves forward in that direction.  I am interested to have that, but I
don't think I have the bandwidth to volunteer much on the coding front.

I'd be happy to review things or brainstorm though.

-- 
Thanks,
Maxim