go-build-system possible improvments

go-build-system possible improvments

[François]

Hello,

I come back to public guix-devel as I think it can be of interest to
others too.

On Sat, Apr 17, 2021 at 08:57:03PM -0400, Maxim Cournoyer wrote:
> JOULAUD François writes:
> 
> > I am still unsure of what to do with versions and Go but the ability to
> > pin version will surely be useful in some way.
> 
> Yeah, I was expecting it might unlock building the new protobuf package
> especially a couple version backs, it had a really messy dependency
> chain including past versions of itself, but it turns out that for now
> the bigger problem is the lack of support for Go modules.

Yes. This is the biggest lock. Supporting go modules will really unlock
a lot of things.

> I'm glad to hear it!  I had read lots about Go and had some kind of plan
> for the GOPROXY (the project doing best in this regard is Gentoo); in
> theory it's possible to populate a cache with the source artifacts in a
> given directory (probably via a union of the individual packages
> contribution to the cache), then setup GOPROXY as a file server to
> provide these at build time.

I just made a lot of reading this week-end and I was more oriented towards
using GOPROXY=off and populating on-disk [module cache] which have the
exact same organisation as GOPROXY as it is indeed the last layer of
proxying. If it works it will avoid the need for a specific local server.

Concerning the former dependeency loop with google-protobuf it was
(before 1.26) indeed necessary to have all go.mod files for all versions
referenced recursively. Go downloads (or need to have in cache) only
the go.mod files though because once it have the go.mod the [Minimal
Version Selection] kicks in and keep only the higher version
requested[^1].

So we could cheat by putting only go.mod files for older versions in
out local cache without the need for anything else and it should work.

The same thing applies to conditional compilation. Go needs the go.mod
files but don't need the code itself until it tries to compiles it.

So we have probably a safe way to import those packages without too
many useless packages in our dependency graph even if it can require a
solution to easily add specific go.mod files in our outputs.

They, at Go, are nevertheless [working][lazy-loading] to modify this
behaviour in order to avoid useless network calls to GOPROXY but it
could not land in Go 1.15. It should be there in next release.


[module cache]: https://golang.org/ref/mod#module-cache
[lazy-loading]: https://go.googlesource.com/proposal/+/master/design/36460-lazy-module-loading.md
[Minimal Version Selection]: https://research.swtch.com/vgo-mvs
[^1]: Which seems strange until you understand that you always require a
"minimal" version in Go. The minimal version needed to build a project
is thus the maximum of all minimal required versions. The link explains
this a lot better than me ;-)

François

P.S. just as writing this I found that if we want to limit the number
of versions of Guix-packaged Go modules we could devise a mechanism to
force upgrade (go get -u) to the latest in-Guix version. This open a
lots more question to me than it resolves.

P.P.S. other solution, perhaps simpler is to not support go.mod at all
(GO11MODULE=off) and to populate the local GOPATH as we see fit. I think
(but I am not sure) that Debian took this route. Not found of this
solution, I think we have the means to do better in Guix and to keep
more in touch with upstream methodologies.

Re: go-build-system possible improvments

[Maxim Cournoyer]

Hi François,

François <francois-oss@avalenn.eu> writes:

> Hello,
>
> I come back to public guix-devel as I think it can be of interest to
> others too.
>
> On Sat, Apr 17, 2021 at 08:57:03PM -0400, Maxim Cournoyer wrote:
>> JOULAUD François writes:
>>
>> > I am still unsure of what to do with versions and Go but the ability to
>> > pin version will surely be useful in some way.
>>
>> Yeah, I was expecting it might unlock building the new protobuf package
>> especially a couple version backs, it had a really messy dependency
>> chain including past versions of itself, but it turns out that for now
>> the bigger problem is the lack of support for Go modules.
>
> Yes. This is the biggest lock. Supporting go modules will really unlock
> a lot of things.

[...]

> I just made a lot of reading this week-end and I was more oriented towards
> using GOPROXY=off and populating on-disk [module cache] which have the
> exact same organisation as GOPROXY as it is indeed the last layer of
> proxying. If it works it will avoid the need for a specific local server.

That'd be nice!

> Concerning the former dependeency loop with google-protobuf it was
> (before 1.26) indeed necessary to have all go.mod files for all versions
> referenced recursively. Go downloads (or need to have in cache) only
> the go.mod files though because once it have the go.mod the [Minimal
> Version Selection] kicks in and keep only the higher version
> requested[^1].
>
> So we could cheat by putting only go.mod files for older versions in
> out local cache without the need for anything else and it should work.
>
> The same thing applies to conditional compilation. Go needs the go.mod
> files but don't need the code itself until it tries to compiles it.
>
> So we have probably a safe way to import those packages without too
> many useless packages in our dependency graph even if it can require a
> solution to easily add specific go.mod files in our outputs.

Interesting ideas!  I didn't know that.  My high level idea was to mimic
GOPATH but with GOPROXY; instead of source files, we'd have the metadata
+ zip (sources) populated in a given directory (say, the gocache/
subdirectory) of every Go package in Guix, and then could build a union
of these various directories into one (matching the expected GOPROXY
file layout) in the Go build system and profile hook.

> They, at Go, are nevertheless [working][lazy-loading] to modify this
> behaviour in order to avoid useless network calls to GOPROXY but it
> could not land in Go 1.15. It should be there in next release.

You meant Go 1.16, right?  I'm happy if they're working to simplify
this.

> P.P.S. other solution, perhaps simpler is to not support go.mod at all
> (GO11MODULE=off) and to populate the local GOPATH as we see fit. I think
> (but I am not sure) that Debian took this route. Not found of this
> solution, I think we have the means to do better in Guix and to keep
> more in touch with upstream methodologies.

That's exactly what we're doing now (disable Go module with
GO11MODULE=off and use GOPATH to find the sources).  Debian does
something lazy like just calling 'go module vendor', which builds a huge
directory filled with all the sources needed.  So does Nix.  Only Gentoo
goes to some length, from what I had seen last I check.

Thank you,

Maxim

Re: go-build-system possible improvments

[François]

Hello Maxim,

On Tue, Apr 20, 2021 at 10:00:15PM -0400, Maxim Cournoyer wrote:
> That's exactly what we're doing now (disable Go module with
> GO11MODULE=off and use GOPATH to find the sources).  Debian does
> something lazy like just calling 'go module vendor', which builds a huge
> directory filled with all the sources needed.  So does Nix.  Only Gentoo
> goes to some length, from what I had seen last I check.

Do you have some links to share on how Gentoo does it?

Re: go-build-system possible improvments

[Maxim Cournoyer]

Hi François,

François <francois-oss@avalenn.eu> writes:

> Hello Maxim,
>
> On Tue, Apr 20, 2021 at 10:00:15PM -0400, Maxim Cournoyer wrote:
>> That's exactly what we're doing now (disable Go module with
>> GO11MODULE=off and use GOPATH to find the sources).  Debian does
>> something lazy like just calling 'go module vendor', which builds a huge
>> directory filled with all the sources needed.  So does Nix.  Only Gentoo
>> goes to some length, from what I had seen last I check.
>
> Do you have some links to share on how Gentoo does it?

I think I ended up looking in their git repo, but otherwise there's this
page for their go-module.eclass module:
https://devmanual.gentoo.org/eclass-reference/go-module.eclass/index.html.

HTH,

Maxim