[bug#50227] [PATCH 0/3] go-build-system and GOPATH improvements

[Sarah Morgensen <iskarian@mgsn.dev>]

Hello,

Marius Bakke <marius@gnu.org> writes:

> That is excellent news, thank you!  I briefly looked into "modules"
> while researching this, but realized that the rabbit hole would be too
> deep for me.  I did not notice that GOPATH was being deprecated however.

The Go 1.16 release notes actually stated that a deprecation notice
would be printed in Go 1.17 but then they didn't do that!  I think
they're waiting for the "workspaces" feature [0] to be implemented
first.

[0] https://github.com/golang/go/issues/45713

>> 4. Use -trimpath instead of remove-go-references, as you did.  Also, to
>> avoid rebuilding the standard library with '-trimpath' for every package
>> (since the Go build cache does not persist between build environments):
>>
>>   a) modify the Go package to build standard libraries with -trimpath,
>>      which would unfortunately mean most users of the Go package would
>>      find that ~180MB of space wasted; or
>>   
>>   b) build a '-trimpath' version of the standard library separately and
>>      use it with '-pkgdir' (which would prevent #3 from working) or by
>>      building a directory union of Go and Go-std-library-with-trimpath
>>      and setting GOROOT=/path/to/union.
>>
>> Personally, I'm partial to a), along with removing the pre-compiled
>> standard library from the Go package since it ends up recompiled more
>> often than not, is very fast to recompile, and it will eventually no
>> longer be distributed or used by Go [0].
>
> Removing the compiled libraries sounds fine to me.  I suppose we'll
> still need -trimpath for executables ("main.go")?

Yes, we'll need '-trimpath' in all invocations of build tools, including
'test' as well, or it will actually recompile everything without
'-trimpath' and then test that!

>> (A digression: the current issue with fully implementing module-aware
>> mode is that Go really wants a specific version for each dependency.  If
>> we just populate the module cache with the versions we have, it will
>> inevitable complain when a package we try to build wants a version we do
>> not have.  I see a few solutions:
>>
>> 1. Put all dependencies in the module cache, and rewrite the main
>> module's go.mod (that is, add replace directives) to replace all
>> dependencies with the versions we have.
>>
>> 2. Rewrite the go.mod to replace all dependencies with the local
>> filesystem path of the versions we have.
>>
>> 3. Put all dependencies in the vendor/ directory, and use -mod=vendor.
>> Any pre-existing vendor directory must be handled properly.
>>
>> These three solutions fail to allow re-using the build cache (and
>> therefore build artifacts), because Go computes the build cache keys
>> differently for main and non-main modules.  Building in Go is generally
>> fast, so we probably shouldn't compromise much to enable reusing the
>> build cache, but a few ideas for doing so:
>>
>> 4. Set up a dummy go.mod out of the source tree, which 'replace's all
>> dependencies AND the module we're building like in 1) or 2).  This may
>> have to account for replace directives in the go.mod of the module we're
>> building, though.
>>
>> 5. Put the module we're building in the module cache, and build it with
>> "go install module@version".  The same caveat as in 4) applies, as well
>> as that "go install module@version" only works for main packages (that
>> is, packages which produce an executable).)
>
> Thank you for this analysis.  The vendoring option is compelling, if it
> does not require patching the go.mod files, and can work also for
> packages where unbundling is not feasible (or downstream channels with
> less strict packaging policies).

Yes, and it has the upside that '-mod=vendor' automatically disables
network access for most tools.

On the other hand, I think the only solution that properly moves in the
direction of allowing hacking on Go packages is the GOPROXY search-path
approach.

This requires some more thought, for sure.

> For reusing build artifacts, perhaps we can piggy-back on whatever is
> implemented for Bazel as mentioned in [0].

Bazel is an entire build system replacing cmd/go, which (presumably)
constructs its own build graph, etc, so probably a little heavy-handed
for our use-case.  We probably want to stay as close as possible to
vanilla tooling.

Probably the best we can do is saving and re-using the build cache,
since they seem insistent on moving everything into the cache.

> Given the conflicting work here, what do you think we should do?  I'm
> happy to scrap this PR as it was largely an exercise to learn
> go-build-system, in addition to scratching a very minor itch.
>
> Is the reduced complexity worth it while waiting for the gomodules
> rewrite, and if so, are there parts that can be merged with your work
> such as using $out/share/go?
>
> Let me know if I can be of assistance.  :-)
>
>> [0] https://github.com/golang/go/issues/47257

I think I would suggest breaking up the Go build system changes into:

1. Make the changes roughly included in your patch, along with making a
"go-std-cache-for-build" package (hidden?) which will be an implicit
input in the Go build system (perhaps non-substitutable? it will be
faster to build than download on nearly all machines), and seeding the
build cache with it in 'setup-go-environment'.  We skip re-using build
artifacts.

2. Update Go build system to use Go 1.16, leaving only docker with Go
1.14 (via "#:go ,go-1.14").  We should be ready to do this as soon as
[0] is fixed [1].

3. Enable building multiple Go packages in one Guix package, and merge
all Guix packages such that one Guix package == one module path.

4. Make gomodules changes.

5. Release Go 1.18, which will bootstrap from Go 1.16 or Gccgo 11 [2].

6. Update Go build system to use Go 1.18.

[0] https://issues.guix.gnu.org/49921
[1] https://logs.guix.gnu.org/guix/2021-08-11.log#213401
[2] <https://github.com/golang/go/issues/44505> build: Adopt Go 11.6 as
bootstrap toolchain for Go 1.18

--
Sarah