Packaging gx (for IPFS): Need to update default Go to 1.11?

Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 549 bytes --]

Hi Guix!

I'm trying to package gx (a package manager written in Go), but it
requires Go 1.11+ to run properly.

Currently all Go packages are built with the default Go version which is
1.9.

Is there a good reason for sticking to 1.9 or should we update to 1.11?

If we've got to stick to 1.11, how should I proceed to package gx?
I  could define a new build-system like %bootstrap-guile does.  Is there
a convenient to just change the Go version of the "lower" of the
go-build-system?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 151 bytes --]

Correction: gx itself does not need Go 1.11, but go-ipfs requires that gx be
compiled with Go 1.11+.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 566 bytes --]

On Wed, Oct 24, 2018 at 01:40:26PM +0200, Pierre Neidhardt wrote:
> Is there a good reason for sticking to 1.9 or should we update to 1.11?

There are two reasons.

First, the implementation of go-build-system is really inefficient for
Go 1.11, especially since things compiled with Go 1.11 keep a huge
run-time dependency graph:

https://bugs.gnu.org/32949

That could probably be fixed based on the info in that thread.

Second, there are packages that don't yet support Go 1.9. Syncthing is a
case where there are dependencies that need to be updated and tested.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]


> First, the implementation of go-build-system is really inefficient for
> Go 1.11, especially since things compiled with Go 1.11 keep a huge
> run-time dependency graph:

Is it _only_ inefficient because of issue 32949 or is there another reason?

> https://bugs.gnu.org/32949
> 
> That could probably be fixed based on the info in that thread.

OK, if Gábor's clue happens to be the right one, it should not be too hard.
I'll look into it later.

> Second, there are packages that don't yet support Go 1.9. Syncthing is a
> case where there are dependencies that need to be updated and tested.

I think you meant Go 1.11 ;)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 666 bytes --]

On Wed, Oct 24, 2018 at 04:02:46PM +0200, Pierre Neidhardt wrote:
> 
> > First, the implementation of go-build-system is really inefficient for
> > Go 1.11, especially since things compiled with Go 1.11 keep a huge
> > run-time dependency graph:
> 
> Is it _only_ inefficient because of issue 32949 or is there another reason?

There is also <https://bugs.gnu.org/32919> but IMO it's not a blocker
considering that Go 1.9 is no longer supported upstream.

> > Second, there are packages that don't yet support Go 1.9. Syncthing is a
> > case where there are dependencies that need to be updated and tested.
> 
> I think you meant Go 1.11 ;)

Yes :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 196 bytes --]

I see you've fixed this on November 1st.  I'll test Syncthing with go 1.11.  If
it works, do you think it's safe to move to 1.11 for the default?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

Also, if we move to go 1.11, should we remove go 1.9?  Considering there is the
slowdown mentioned in https://bugs.gnu.org/32919, maybe it would be smarter to
keep 1.9 around?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

Actually bug 32919 matters, it breaks Demlo and all packages that depend on
packages that need special compilation flags (e.g. -tags "xyz").

Concretely, say A depends on B and B must be built with "-tags xyz".
When packaging B, we produce a pkg.
If we try to package A _without the tags flag_, since B is not re-used, Go will
rebuild B without the flag which will not produce the desired result.

The most obvious workaround is to pass the right flags to A.  But if A needs
different flags, then we are stuck.

Conclusion: we need to update the build system! :D

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Tue, Nov 06, 2018 at 06:04:54PM +0100, Pierre Neidhardt wrote:
> Actually bug 32919 matters, it breaks Demlo and all packages that depend on
> packages that need special compilation flags (e.g. -tags "xyz").
> 
> Concretely, say A depends on B and B must be built with "-tags xyz".
> When packaging B, we produce a pkg.
> If we try to package A _without the tags flag_, since B is not re-used, Go will
> rebuild B without the flag which will not produce the desired result.
> 
> The most obvious workaround is to pass the right flags to A.  But if A needs
> different flags, then we are stuck.
> 
> Conclusion: we need to update the build system! :D

Hm, quel dommage!

If most of the Go packages are ready for Go 1.11, we could make it the
dfeault and then use Go 1.9 for the packages that are lagging behind. Or
vice versa.

In general, we should try to move to 1.11 since 1.9 is no longer
supported upstream.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 238 bytes --]


> If most of the Go packages are ready for Go 1.11, we could make it the
> dfeault and then use Go 1.9 for the packages that are lagging behind. Or
> vice versa.

How do you do that?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1413 bytes --]

On Tue, Nov 06, 2018 at 06:33:53PM +0100, Pierre Neidhardt wrote:
> > If most of the Go packages are ready for Go 1.11, we could make it the
> > dfeault and then use Go 1.9 for the packages that are lagging behind. Or
> > vice versa.
> 
> How do you do that?

The default Go is defined in ((guix build-system go) default-go).
Updates can be tested with this diff:

diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index cf9116327..80894677f 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -44,7 +44,7 @@
 (define (default-go)
   ;; Lazily resolve the binding to avoid a circular dependency.
   (let ((go (resolve-interface '(gnu packages golang))))
-    (module-ref go 'go)))
+    (module-ref go 'go-1.11)))
 
 (define* (lower name
                 #:key source inputs native-inputs outputs system target


And individual packages can be made to use non-default Go compilers with
the #:go key, like this:

diff --git a/gnu/packages/curl.scm b/gnu/packages/curl.scm
index 2fb87251f..4c8c83d84 100644
--- a/gnu/packages/curl.scm
+++ b/gnu/packages/curl.scm
@@ -171,6 +171,7 @@ tunneling, and so on.")
     (arguments
      `(#:import-path "gitlab.com/davidjpeacock/kurly"
        #:install-source? #f
+       #:go ,go-1.11
        #:phases
        (modify-phases %standard-phases
          (add-after 'install 'install-documentation


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 133 bytes --]

Oh, cool, I did not know about the #:go key!
Great, I'll udpate the patchset then!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

Hmm, actually it seems that go1.9 is not able to build a package against
dependencies built with go1.11.

For Demlo, that would mean that I need to add the 1.9 specialization to 10
packages, but this is brittle and it will conflict with other programs that
share some dependencies while using go1.11.

Suggestion?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

On Wed, Nov 07, 2018 at 05:22:44PM +0100, Pierre Neidhardt wrote:
> Hmm, actually it seems that go1.9 is not able to build a package against
> dependencies built with go1.11.

Yeah, I don't think there is any notion of backwards compatibility
there.

> For Demlo, that would mean that I need to add the 1.9 specialization to 10
> packages, but this is brittle and it will conflict with other programs that
> share some dependencies while using go1.11.
> 
> Suggestion?

Maybe you could take a look at how we handle packaging for both Python 3
and Python 2? There, we have package variants for both Python versions,
as necessary.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 244 bytes --]

Sure, but isn't this too much a hassle (and more pollution added to the package
namespace) for a temporary workaround?

Shouldn't we focus on fixing the cache bug in the build system instead?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 549 bytes --]

On Wed, Nov 07, 2018 at 09:09:24PM +0100, Pierre Neidhardt wrote:
> Sure, but isn't this too much a hassle (and more pollution added to the package
> namespace) for a temporary workaround?

It's a matter of taste :)

> Shouldn't we focus on fixing the cache bug in the build system instead?

Yes, we should, but that problem is not related to packages not building
with Go 1.11 because they need to be updated upstream, right? Even if
our go-build-system was updated, those packages would still need to be
built with Go 1.9? Or did I misunderstand?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]


> Yes, we should, but that problem is not related to packages not building
> with Go 1.11 because they need to be updated upstream, right? Even if
> our go-build-system was updated, those packages would still need to be
> built with Go 1.9? Or did I misunderstand?

In my patchset, there are 2 kinds of fixes:

- Disabling tests because Go 1.11 test policies are stricter and some lax code
  from before does not pass anymore.
  For this issue, upstream should fix their packages.  As far as we are
  concerned, disabling tests is enough.  The go-build-system has nothing to do
  with this issue.

- Manual forwarding of compilation flags (e.g. "-tags").  This is an issue with
  our go-build-system which cannot re-use existing "pkg" objects on Go 1.11.
  Upstream code is fine.

So yes, fixing go-build-system is related to packages not building with
Go 1.11 :p

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Wed, Nov 07, 2018 at 09:48:50PM +0100, Pierre Neidhardt wrote:
> In my patchset, there are 2 kinds of fixes:
> 
> - Disabling tests because Go 1.11 test policies are stricter and some lax code
>   from before does not pass anymore.
>   For this issue, upstream should fix their packages.  As far as we are
>   concerned, disabling tests is enough.  The go-build-system has nothing to do
>   with this issue.

Okay, I think you should go ahead and push these changes now, assuming
the packages still build afterwards.

> - Manual forwarding of compilation flags (e.g. "-tags").  This is an issue with
>   our go-build-system which cannot re-use existing "pkg" objects on Go 1.11.
>   Upstream code is fine.
> 
> So yes, fixing go-build-system is related to packages not building with
> Go 1.11 :p

Okay, I see. I won't be working on this soon, if ever. I don't
understand enough about how Go >= 1.10 builds software in order to do
the work efficiently. Hopefully there is a volunteer :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2644 bytes --]

I've been studying the situation:

From https://golang.org/doc/go1.10#build:

> The go build command now maintains a cache of recently built packages,
> separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect
> of the cache should be to speed builds that do not explicitly install packages
> or when switching between different copies of source code (for example, when
> changing back and forth between different branches in a version control system).

From https://golang.org/doc/go1.11:

> Go 1.11 will be the last release to support setting the environment variable
> GOCACHE=off to disable the build cache, introduced in Go 1.10. Starting in Go
> 1.12, the build cache will be required, as a step toward eliminating
> $GOPATH/pkg. The module and package loading support described above already
> require that the build cache be enabled

My understanding is that we cannot rely on the /pkg/ folder anymore, since Go
1.12 won't use it at all.

We could maintain a store-level Go cache, but I'm not sure that's doable and I
don't think it makes much sense.  Without a cache, it seems that it's
effectively impossible to re-use pre-built libraries when building Go packages.
From the user perspective, this would not be a big deal because the cache can be
used in this context.  Only the first build would be slower.

More importantly, that would mean that we no longer need to build Go libraries:
we should simply ship the source code in /src/.  The upside is that it saves
space in the store and makes for smaller substitutes.

For the other issue with flag forwarding, there might be a better solution. 
From https://golang.org/doc/go1.10#build:

> The go build -asmflags, -gcflags, -gccgoflags, and -ldflags options now apply by
> default only to the packages listed directly on the command line. For example,
> go build -gcflags=-m mypkg passes the compiler the -m flag when building mypkg
> but not its dependencies. The new, more general form -asmflags=pattern=flags
> (and similarly for the others) applies the flags only to the packages matching
> the pattern. For example: go install -ldflags=cmd/gofmt=-X=main.version=1.2.3
> cmd/... installs all the commands matching cmd/... but only applies the -X
> option to the linker flags for cmd/gofmt. For more details, see go help build.

I think it does not work for "-tags" however.  So we need to ask golua's
maintainer if there is a solution to build golua differently.
If not, then the best solution I can think of is to patch golua to build with
lua5.3 by default.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 186 bytes --]

Actually I think I'll have to patch golua.  According to
https://golang.org/cmd/cgo/, CGO flags cannot be customized from command line.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 189 bytes --]

One last detail: from go-1.10 onward, we will need to turn all CGO input to
propagated inputs, otherwise referrers won't be able to build.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 137 bytes --]

Alright, I've applied the changes and updated to go-1.11.
Should we remove go-1.9 now?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging gx (for IPFS): Need to update default Go to 1.11?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 382 bytes --]

On Mon, Nov 12, 2018 at 10:45:07AM +0100, Pierre Neidhardt wrote:
> Alright, I've applied the changes and updated to go-1.11.
> Should we remove go-1.9 now?

If all of our Go packages are still working with Go 1.11, then it should
be safe to remove Go 1.9.

We should take care to handle the inheritance of Go 1.11 on Go 1.9 on Go
1.4 correctly. Unfortunately it's a bit tangled...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]