bug#33620: Golang programs keeping references [gnu: go: Update default to 1.11.]

bug#33620: Golang programs keeping references [gnu: go: Update default to 1.11.]

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

----- Forwarded message from Leo Famulari <leo@famulari.name> -----

Date: Mon, 12 Nov 2018 12:29:25 -0500
From: Leo Famulari <leo@famulari.name>
To: guix-devel@gnu.org
Cc: Pierre Neidhardt <ambrevar@gmail.com>
Subject: Golang programs keeping references [gnu: go: Update default to 1.11.]
User-Agent: Mutt/1.10.1 (2018-07-13)

On Mon, Nov 12, 2018 at 04:32:46AM -0500, Pierre Neidhardt wrote:
> commit 9a65a052016572b61e3c4247fcdf9e0478656f71
> Author: Pierre Neidhardt <mail@ambrevar.xyz>
> Date:   Sun Nov 11 22:02:18 2018 +0100
> 
>     gnu: go: Update default to 1.11.
>     
>     * gnu/packages/golang.scm (go): Update default to 1.11.

I noticed that since this change, Go programs (that is, command-line
executables) keep references to their inputs:

$ guix gc --references $(./pre-inst-env guix build --no-grafts kurly)
/gnu/store/2b2md66fbzyspsmd5dj6zkj9hilac40r-tzdata-2018e
/gnu/store/4iwksvq53rlzphfp3xvp63ihlw226c0n-go-github-com-aki237-nscjar-0.0.0-0.e2df936
/gnu/store/5rxdjbk8h0bh1hbaan8y8ib13va2bcmw-net-base-5.3
/gnu/store/ahvdlp6y44qj6kx63rmx1sq8r61x3zc2-go-github-com-alsm-ioprogress-0.0.0-0.063c372
/gnu/store/f8yps0l8p371jgzh6cki0z5n2kgfjiwy-go-github-com-urfave-cli-1.19.1-0.934abfb
/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27
/gnu/store/pp0bakrbyv9xmp1kyv2114l19s11b74z-gcc-6.4.0-lib

Previously, they did not:

$ guix gc --references $(guix build --no-grafts kurly)
/gnu/store/2b2md66fbzyspsmd5dj6zkj9hilac40r-tzdata-2018e
/gnu/store/5rxdjbk8h0bh1hbaan8y8ib13va2bcmw-net-base-5.3
/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27
/gnu/store/pp0bakrbyv9xmp1kyv2114l19s11b74z-gcc-6.4.0-lib

Is this expected? I thought that Go programs were always statically
linked.



----- End forwarded message -----

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#33620: [mail@ambrevar.xyz: Re: Golang programs keeping references [gnu: go: Update default to 1.11.]]

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1790 bytes --]

----- Forwarded message from Pierre Neidhardt <mail@ambrevar.xyz> -----

Date: Sun, 09 Dec 2018 19:46:03 +0100
From: Pierre Neidhardt <mail@ambrevar.xyz>
To: Leo Famulari <leo@famulari.name>
Cc: guix-devel@gnu.org
Subject: Re: Golang programs keeping references [gnu: go: Update default to 1.11.]
User-agent: mu4e 1.0; emacs 26.1

I've investigated the possible solutions to avoid including the paths into the
binaries.

I've found this:

https://github.com/golang/go/issues/16860

It's still unresolved and only planned for Go 1.13.

In the meantime, I've played with the -trimpath option to see if I could get
something out of it.

I've added this to Go's (build) function:

--8<---------------cut here---------------start------------->8---
              "-asmflags=all=-trimpath=/gnu/store"
              "-gcflags=all=-trimpath=/gnu/store"
--8<---------------cut here---------------end--------------->8---

The resulting binary is indeed trimmed, but that's not enough: it seems that
Guix detects the remaining part of the path as a store item and includes it in
the list of requisites.  Is this really how Guix works?  It does not need the
full path?

Go supports only one call to -trimpath, so we can't even set this to the Go
inputs.

Regarding Boyer-Moore over the binary, we would have to apply the changes for
all recursive Go libraries.  Now is there a reliable way to detect what's a Go
library and what is not?  We don't want to patch non-Go libraries, right?
(Let's not forget that there is CGo...)

If we can't think of a way to detect a Go library, Boyer-Moore does not seem to
be a solution either.  And we might have to wait for Go 1.13...

-- 
Pierre Neidhardt
https://ambrevar.xyz/



----- End forwarded message -----

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#33620: [mail@ambrevar.xyz: Re: Golang programs keeping references [gnu: go: Update default to 1.11.]]

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 2234 bytes --]

The issue of Go packages keeping references to all their Go-language is
inputs is resolved with commit e3900a4d64e4bf6f426809d6bff058e5a2ae9bc8.

This commit basically avoids bringing the store paths into the build
environment at all by symlinking them into a union directory in the
TMPDIR. This is a bit of a hack but it's actually more idiomatic in Go
to have all the inputs in a single directory ("workspace" in Go). The
previous technique of passing a list of directories in the GOPATH
variable is officially supported but is definitely a 2nd-class technique
in practice.

----- Forwarded message from Pierre Neidhardt <mail@ambrevar.xyz> -----
> I've added this to Go's (build) function:
> 
> --8<---------------cut here---------------start------------->8---
>               "-asmflags=all=-trimpath=/gnu/store"
>               "-gcflags=all=-trimpath=/gnu/store"
> --8<---------------cut here---------------end--------------->8---
> 
> The resulting binary is indeed trimmed, but that's not enough: it seems that
> Guix detects the remaining part of the path as a store item and includes it in
> the list of requisites.  Is this really how Guix works?  It does not need the
> full path?

Not having read the reference scanner code carefully, I expect that it
ignores the '/gnu/store/' since this path is actually configurable.

> Regarding Boyer-Moore over the binary, we would have to apply the changes for
> all recursive Go libraries.  Now is there a reliable way to detect what's a Go
> library and what is not?  We don't want to patch non-Go libraries, right?
> (Let's not forget that there is CGo...)

In (guix build-system go), I'd like to construct of list of inputs that
use the go-build-system and pass this list to the biuld side. This would
be useful for other things, but could also be used to detect which
inputs are Go libraries.

> If we can't think of a way to detect a Go library, Boyer-Moore does not seem to
> be a solution either.  And we might have to wait for Go 1.13...

Waiting for upstream is always the easiest way :)

But it would still be nice to have Boyer-Moore available in (guix build
utils) for whenever we want to use it. Or even in Guile itself...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]