Go packaging

Go packaging

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 2396 bytes --]

Based on my work creating a go-build-system and packaging a non-trivial
Go application [0], I want to start a discussion on how we can
efficiently package Go software in Guix.

Go software is developed rather differently from most of what we
package, and I think our package abstraction does not fit Go libraries
well.

The primary differences are 0) Go libraries are typically bundled as
source code, 1) Go software is typically unversioned, 2) static archives
(.a) are the norm, and 3) Go libraries are expected to be built as part
of the build process of the calling application. That is, there is no
standard way to build an entire Go library on its own.

Elaboration:

0,1) The problem with every application bundling unversioned libraries
is that we need to package a different Git commit of the library for
every application we package, or risk our packages not working. Adding
package variants per-version is a bit messy currently.

3) As an example of how Go libaries are built piecemeal, take the core
networking library, 'golang.org/x/net' [1]. It includes dozens of
submodules such as bpf, icmp, ipv4, ipv6, etc. There is no way to build
all these submodules with a single command. Instead, each one is built
when it is needed during the build process of the calling application.
There are no build scripts. The library compilation process is
standardized as, for example, `go install golang.org/x/net/ipv4`.

This means that the entire networking library would consist of several
dozen Guix packages, multiplied by the number of different Git commits
required by the calling applications. It's unreasonable, in my opinion.

My suggestion is that we have two layers of Go library packages: a) a
template layer that includes the source URI, unpack-path, and other
package metadata, and b) a layer to create instances of the package
within the inputs field of the calling package.

Perhaps the instantiation layer could look like this in practice:

(define-public my-go-program
  [...]
  (inputs
   `(("golang.org/x/net"
      ,(go-package golang-org-x-net
        (version "ffcf1bedda")
	(import-paths '("golang.org/x/net/ipv4"
	                "golang.org/x/net/context"
			"golang.org/x/net/ipv6"
			"golang.org/x/net/internal/iana")))))))

Or something like that. What do you think?

[0]
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28586#26
[1]
https://godoc.org/golang.org/x/net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Go packaging

[Pjotr Prins]

Thanks Leo for the explanation. Now I understand why Go programs, such
as the IPFS implementation, have so many dependencies... What I
understand now is that packages get built 'lazily' and there is really
no way to force a build - other than running the target software. I
noticed the hash values in dependencies, so these are git checkouts
(i.e., they are versioned, but not in the standard sense).

It is an interesting approach - because it does guarantee people are
using the same (deep) software stack. I noticed that too:

On Tue, Oct 03, 2017 at 11:15:04AM -0400, Leo Famulari wrote:
> Based on my work creating a go-build-system and packaging a non-trivial
> Go application [0], I want to start a discussion on how we can
> efficiently package Go software in Guix.
> 
> Go software is developed rather differently from most of what we
> package, and I think our package abstraction does not fit Go libraries
> well.
> 
> The primary differences are 0) Go libraries are typically bundled as
> source code, 1) Go software is typically unversioned, 2) static archives
> (.a) are the norm, and 3) Go libraries are expected to be built as part
> of the build process of the calling application. That is, there is no
> standard way to build an entire Go library on its own.
> 
> Elaboration:
> 
> 0,1) The problem with every application bundling unversioned libraries
> is that we need to package a different Git commit of the library for
> every application we package, or risk our packages not working. Adding
> package variants per-version is a bit messy currently.
> 
> 3) As an example of how Go libaries are built piecemeal, take the core
> networking library, 'golang.org/x/net' [1]. It includes dozens of
> submodules such as bpf, icmp, ipv4, ipv6, etc. There is no way to build
> all these submodules with a single command. Instead, each one is built
> when it is needed during the build process of the calling application.
> There are no build scripts. The library compilation process is
> standardized as, for example, `go install golang.org/x/net/ipv4`.
> 
> This means that the entire networking library would consist of several
> dozen Guix packages, multiplied by the number of different Git commits
> required by the calling applications. It's unreasonable, in my opinion.
> 
> My suggestion is that we have two layers of Go library packages: a) a
> template layer that includes the source URI, unpack-path, and other
> package metadata, and b) a layer to create instances of the package
> within the inputs field of the calling package.
> 
> Perhaps the instantiation layer could look like this in practice:
> 
> (define-public my-go-program
>   [...]
>   (inputs
>    `(("golang.org/x/net"
>       ,(go-package golang-org-x-net
>         (version "ffcf1bedda")
> 	(import-paths '("golang.org/x/net/ipv4"
> 	                "golang.org/x/net/context"
> 			"golang.org/x/net/ipv6"
> 			"golang.org/x/net/internal/iana")))))))

When I read the package.json file it includes things like:

  "gxDependencies": [
    {
      "hash": "QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52",
      "name": "go-log",
      "version": "1.2.0"
    },
    {
      "author": "whyrusleeping",
      "hash": "QmZfwmhbcgSDGqGaoMMYx8jxBGauZw75zPjnZAyfwPso7M",
      "name": "go-libp2p-secio",
      "version": "1.1.8"
    },
    {
      "author": "whyrusleeping",
      "hash": "QmaPbCnUMBohSGo3KnxEa2bHqyJVVeEEcwtqJAYxerieBo",
      "name": "go-libp2p-crypto",
      "version": "1.5.0"
    },
pac

indeed, there are even two versions in there for 'whyrusleeping'
deep dependencies ;). (would that be a reference to Ruby's Why?).

On the surface, similar to Rubygems, I think it is no problem to
distribute source packages from Guix and have them compile on the fly.
In a way that is also an interesting model for late optimizations -
something we are lacking in our current infrastructure. What I think
we should do is import above json file and generate GNU binary
packages that are GO source bundles. Provided GO can use a live build
directory outside the store it will only compile bundles once, on
demand. Ruby 'compiles' or interprets every time, so that is one up on
Go ;). That target directory would be mutable, so that is a downside,
a potential security risk.

What you are saying is that, inside the build system, we pull in all
packages as sources and do a complete compile - which flies in the
face of how dependencies are built independently today. One question
is, can you force that full compilation? 

We ought to have a look at how Nix packaged Go builds because they are
already have a solution. Be interesting to see if they found a way to
compile packages 'greedily', the way Python does it.

Pj.

Re: Go packaging

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 3990 bytes --]

On Wed, Oct 04, 2017 at 06:19:18AM +0200, Pjotr Prins wrote:

Thanks for your comments, Pjotr!

> Thanks Leo for the explanation. Now I understand why Go programs, such
> as the IPFS implementation, have so many dependencies...

Yes, so many. As for transitive dependencies... well, I probably won't
package IPFS in my free time because of the huge number of packages
required.

> What I understand now is that packages get built 'lazily' and there is
> really no way to force a build - other than running the target
> software.

Not exactly — you can build a Go library module with `go install
name-of-module`. My packaging of Syncthing's dependencies does this,
creating '.a' static library archives, which are used later when
building Syncthing itself.

https://github.com/lfam/guix/blob/contrib-syncthing/gnu/packages/syncthing.scm

The difficulty for Guix is that we can't build the entire library (all
its modules) with a single command. As far as I can tell, we'd have to
recurse the filesystem tree of the library source code and try building
each directory. It sounds unreliable to me.

> I noticed the hash values in dependencies, so these are git checkouts
> (i.e., they are versioned, but not in the standard sense).

Right, there are useful "version" references in the form of the Git
commit hashes, but the problem is that it seems each Go application uses
a different commit.

> When I read the package.json file it includes things like:
> 
>   "gxDependencies": [
>     {
>       "hash": "QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52",
>       "name": "go-log",
>       "version": "1.2.0"
>     },
>     {
>       "author": "whyrusleeping",
>       "hash": "QmZfwmhbcgSDGqGaoMMYx8jxBGauZw75zPjnZAyfwPso7M",
>       "name": "go-libp2p-secio",
>       "version": "1.1.8"
>     },
>     {
>       "author": "whyrusleeping",
>       "hash": "QmaPbCnUMBohSGo3KnxEa2bHqyJVVeEEcwtqJAYxerieBo",
>       "name": "go-libp2p-crypto",
>       "version": "1.5.0"
>     },
> pac
> 
> indeed, there are even two versions in there for 'whyrusleeping'
> deep dependencies ;). (would that be a reference to Ruby's Why?).

In this case, I don't know what whyrusleeping refers to.
go-libp2p-secio and go-libp2p-crypto are the names of the programs
themselves.

> On the surface, similar to Rubygems, I think it is no problem to
> distribute source packages from Guix and have them compile on the fly.
> In a way that is also an interesting model for late optimizations -
> something we are lacking in our current infrastructure. What I think
> we should do is import above json file and generate GNU binary
> packages that are GO source bundles.

That package.json file is not a standard thing in the Go world.
I've found that Go applications use a variety of dependency manifest
formats, or just use Git submodules.

> Provided GO can use a live build directory outside the store it will
> only compile bundles once, on demand. Ruby 'compiles' or interprets
> every time, so that is one up on Go ;). That target directory would be
> mutable, so that is a downside, a potential security risk.

I considered something like this earlier in this effort, but it's not
what I'm proposing now. As you say, it's messy to compile outside of the
store at run-time, and I don't think we need to do it.

Rather, I think we should have a special go-package procedure, used in
the inputs field of the calling application, which would build the
relevant library modules of the correct Git commit. Does that make
sense?

> We ought to have a look at how Nix packaged Go builds because they are
> already have a solution. Be interesting to see if they found a way to
> compile packages 'greedily', the way Python does it.

I looked at their build system itself a few weeks ago when I was still
learning how Go compilation works. I agree, it would be fruitful to see
how they handle the issues I've raised here.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Go packaging

[Pjotr Prins]

On Wed, Oct 04, 2017 at 10:22:25AM -0400, Leo Famulari wrote:
> That package.json file is not a standard thing in the Go world.
> I've found that Go applications use a variety of dependency manifest
> formats, or just use Git submodules.

Guix is a good thing then :). Also it means that they don't really
enforce a dependency graph. How can you enforce something if you have
many implementations? The enforcement is only at the single package
level.

> Rather, I think we should have a special go-package procedure, used in
> the inputs field of the calling application, which would build the
> relevant library modules of the correct Git commit. Does that make
> sense?

Yes. Since you can do a 'go build' I think it is possible to do this
in a traditional way. It sucks that GO has so many small dependencies
(similar to the node mess). But maybe we can import them somehow. Does
the build tool show the graph?

It is interesting that different packages have different git checkout
dependencies (so different hash values for the same package go-ssl or
whatever). For developers this is great because users end up with the
exact same dependency graph. But for Guix I think we can ignore this.

It is what we are doing today. It is easy to create a deployment
environment in Guix that was never tried before. Therefore, we also
don't really care. We just provide the latest and see if that works.
So, I suggest to import just one version of go-ssl and cross fingers
it works. If it doesn't - well then it gets a bit harder and we'll
have to deal with multiple versions.

> > We ought to have a look at how Nix packaged Go builds because they are
> > already have a solution. Be interesting to see if they found a way to
> > compile packages 'greedily', the way Python does it.
> 
> I looked at their build system itself a few weeks ago when I was still
> learning how Go compilation works. I agree, it would be fruitful to see
> how they handle the issues I've raised here.

Any update?

Pj.


-- 

Re: Go packaging

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On Tue, Oct 03, 2017 at 11:15:04AM -0400, Leo Famulari wrote:
> Based on my work creating a go-build-system and packaging a non-trivial
> Go application [0], I want to start a discussion on how we can
> efficiently package Go software in Guix.

Another question, which is bikesheddy, is how to name Go packages in
Guix.

So far, I've used "golang-$upstreamname", which is how Petter named the
work-in-progress packages I am finishing. This leads to very long
package names, but I don't know a better way to ensure unique names for
each package.

I think we should do "go-$upstreamname" instead. Golang is not the name
of the language, but rather the domain name (go.org was apparently not
available), and a term that has been adopted by the community. But, it
would be good to save 4 characters here.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Go packaging

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> I think we should do "go-$upstreamname" instead. Golang is not the name
> of the language, but rather the domain name (go.org was apparently not
> available), and a term that has been adopted by the community. But, it
> would be good to save 4 characters here.

I’m all for “go-” instead of “golang-”.

Ludo’, happy to help paint the shed.  :-)

Re: Go packaging

[ng0]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

Ludovic Courtès transcribed 0.4K bytes:
> Leo Famulari <leo@famulari.name> skribis:
> 
> > I think we should do "go-$upstreamname" instead. Golang is not the name
> > of the language, but rather the domain name (go.org was apparently not
> > available), and a term that has been adopted by the community. But, it
> > would be good to save 4 characters here.
> 
> I’m all for “go-” instead of “golang-”.
> 
> Ludo’, happy to help paint the shed.  :-)

Yes, go for it.
My 2go's (and good to see some progress in Go on Guix).
-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://dist.ng0.infotropique.org/dist/keys/
https://www.infotropique.org https://ng0.infotropique.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]