Re: Go packaging

[Leo Famulari <leo@famulari.name>]

[-- Attachment #1: Type: text/plain, Size: 3990 bytes --]

On Wed, Oct 04, 2017 at 06:19:18AM +0200, Pjotr Prins wrote:

Thanks for your comments, Pjotr!

> Thanks Leo for the explanation. Now I understand why Go programs, such
> as the IPFS implementation, have so many dependencies...

Yes, so many. As for transitive dependencies... well, I probably won't
package IPFS in my free time because of the huge number of packages
required.

> What I understand now is that packages get built 'lazily' and there is
> really no way to force a build - other than running the target
> software.

Not exactly — you can build a Go library module with `go install
name-of-module`. My packaging of Syncthing's dependencies does this,
creating '.a' static library archives, which are used later when
building Syncthing itself.

https://github.com/lfam/guix/blob/contrib-syncthing/gnu/packages/syncthing.scm

The difficulty for Guix is that we can't build the entire library (all
its modules) with a single command. As far as I can tell, we'd have to
recurse the filesystem tree of the library source code and try building
each directory. It sounds unreliable to me.

> I noticed the hash values in dependencies, so these are git checkouts
> (i.e., they are versioned, but not in the standard sense).

Right, there are useful "version" references in the form of the Git
commit hashes, but the problem is that it seems each Go application uses
a different commit.

> When I read the package.json file it includes things like:
> 
>   "gxDependencies": [
>     {
>       "hash": "QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52",
>       "name": "go-log",
>       "version": "1.2.0"
>     },
>     {
>       "author": "whyrusleeping",
>       "hash": "QmZfwmhbcgSDGqGaoMMYx8jxBGauZw75zPjnZAyfwPso7M",
>       "name": "go-libp2p-secio",
>       "version": "1.1.8"
>     },
>     {
>       "author": "whyrusleeping",
>       "hash": "QmaPbCnUMBohSGo3KnxEa2bHqyJVVeEEcwtqJAYxerieBo",
>       "name": "go-libp2p-crypto",
>       "version": "1.5.0"
>     },
> pac
> 
> indeed, there are even two versions in there for 'whyrusleeping'
> deep dependencies ;). (would that be a reference to Ruby's Why?).

In this case, I don't know what whyrusleeping refers to.
go-libp2p-secio and go-libp2p-crypto are the names of the programs
themselves.

> On the surface, similar to Rubygems, I think it is no problem to
> distribute source packages from Guix and have them compile on the fly.
> In a way that is also an interesting model for late optimizations -
> something we are lacking in our current infrastructure. What I think
> we should do is import above json file and generate GNU binary
> packages that are GO source bundles.

That package.json file is not a standard thing in the Go world.
I've found that Go applications use a variety of dependency manifest
formats, or just use Git submodules.

> Provided GO can use a live build directory outside the store it will
> only compile bundles once, on demand. Ruby 'compiles' or interprets
> every time, so that is one up on Go ;). That target directory would be
> mutable, so that is a downside, a potential security risk.

I considered something like this earlier in this effort, but it's not
what I'm proposing now. As you say, it's messy to compile outside of the
store at run-time, and I don't think we need to do it.

Rather, I think we should have a special go-package procedure, used in
the inputs field of the calling application, which would build the
relevant library modules of the correct Git commit. Does that make
sense?

> We ought to have a look at how Nix packaged Go builds because they are
> already have a solution. Be interesting to see if they found a way to
> compile packages 'greedily', the way Python does it.

I looked at their build system itself a few weeks ago when I was still
learning how Go compilation works. I agree, it would be fruitful to see
how they handle the issues I've raised here.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]