Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Maxime Devos <maximedevos@telenet.be>]

[-- Attachment #1: Type: text/plain, Size: 8217 bytes --]

Attila Lendvai schreef op wo 01-09-2021 om 14:29 [+0000]:
> On Wednesday, September 1st, 2021 at 00:21, Maxime Devos <maximedevos@telenet.be> wrote:
> 
> > Hi,
> > 
> > Warning: I haven't actually ever touched a go package. Take my mail
> > with a huge grain of salt.
> > Much of this you've probably already heard at
> > https://logs.guix.gnu.org/guix/2021-08-31.log#024401.
> > 
> > Attila Lendvai schreef op ma 30-08-2021 om 21:52 [+0000]:
> > 
> > > [...]
> > > 
> > > so, regarding go-ethereum, i've seen this:
> > > https://issues.guix.gnu.org/43872
> > > the initial conclusion was that the proper way to package a go project is to package the pinned transitive
> > > closure of every dependency. there's a go importer now, which is functional/hackable enough that this is not
> > > a hopeless task, but... i'm doubtful that it's a good idea to multiply the number of Guix packages by such an endeavor... :)
> > 
> > This situation doesn't seem all that different from, say, importing 'evolution'
> > (a GNOME e-mail program) in the hypothetical situation that guix doesn't have
> > any GTK or GNOME library already packaged. I don't think you have to worry
> > about adding many guix packages. Presumably, the new guix packages would
> 
> would you have the same opinion if some guix go packages would have
> 10+ versions in parallel? (see later)

Were is the ‘later’?  I haven't seen anything about multiple versions in parallel
below ...

> 
> > have uses outside go-ethereum, so they can be re-used as dependencies of
> > new go packages, so over time, having to define many new packages when importing
> > a go application should become less and less of a problem.
> > 
> > (About version pinning: I'm ignoring version incompatibilities here. I don't know
> > how much of a problem that is in practice ...)
> 
> i'm not a go expert at all, but i think it's a rather frequent
> situation in go land to move a dependency one git commit ahead, only
> to pick up a bugfix that has just been pushed into its repo.

... unless you meant this here?  But this seems to imply that version
incompatibilities typically don't happen (otherwise just picking up
a bugfix with a single commit doesn't seem plausible), so we can
typically just use the latest version of the go module in guix, no?

> > Maybe I'm spouting nonsense here though, (gnu packages golang) has been around
> > since 2016, and possibly go-ethereum has much more (indirect) dependencies than
> > 'evolution'.
> 
> go-ethereum has 70 dependencies listed in its go.mod file, but it has
> 657 entries in the go.sum, which contains the hash of the transitive
> closure of dependencies.

I'm not sure if you mean that go-ethereum has very many or little dependencies
here.  To compare, openttd has about 494 indirect dependencies (looking at
guix graph --type=package openttd).  Many of these are python packages (82),
texlive packages (120), the X libraries (about 28 or so), the GTK+ stack,
basic stuff like autoconf and automake ...  Most of them packages that wouldn't
seem unique to openttd.

> this mechanism is part of how go builds are reproducible.

> building go-ethereum locally should yield the same binary that they
> have released.

FWIW, ‘reproducibility’ in guix means that, if the build is repeated on another
machine, with the same architecture and guix version, the binaries will be
exactly the same.  Bit-for-bit reproducibility isn't guaranteed when the builds
are performed on different distro's.

More practically speaking, if the guix package definition makes even a tiny change,
the binaries will be different.  E.g., the guix package definition of go replaces
"/etc/protocols" with "/gnu/store/[HASH-...]/etc/protocols, which changes the resulting
hash.

As a demonstration, consider the 'syncthing' package.

$ guix build syncthing
[... downloading ...]
/gnu/store/9fmbnnjjhp2y578lgrg9s4ywwr4kn3xg-syncthing-1.16.1
/gnu/store/yg79kc20y8smjglxbd75w100ypbszj3c-syncthing-1.16.1-utils

$ guix gc --references /gnu/store/9fmbnnjjhp2y578lgrg9s4ywwr4kn3xg-syncthing-1.16.1
/gnu/store/kl68v5mclwp511xgpsl2h1s9gmsdxpzh-tzdata-2021a
/gnu/store/zfbbn61ij7w0bl4wbrwi87x5ghqx968c-net-base-5.3

When tzdata or net-base is updated, their hash will changes so the binaries of syncthing
will change.

Another problem: looking at the definition of go@1.14, it appears go assumes
the loader to be at /lib/lib(64)/ld-linux.[...].so.[...], but that doesn't even
exist on Guix, so binaries created on another distro cannot be run on guix,
and not the other way around (unless they are statically linked perhaps?
But static linking prevents grafting of things like glibc, which sometimes
needs security fixes ...  Why are go programs statically linked in Guix anyways?)

Outside some very specific cases (e.g.
https://reproducible-builds.org/news/2019/12/21/reproducible-bootstrap-of-mes-c-compiler/),
‘cross-distro bit-for-bit reproducibility’ doesn't seem very plausible.

Also ...

> building go-ethereum locally should yield the same binary that they
> have released.

... why should it?

> > [...]
> > Another problem is: if a go package has many (transitive) dependencies, how do
> > we check that it doesn't contain any malware or non-free components? That needs
> 
> go-ethereum is a software that handles ethereum wallets with several
> zeroes. they are probably equally worried about this if not more.

I'm sure the go-ethereum peope wouldn't mind if we double-check that no malware
slipt through their review process.

> > Using the go importer (in --recursive mode I presume) seems good, but if with "pinned"
> > you mean "multiple versions of the same go module in Guix", I would avoid that if
> > possible, due to the reasons I noted above.
> > If the various dependents of a go package aren't to picky about the exact version,
> > you could use "guix refresh --type=go" to update the indirect dependencies of
> > go-ethereum. (Note: guix refresh doesn't seem to support go yet.)
> 
> for projects like go-ethereum, it's not an option for the packager to
> make decisions about the version of any of its dependencies.

Why isn't this an option?  Choosing different versions from upstream is already
done in guix, see e.g. https://issues.guix.gnu.org/50217 (ok that's not merged
yet, not the best example --- note, editing the version requirement was
on my advice, see https://logs.guix.gnu.org/guix/2021-08-26.log).

>  in fact,
> i think the go.mod file contains the transitive closure of
> dependencies exactly for this reason: to pin down the exact version of
> every dependency.

That's my understanding as well.  However, we don't have to follow
the choices made upstream.

> with that in sight, i'm not sure anymore that it makes sense to
> isolate go-ethereum and all its dependencies into a separate scheme
> module and file. maybe every go library should be added to golang.scm
> and toplevel applications like go-ethereum into their own file, or
> into finance.scm (where bitcoin lives now).
> 
> to sum it up: with my limited experience, i only see two viable
> options: 1) allow go to fetch, authenticate, and build everything, or
> 2) use `guix import go` to mirror what is contained in the go.mod and
> go.sum files on the guix side (but to me it feels to be an enormous
> effort compared to what we gain with it).

I see a third viable option (3): treat the "go.sum" as a mere
‘friendly suggestion’, and just use the latest version when feasible.
Again, I don't see much difference with, say, haskell, python, ruby,
guile, java ... packages.

Have there been any problems in practice with just using the latest
version (updating the version currently in guix where applicable)?

I hope my reply has been valueable.  It seems like this discussion
is going to end with ‘agreeing to disagree’ and I think I said all
I've got to say about the philosophical part of the matter, so I
don't think I'll sending further replies on the philosophical parts.
But if you have any technical questions, feel free to ask me.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]