Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Attila Lendvai <attila@lendvai.name>]

On Wednesday, September 1st, 2021 at 00:21, Maxime Devos <maximedevos@telenet.be> wrote:

> Hi,
>
> Warning: I haven't actually ever touched a go package. Take my mail
> with a huge grain of salt.
> Much of this you've probably already heard at
> https://logs.guix.gnu.org/guix/2021-08-31.log#024401.
>
> Attila Lendvai schreef op ma 30-08-2021 om 21:52 [+0000]:
>
> > [...]
> >
> > so, regarding go-ethereum, i've seen this:
> > https://issues.guix.gnu.org/43872
> > the initial conclusion was that the proper way to package a go project is to package the pinned transitive
> > closure of every dependency. there's a go importer now, which is functional/hackable enough that this is not
> > a hopeless task, but... i'm doubtful that it's a good idea to multiply the number of Guix packages by such an endeavor... :)
>
> This situation doesn't seem all that different from, say, importing 'evolution'
> (a GNOME e-mail program) in the hypothetical situation that guix doesn't have
> any GTK or GNOME library already packaged. I don't think you have to worry
> about adding many guix packages. Presumably, the new guix packages would


would you have the same opinion if some guix go packages would have
10+ versions in parallel? (see later)


> have uses outside go-ethereum, so they can be re-used as dependencies of
> new go packages, so over time, having to define many new packages when importing
> a go application should become less and less of a problem.
>
> (About version pinning: I'm ignoring version incompatibilities here. I don't know
> how much of a problem that is in practice ...)


i'm not a go expert at all, but i think it's a rather frequent
situation in go land to move a dependency one git commit ahead, only
to pick up a bugfix that has just been pushed into its repo.


> Maybe I'm spouting nonsense here though, (gnu packages golang) has been around
> since 2016, and possibly go-ethereum has much more (indirect) dependencies than
> 'evolution'.


go-ethereum has 70 dependencies listed in its go.mod file, but it has
657 entries in the go.sum, which contains the hash of the transitive
closure of dependencies.

this mechanism is part of how go builds are reproducible.

building go-ethereum locally should yield the same binary that they
have released.


> > then Helio Machado proposed something smarter in a later comment:
> > https://issues.guix.gnu.org/43872#3
> >
> > IIUC, he proposes a way to instead use the go module system to download all the dependencies,
> > and yet authenticate all the downloaded go code.
>
> (Parts of what I write below is written in the manual (guix)Submitting patches,
> search for ‘6. Make sure the package does not use bundled copies of software
> already available as separate packages.’)
>
> One problem with this approach, is that a go package can be using very old versions
> (possibly with bugs that have long been fixed in the latest versions, or with security
> issues) of dependencies without any indication thereof in the package definition,
> and "guix refresh -t go" and "guix lint -c cve" can't indicate problems ().
>
> () I don't know if these commands currently work on go packages.
>
> Another problem is: if a go package has many (transitive) dependencies, how do
> we check that it doesn't contain any malware or non-free components? That needs


go-ethereum is a software that handles ethereum wallets with several
zeroes. they are probably equally worried about this if not more.


> to be checked manually, per package. With the status quo (only have one copy
> of everything whenever feasible), this only has to be done once per ‘go software’
> (go module? I'm not familiar with the terminology). But when using something
> like ‘https://issues.guix.gnu.org/43872#3’, if multiple ‘guix go packages’
> use the same ‘go module(?)’, then both variants of the module need to be checked.
>
> So to conclude, I don't think this approach can scale safely, and this approach
> actually seems more work to me.
>
> (Also think of network traffic and build times, which would presumably be much
> increased by this approach. Disk space shouldn't be much of a problem due to
> the ‘content(/store?) deduplication’ feature of guix.)
>
> > his work is not merged yet, and i think it's not even ready for merging yet.
> > now, i'm rather motivated to work on this, maybe even willing to use the go importer
> > and add countless pinned go packages... but is that desirable? is that the ultimate
> > solution/goal?
>
> Using the go importer (in --recursive mode I presume) seems good, but if with "pinned"
> you mean "multiple versions of the same go module in Guix", I would avoid that if
> possible, due to the reasons I noted above.
> If the various dependents of a go package aren't to picky about the exact version,
> you could use "guix refresh --type=go" to update the indirect dependencies of
> go-ethereum. (Note: guix refresh doesn't seem to support go yet.)


for projects like go-ethereum, it's not an option for the packager to
make decisions about the version of any of its dependencies. in fact,
i think the go.mod file contains the transitive closure of
dependencies exactly for this reason: to pin down the exact version of
every dependency.

with that in sight, i'm not sure anymore that it makes sense to
isolate go-ethereum and all its dependencies into a separate scheme
module and file. maybe every go library should be added to golang.scm
and toplevel applications like go-ethereum into their own file, or
into finance.scm (where bitcoin lives now).

to sum it up: with my limited experience, i only see two viable
options: 1) allow go to fetch, authenticate, and build everything, or
2) use `guix import go` to mirror what is contained in the go.mod and
go.sum files on the guix side (but to me it feels to be an enormous
effort compared to what we gain with it).

a variation of 1) could be to do "vendoring" (ala NixOS): build an
authenticated cache of every dependency that then gets fetched from a
substitute server as a single archive whenever someone wants to build
any go project that has the same mod.go and mod.sums files.

does this make sense?

- attila