Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Maxime Devos <maximedevos@telenet.be>]

[-- Attachment #1: Type: text/plain, Size: 4623 bytes --]

Hi,

Warning: I haven't actually ever touched a go package.  Take my mail
with a huge grain of salt.

Much of this you've probably already heard at
<https://logs.guix.gnu.org/guix/2021-08-31.log#024401>.

Attila Lendvai schreef op ma 30-08-2021 om 21:52 [+0000]:
> [...]
> so, regarding go-ethereum, i've seen this:
> 
> https://issues.guix.gnu.org/43872
> 
> the initial conclusion was that the proper way to package a go project is to package the pinned transitive
> closure of every dependency. there's a go importer now, which is functional/hackable enough that this is not
> a hopeless task, but... i'm doubtful that it's a good idea to multiply the number of Guix packages by such an endeavor... :)

This situation doesn't seem all that different from, say, importing 'evolution'
(a GNOME e-mail program) in the hypothetical situation that guix doesn't have
any GTK or GNOME library already packaged.  I don't think you have to worry
about adding many guix packages.  Presumably, the new guix packages would
have uses outside go-ethereum, so they can be re-used as dependencies of
new go packages, so over time, having to define many new packages when importing
a go application should become less and less of a problem.

(About version pinning: I'm ignoring version incompatibilities here.  I don't know
how much of a problem that is in practice ...)  

Maybe I'm spouting nonsense here though, (gnu packages golang) has been around
since 2016, and possibly go-ethereum has much more (indirect) dependencies than
'evolution'.

> then Helio Machado proposed something smarter in a later comment:
> 
> https://issues.guix.gnu.org/43872#3
> 
> IIUC, he proposes a way to instead use the go module system to download all the dependencies,
> and yet authenticate all the downloaded go code.

(Parts of what I write below is written in the manual (guix)Submitting patches,
search for ‘6. Make sure the package does not use bundled copies of software
already available as separate packages.’)

One problem with this approach, is that a go package can be using very old versions
(possibly with bugs that have long been fixed in the latest versions, or with security
issues) of dependencies without any indication thereof in the package definition,
and "guix refresh -t go" and "guix lint -c cve" can't indicate problems (*).

(*) I don't know if these commands currently work on go packages.

Another problem is: if a go package has many (transitive) dependencies, how do
we check that it doesn't contain any malware or non-free components?  That needs
to be checked manually, per package.  With the status quo (only have one copy
of everything whenever feasible), this only has to be done once per ‘go software’
(go module? I'm not familiar with the terminology).  But when using something
like ‘https://issues.guix.gnu.org/43872#3’, if multiple ‘guix go packages’
use the same ‘go module(?)’, then both variants of the module need to be checked.

So to conclude, I don't think this approach can scale safely, and this approach
actually seems more work to me.

(Also think of network traffic and build times, which would presumably be much
increased by this approach.  Disk space shouldn't be much of a problem due to
the ‘content(/store?) deduplication’ feature of guix.)

>  his work is not merged yet, and i think it's not even ready for merging yet.
> 
> now, i'm rather motivated to work on this, maybe even willing to use the go importer
> and add countless pinned go packages... but is that desirable? is that the ultimate
> solution/goal?

Using the go importer (in --recursive mode I presume) seems good, but if with "pinned"
you mean "multiple versions of the same go module in Guix", I would avoid that if
possible, due to the reasons I noted above.

If the various dependents of a go package aren't to picky about the exact version,
you could use "guix refresh --type=go" to update the indirect dependencies of
go-ethereum. (Note: guix refresh doesn't seem to support go yet.)

>  or should i wait until Helio's clever hack is merged? or shall i try to finish up his
>  hack to be merge-ready?

I don't think it will ever be merged as it only works for x86_64 ...
<https://issues.guix.gnu.org/43872#2>

> i'd really appreciate some guidance and/or coordination regarding where i should put my energy.

I don't know how Go in Guix will eventually look like, but I'd assume
having an updater for go would be uncontroversial, appreciated and useful
for packaging go-ethereum.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]