packaging go-ethereum, and ultimately bee (of ethswarm.org)

packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Attila Lendvai]

[-- Attachment #1: Type: text/plain, Size: 1965 bytes --]

hello everyone,

it's my first mail here, so let me briefly into myself: i'm a long-time lisper (mostly CL, i'm new to Scheme, although i worked on a scheme-like tiny lisp: https://github.com/attila-lendvai/maru). i'm coming over from NixOS after a few months of loving it as a user, but hating it as a developer. i have packaged bee for NixOS (a client of https://www.ethswarm.org/ written in go), but i got tired of the random, undebuggable obstacles of Nix, especially the implementation of the NixOS module system.

bee depends on go-ethereum (it uses its clef binary to manage the ethereum keys).

so, regarding go-ethereum, i've seen this:

[https://issues.guix.gnu.org/43872](https://issues.guix.gnu.org/43872#3)

the initial conclusion was that the proper way to package a go project is to package the pinned transitive closure of every dependency. there's a go importer now, which is functional/hackable enough that this is not a hopeless task, but... i'm doubtful that it's a good idea to multiply the number of Guix packages by such an endeavor... :)

then Helio Machado proposed something smarter in a later comment:

https://issues.guix.gnu.org/43872#3

IIUC, he proposes a way to instead use the go module system to download all the dependencies, and yet authenticate all the downloaded go code. his work is not merged yet, and i think it's not even ready for merging yet.

now, i'm rather motivated to work on this, maybe even willing to use the go importer and add countless pinned go packages... but is that desirable? is that the ultimate solution/goal? or should i wait until Helio's clever hack is merged? or shall i try to finish up his hack to be merge-ready?

i'd really appreciate some guidance and/or coordination regarding where i should put my energy.

- attila
PGP: 5D5F 45C7 DFCD 0A39

PS: even though Guix has more rough edges from a user perspective, it feels much more natural with my developer hat on. thank you for that experience!

[-- Attachment #2: Type: text/html, Size: 2608 bytes --]

Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Attila Lendvai]

[-- Attachment #1: Type: text/plain, Size: 513 bytes --]

FTR, we had a discussion about this on IRC:

https://logs.guix.gnu.org/guix/2021-08-31.log#024401

conclusion:

ideally, Guix should have a pinned package for each go dependency, and with the go importer it's not hopeless to do anymore.

it conflicts with the general Guix policy to package the latest releases, so i'm putting all the pinned packages generated by `guix go import` into a separate go-ethereum.scm, and into a separate module.

any correction/guidance is welcome!

- attila
PGP: 5D5F 45C7 DFCD 0A39

[-- Attachment #2: Type: text/html, Size: 894 bytes --]

Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 4623 bytes --]

Hi,

Warning: I haven't actually ever touched a go package.  Take my mail
with a huge grain of salt.

Much of this you've probably already heard at
<https://logs.guix.gnu.org/guix/2021-08-31.log#024401>.

Attila Lendvai schreef op ma 30-08-2021 om 21:52 [+0000]:
> [...]
> so, regarding go-ethereum, i've seen this:
> 
> https://issues.guix.gnu.org/43872
> 
> the initial conclusion was that the proper way to package a go project is to package the pinned transitive
> closure of every dependency. there's a go importer now, which is functional/hackable enough that this is not
> a hopeless task, but... i'm doubtful that it's a good idea to multiply the number of Guix packages by such an endeavor... :)

This situation doesn't seem all that different from, say, importing 'evolution'
(a GNOME e-mail program) in the hypothetical situation that guix doesn't have
any GTK or GNOME library already packaged.  I don't think you have to worry
about adding many guix packages.  Presumably, the new guix packages would
have uses outside go-ethereum, so they can be re-used as dependencies of
new go packages, so over time, having to define many new packages when importing
a go application should become less and less of a problem.

(About version pinning: I'm ignoring version incompatibilities here.  I don't know
how much of a problem that is in practice ...)  

Maybe I'm spouting nonsense here though, (gnu packages golang) has been around
since 2016, and possibly go-ethereum has much more (indirect) dependencies than
'evolution'.

> then Helio Machado proposed something smarter in a later comment:
> 
> https://issues.guix.gnu.org/43872#3
> 
> IIUC, he proposes a way to instead use the go module system to download all the dependencies,
> and yet authenticate all the downloaded go code.

(Parts of what I write below is written in the manual (guix)Submitting patches,
search for ‘6. Make sure the package does not use bundled copies of software
already available as separate packages.’)

One problem with this approach, is that a go package can be using very old versions
(possibly with bugs that have long been fixed in the latest versions, or with security
issues) of dependencies without any indication thereof in the package definition,
and "guix refresh -t go" and "guix lint -c cve" can't indicate problems (*).

(*) I don't know if these commands currently work on go packages.

Another problem is: if a go package has many (transitive) dependencies, how do
we check that it doesn't contain any malware or non-free components?  That needs
to be checked manually, per package.  With the status quo (only have one copy
of everything whenever feasible), this only has to be done once per ‘go software’
(go module? I'm not familiar with the terminology).  But when using something
like ‘https://issues.guix.gnu.org/43872#3’, if multiple ‘guix go packages’
use the same ‘go module(?)’, then both variants of the module need to be checked.

So to conclude, I don't think this approach can scale safely, and this approach
actually seems more work to me.

(Also think of network traffic and build times, which would presumably be much
increased by this approach.  Disk space shouldn't be much of a problem due to
the ‘content(/store?) deduplication’ feature of guix.)

>  his work is not merged yet, and i think it's not even ready for merging yet.
> 
> now, i'm rather motivated to work on this, maybe even willing to use the go importer
> and add countless pinned go packages... but is that desirable? is that the ultimate
> solution/goal?

Using the go importer (in --recursive mode I presume) seems good, but if with "pinned"
you mean "multiple versions of the same go module in Guix", I would avoid that if
possible, due to the reasons I noted above.

If the various dependents of a go package aren't to picky about the exact version,
you could use "guix refresh --type=go" to update the indirect dependencies of
go-ethereum. (Note: guix refresh doesn't seem to support go yet.)

>  or should i wait until Helio's clever hack is merged? or shall i try to finish up his
>  hack to be merge-ready?

I don't think it will ever be merged as it only works for x86_64 ...
<https://issues.guix.gnu.org/43872#2>

> i'd really appreciate some guidance and/or coordination regarding where i should put my energy.

I don't know how Go in Guix will eventually look like, but I'd assume
having an updater for go would be uncontroversial, appreciated and useful
for packaging go-ethereum.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Attila Lendvai]

On Wednesday, September 1st, 2021 at 00:21, Maxime Devos <maximedevos@telenet.be> wrote:

> Hi,
>
> Warning: I haven't actually ever touched a go package. Take my mail
> with a huge grain of salt.
> Much of this you've probably already heard at
> https://logs.guix.gnu.org/guix/2021-08-31.log#024401.
>
> Attila Lendvai schreef op ma 30-08-2021 om 21:52 [+0000]:
>
> > [...]
> >
> > so, regarding go-ethereum, i've seen this:
> > https://issues.guix.gnu.org/43872
> > the initial conclusion was that the proper way to package a go project is to package the pinned transitive
> > closure of every dependency. there's a go importer now, which is functional/hackable enough that this is not
> > a hopeless task, but... i'm doubtful that it's a good idea to multiply the number of Guix packages by such an endeavor... :)
>
> This situation doesn't seem all that different from, say, importing 'evolution'
> (a GNOME e-mail program) in the hypothetical situation that guix doesn't have
> any GTK or GNOME library already packaged. I don't think you have to worry
> about adding many guix packages. Presumably, the new guix packages would


would you have the same opinion if some guix go packages would have
10+ versions in parallel? (see later)


> have uses outside go-ethereum, so they can be re-used as dependencies of
> new go packages, so over time, having to define many new packages when importing
> a go application should become less and less of a problem.
>
> (About version pinning: I'm ignoring version incompatibilities here. I don't know
> how much of a problem that is in practice ...)


i'm not a go expert at all, but i think it's a rather frequent
situation in go land to move a dependency one git commit ahead, only
to pick up a bugfix that has just been pushed into its repo.


> Maybe I'm spouting nonsense here though, (gnu packages golang) has been around
> since 2016, and possibly go-ethereum has much more (indirect) dependencies than
> 'evolution'.


go-ethereum has 70 dependencies listed in its go.mod file, but it has
657 entries in the go.sum, which contains the hash of the transitive
closure of dependencies.

this mechanism is part of how go builds are reproducible.

building go-ethereum locally should yield the same binary that they
have released.


> > then Helio Machado proposed something smarter in a later comment:
> > https://issues.guix.gnu.org/43872#3
> >
> > IIUC, he proposes a way to instead use the go module system to download all the dependencies,
> > and yet authenticate all the downloaded go code.
>
> (Parts of what I write below is written in the manual (guix)Submitting patches,
> search for ‘6. Make sure the package does not use bundled copies of software
> already available as separate packages.’)
>
> One problem with this approach, is that a go package can be using very old versions
> (possibly with bugs that have long been fixed in the latest versions, or with security
> issues) of dependencies without any indication thereof in the package definition,
> and "guix refresh -t go" and "guix lint -c cve" can't indicate problems ().
>
> () I don't know if these commands currently work on go packages.
>
> Another problem is: if a go package has many (transitive) dependencies, how do
> we check that it doesn't contain any malware or non-free components? That needs


go-ethereum is a software that handles ethereum wallets with several
zeroes. they are probably equally worried about this if not more.


> to be checked manually, per package. With the status quo (only have one copy
> of everything whenever feasible), this only has to be done once per ‘go software’
> (go module? I'm not familiar with the terminology). But when using something
> like ‘https://issues.guix.gnu.org/43872#3’, if multiple ‘guix go packages’
> use the same ‘go module(?)’, then both variants of the module need to be checked.
>
> So to conclude, I don't think this approach can scale safely, and this approach
> actually seems more work to me.
>
> (Also think of network traffic and build times, which would presumably be much
> increased by this approach. Disk space shouldn't be much of a problem due to
> the ‘content(/store?) deduplication’ feature of guix.)
>
> > his work is not merged yet, and i think it's not even ready for merging yet.
> > now, i'm rather motivated to work on this, maybe even willing to use the go importer
> > and add countless pinned go packages... but is that desirable? is that the ultimate
> > solution/goal?
>
> Using the go importer (in --recursive mode I presume) seems good, but if with "pinned"
> you mean "multiple versions of the same go module in Guix", I would avoid that if
> possible, due to the reasons I noted above.
> If the various dependents of a go package aren't to picky about the exact version,
> you could use "guix refresh --type=go" to update the indirect dependencies of
> go-ethereum. (Note: guix refresh doesn't seem to support go yet.)


for projects like go-ethereum, it's not an option for the packager to
make decisions about the version of any of its dependencies. in fact,
i think the go.mod file contains the transitive closure of
dependencies exactly for this reason: to pin down the exact version of
every dependency.

with that in sight, i'm not sure anymore that it makes sense to
isolate go-ethereum and all its dependencies into a separate scheme
module and file. maybe every go library should be added to golang.scm
and toplevel applications like go-ethereum into their own file, or
into finance.scm (where bitcoin lives now).

to sum it up: with my limited experience, i only see two viable
options: 1) allow go to fetch, authenticate, and build everything, or
2) use `guix import go` to mirror what is contained in the go.mod and
go.sum files on the guix side (but to me it feels to be an enormous
effort compared to what we gain with it).

a variation of 1) could be to do "vendoring" (ala NixOS): build an
authenticated cache of every dependency that then gets fetched from a
substitute server as a single archive whenever someone wants to build
any go project that has the same mod.go and mod.sums files.

does this make sense?

- attila

Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 8217 bytes --]

Attila Lendvai schreef op wo 01-09-2021 om 14:29 [+0000]:
> On Wednesday, September 1st, 2021 at 00:21, Maxime Devos <maximedevos@telenet.be> wrote:
> 
> > Hi,
> > 
> > Warning: I haven't actually ever touched a go package. Take my mail
> > with a huge grain of salt.
> > Much of this you've probably already heard at
> > https://logs.guix.gnu.org/guix/2021-08-31.log#024401.
> > 
> > Attila Lendvai schreef op ma 30-08-2021 om 21:52 [+0000]:
> > 
> > > [...]
> > > 
> > > so, regarding go-ethereum, i've seen this:
> > > https://issues.guix.gnu.org/43872
> > > the initial conclusion was that the proper way to package a go project is to package the pinned transitive
> > > closure of every dependency. there's a go importer now, which is functional/hackable enough that this is not
> > > a hopeless task, but... i'm doubtful that it's a good idea to multiply the number of Guix packages by such an endeavor... :)
> > 
> > This situation doesn't seem all that different from, say, importing 'evolution'
> > (a GNOME e-mail program) in the hypothetical situation that guix doesn't have
> > any GTK or GNOME library already packaged. I don't think you have to worry
> > about adding many guix packages. Presumably, the new guix packages would
> 
> would you have the same opinion if some guix go packages would have
> 10+ versions in parallel? (see later)

Were is the ‘later’?  I haven't seen anything about multiple versions in parallel
below ...

> 
> > have uses outside go-ethereum, so they can be re-used as dependencies of
> > new go packages, so over time, having to define many new packages when importing
> > a go application should become less and less of a problem.
> > 
> > (About version pinning: I'm ignoring version incompatibilities here. I don't know
> > how much of a problem that is in practice ...)
> 
> i'm not a go expert at all, but i think it's a rather frequent
> situation in go land to move a dependency one git commit ahead, only
> to pick up a bugfix that has just been pushed into its repo.

... unless you meant this here?  But this seems to imply that version
incompatibilities typically don't happen (otherwise just picking up
a bugfix with a single commit doesn't seem plausible), so we can
typically just use the latest version of the go module in guix, no?

> > Maybe I'm spouting nonsense here though, (gnu packages golang) has been around
> > since 2016, and possibly go-ethereum has much more (indirect) dependencies than
> > 'evolution'.
> 
> go-ethereum has 70 dependencies listed in its go.mod file, but it has
> 657 entries in the go.sum, which contains the hash of the transitive
> closure of dependencies.

I'm not sure if you mean that go-ethereum has very many or little dependencies
here.  To compare, openttd has about 494 indirect dependencies (looking at
guix graph --type=package openttd).  Many of these are python packages (82),
texlive packages (120), the X libraries (about 28 or so), the GTK+ stack,
basic stuff like autoconf and automake ...  Most of them packages that wouldn't
seem unique to openttd.

> this mechanism is part of how go builds are reproducible.

> building go-ethereum locally should yield the same binary that they
> have released.

FWIW, ‘reproducibility’ in guix means that, if the build is repeated on another
machine, with the same architecture and guix version, the binaries will be
exactly the same.  Bit-for-bit reproducibility isn't guaranteed when the builds
are performed on different distro's.

More practically speaking, if the guix package definition makes even a tiny change,
the binaries will be different.  E.g., the guix package definition of go replaces
"/etc/protocols" with "/gnu/store/[HASH-...]/etc/protocols, which changes the resulting
hash.

As a demonstration, consider the 'syncthing' package.

$ guix build syncthing
[... downloading ...]
/gnu/store/9fmbnnjjhp2y578lgrg9s4ywwr4kn3xg-syncthing-1.16.1
/gnu/store/yg79kc20y8smjglxbd75w100ypbszj3c-syncthing-1.16.1-utils

$ guix gc --references /gnu/store/9fmbnnjjhp2y578lgrg9s4ywwr4kn3xg-syncthing-1.16.1
/gnu/store/kl68v5mclwp511xgpsl2h1s9gmsdxpzh-tzdata-2021a
/gnu/store/zfbbn61ij7w0bl4wbrwi87x5ghqx968c-net-base-5.3

When tzdata or net-base is updated, their hash will changes so the binaries of syncthing
will change.

Another problem: looking at the definition of go@1.14, it appears go assumes
the loader to be at /lib/lib(64)/ld-linux.[...].so.[...], but that doesn't even
exist on Guix, so binaries created on another distro cannot be run on guix,
and not the other way around (unless they are statically linked perhaps?
But static linking prevents grafting of things like glibc, which sometimes
needs security fixes ...  Why are go programs statically linked in Guix anyways?)

Outside some very specific cases (e.g.
https://reproducible-builds.org/news/2019/12/21/reproducible-bootstrap-of-mes-c-compiler/),
‘cross-distro bit-for-bit reproducibility’ doesn't seem very plausible.

Also ...

> building go-ethereum locally should yield the same binary that they
> have released.

... why should it?

> > [...]
> > Another problem is: if a go package has many (transitive) dependencies, how do
> > we check that it doesn't contain any malware or non-free components? That needs
> 
> go-ethereum is a software that handles ethereum wallets with several
> zeroes. they are probably equally worried about this if not more.

I'm sure the go-ethereum peope wouldn't mind if we double-check that no malware
slipt through their review process.

> > Using the go importer (in --recursive mode I presume) seems good, but if with "pinned"
> > you mean "multiple versions of the same go module in Guix", I would avoid that if
> > possible, due to the reasons I noted above.
> > If the various dependents of a go package aren't to picky about the exact version,
> > you could use "guix refresh --type=go" to update the indirect dependencies of
> > go-ethereum. (Note: guix refresh doesn't seem to support go yet.)
> 
> for projects like go-ethereum, it's not an option for the packager to
> make decisions about the version of any of its dependencies.

Why isn't this an option?  Choosing different versions from upstream is already
done in guix, see e.g. https://issues.guix.gnu.org/50217 (ok that's not merged
yet, not the best example --- note, editing the version requirement was
on my advice, see https://logs.guix.gnu.org/guix/2021-08-26.log).

>  in fact,
> i think the go.mod file contains the transitive closure of
> dependencies exactly for this reason: to pin down the exact version of
> every dependency.

That's my understanding as well.  However, we don't have to follow
the choices made upstream.

> with that in sight, i'm not sure anymore that it makes sense to
> isolate go-ethereum and all its dependencies into a separate scheme
> module and file. maybe every go library should be added to golang.scm
> and toplevel applications like go-ethereum into their own file, or
> into finance.scm (where bitcoin lives now).
> 
> to sum it up: with my limited experience, i only see two viable
> options: 1) allow go to fetch, authenticate, and build everything, or
> 2) use `guix import go` to mirror what is contained in the go.mod and
> go.sum files on the guix side (but to me it feels to be an enormous
> effort compared to what we gain with it).

I see a third viable option (3): treat the "go.sum" as a mere
‘friendly suggestion’, and just use the latest version when feasible.
Again, I don't see much difference with, say, haskell, python, ruby,
guile, java ... packages.

Have there been any problems in practice with just using the latest
version (updating the version currently in guix where applicable)?

I hope my reply has been valueable.  It seems like this discussion
is going to end with ‘agreeing to disagree’ and I think I said all
I've got to say about the philosophical part of the matter, so I
don't think I'll sending further replies on the philosophical parts.
But if you have any technical questions, feel free to ask me.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Attila Lendvai]

[with some reorderings]

> I hope my reply has been valueable. It seems like this discussion
> is going to end with ‘agreeing to disagree’ and I think I said all
> I've got to say about the philosophical part of the matter, so I
> don't think I'll sending further replies on the philosophical parts.
> But if you have any technical questions, feel free to ask me.


ok, noted. your feedback is much appreciated, and i don't mean to
argue against the guix philosopy. i'm converted after all! :)

so, i'll try to focus on only pointing out perspectives that you may
not be aware of, and proposing solutions.


> > this mechanism is part of how go builds are reproducible.
> > building go-ethereum locally should yield the same binary that they
> > have released.
>
> FWIW, ‘reproducibility’ in guix means that, if the build is repeated on another
> machine, with the same architecture and guix version, the binaries will be
> exactly the same. Bit-for-bit reproducibility isn't guaranteed when the builds
> are performed on different distro's.


thank you for clarifying this! i'll use clearer language when talking
about reproducibility.


> the loader to be at /lib/lib(64)/ld-linux.[...].so.[...], but that doesn't even
> exist on Guix, so binaries created on another distro cannot be run on guix,


FTR, the official binary seems to start up fine on my guix, and begins
syncing with some peers:

$ ~/.guix-profile/lib/ld-linux-x86-64.so.2 ./geth-linux-amd64-1.10.8-26675454/geth --syncmode light

it's not statically linked:

$ ldd ./geth-linux-amd64-1.10.8-26675454/geth
	linux-vdso.so.1 (0x00007fff26da6000)
	libpthread.so.0 => /gnu/store/ksy2b6fwfmz40gjajvspl87ia4vsfzj7-glibc-2.31/lib/libpthread.so.0 (0x00007f86794af000)
	libm.so.6 => /gnu/store/ksy2b6fwfmz40gjajvspl87ia4vsfzj7-glibc-2.31/lib/libm.so.6 (0x00007f867936e000)
	librt.so.1 => /gnu/store/ksy2b6fwfmz40gjajvspl87ia4vsfzj7-glibc-2.31/lib/librt.so.1 (0x00007f8679364000)
	libc.so.6 => /gnu/store/ksy2b6fwfmz40gjajvspl87ia4vsfzj7-glibc-2.31/lib/libc.so.6 (0x00007f86791a7000)
	/lib64/ld-linux-x86-64.so.2 => /gnu/store/ksy2b6fwfmz40gjajvspl87ia4vsfzj7-glibc-2.31/lib/ld-linux-x86-64.so.2 (0x00007f86794d2000)

will this have any pitfalls? is there anything i'm not aware of?

if not, then i propose adding a package that downloads the official
release, authenticates it by checking its signature, and sets up a
wrapper script for it to be able to start it by its proper name.

so, maybe we should add two packages? one that fetches the official
release binary, and one that builds it from source (and doesn't
strictly follow the versions in the project's go.mod file)?

i, as a user, would be more relaxed running the authenticated official
binary release. maybe this is misguided, and in that case i'm happy to
be corrected.


> > building go-ethereum locally should yield the same binary that they
> > have released.
>
> ... why should it?


so that, ideally, we are able to use the trusted guix intrastructure
to rebuild the same binary locally that the project has
released... and with that double-check also the release process of the
upstream.


> > > Another problem is: if a go package has many (transitive) dependencies, how do
> > > we check that it doesn't contain any malware or non-free components? That needs
> >
> > go-ethereum is a software that handles ethereum wallets with several
> > zeroes. they are probably equally worried about this if not more.
>
> I'm sure the go-ethereum peope wouldn't mind if we double-check that no malware
> slipt through their review process.

[...]

> > for projects like go-ethereum, it's not an option for the packager to
> > make decisions about the version of any of its dependencies.
>
> Why isn't this an option? Choosing different versions from upstream is already
> done in guix, see e.g. https://issues.guix.gnu.org/50217 (ok that's not merged
> yet, not the best example --- note, editing the version requirement was
> on my advice, see https://logs.guix.gnu.org/guix/2021-08-26.log).


extra auditing on top of the official release is certainly welcome by
each player. but, realisticly, i doubt that the hundreds of guix go
packages will get comparable amount of security-auditing-attention any
time soon. let alone -- and this is of key importance here -- auditing
any possible cross-interactions of altered versions of the
dependencies! who else is qualified to do that better than the
developers themselves?

for a program like the ethereum client, any such semantic surprises
may be disasterous. (losing money in various ways. e.g. their client
forking off of the consensus of the ethereum network, and if e.g. they
are paricipating in staking, then they will suffer penalties for not
adhering to the rules of the network, etc.)

this is why i would prefer to have a solution that somehow reproduces,
compares, and runs the same binary as the officially released one.

in the follwig days i'll play with reproducing the geth release binary
on my guix, and also planning to mock up a binary downloader package,
and see how it goes. i'll report back with anything worth mentioning.


> I see a third viable option (3): treat the "go.sum" as a mere
> ‘friendly suggestion’, and just use the latest version when feasible.
> Again, I don't see much difference with, say, haskell, python, ruby,
> guile, java ... packages.


my point here is not that all go projects should be packaged like this
(i.e. with exacly pinned dependencies), but that applications like
geth should be, regardless of the language they are written in.

in case of a random image viewer written in go, i'm all in for a
relaxed handling of dependencies.


> Have there been any problems in practice with just using the latest
> version (updating the version currently in guix where applicable)?


not that i'm aware of, but the first such identified issue could turn
out to be very expensive.

- attila

Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Bengt Richter]

On +2021-09-03 08:16:00 +0000, Attila Lendvai wrote:
> 
> [with some reorderings]
> 
> > I hope my reply has been valueable. It seems like this discussion
> > is going to end with ‘agreeing to disagree’ and I think I said all
> > I've got to say about the philosophical part of the matter, so I
> > don't think I'll sending further replies on the philosophical parts.
> > But if you have any technical questions, feel free to ask me.
> 
> 
> ok, noted. your feedback is much appreciated, and i don't mean to
> argue against the guix philosopy. i'm converted after all! :)
> 
> so, i'll try to focus on only pointing out perspectives that you may
> not be aware of, and proposing solutions.
> 
> 
> > > this mechanism is part of how go builds are reproducible.
> > > building go-ethereum locally should yield the same binary that they
> > > have released.
> >
> > FWIW, ‘reproducibility’ in guix means that, if the build is repeated on another
> > machine, with the same architecture and guix version, the binaries will be
> > exactly the same. Bit-for-bit reproducibility isn't guaranteed when the builds
> > are performed on different distro's.
> 
> 
> thank you for clarifying this! i'll use clearer language when talking
> about reproducibility.
> 
> 
> > the loader to be at /lib/lib(64)/ld-linux.[...].so.[...], but that doesn't even
> > exist on Guix, so binaries created on another distro cannot be run on guix,
> 
> 
> FTR, the official binary seems to start up fine on my guix, and begins
> syncing with some peers:
> 
> $ ~/.guix-profile/lib/ld-linux-x86-64.so.2 ./geth-linux-amd64-1.10.8-26675454/geth --syncmode light
> 
> it's not statically linked:
> 
> $ ldd ./geth-linux-amd64-1.10.8-26675454/geth
> 	linux-vdso.so.1 (0x00007fff26da6000)
> 	libpthread.so.0 => /gnu/store/ksy2b6fwfmz40gjajvspl87ia4vsfzj7-glibc-2.31/lib/libpthread.so.0 (0x00007f86794af000)
> 	libm.so.6 => /gnu/store/ksy2b6fwfmz40gjajvspl87ia4vsfzj7-glibc-2.31/lib/libm.so.6 (0x00007f867936e000)
> 	librt.so.1 => /gnu/store/ksy2b6fwfmz40gjajvspl87ia4vsfzj7-glibc-2.31/lib/librt.so.1 (0x00007f8679364000)
> 	libc.so.6 => /gnu/store/ksy2b6fwfmz40gjajvspl87ia4vsfzj7-glibc-2.31/lib/libc.so.6 (0x00007f86791a7000)
> 	/lib64/ld-linux-x86-64.so.2 => /gnu/store/ksy2b6fwfmz40gjajvspl87ia4vsfzj7-glibc-2.31/lib/ld-linux-x86-64.so.2 (0x00007f86794d2000)
> 
> will this have any pitfalls? is there anything i'm not aware of?

For the paranoid, from man ldd, red emphasis mine (if it worked)
--8<---------------cut here---------------start------------->8---
   Security
       Be  aware that in some circumstances (e.g., where the program specifies an ELF interpreter other than ld-linux.so), some versions of ldd may attempt
       to obtain the dependency information by attempting to directly execute the program, which may lead to the execution of whatever code is  defined  in
       the  program's ELF interpreter, and perhaps to execution of the program itself.  (In glibc versions before 2.27, the upstream ldd implementation did
       this for example, although most distributions provided a modified version that did not.)

^[[1m^[[31m
       Thus, you should never employ ldd on an untrusted executable, since this may result in the execution of arbitrary code.  A  safer  alternative  when
       dealing with untrusted executables is:

           $ objdump -p /path/to/program | grep NEEDED

       Note,  however,  that  this alternative shows only the direct dependencies of the executable, while ldd shows the entire dependency tree of the exe‐
       cutable.^[[m\x0f
--8<---------------cut here---------------end--------------->8---

> 
> if not, then i propose adding a package that downloads the official
> release, authenticates it by checking its signature, and sets up a
> wrapper script for it to be able to start it by its proper name.
> 
> so, maybe we should add two packages? one that fetches the official
> release binary, and one that builds it from source (and doesn't
> strictly follow the versions in the project's go.mod file)?
> 
> i, as a user, would be more relaxed running the authenticated official
> binary release. maybe this is misguided, and in that case i'm happy to
> be corrected.
> 
> 
> > > building go-ethereum locally should yield the same binary that they
> > > have released.
> >
> > ... why should it?
> 
> 
> so that, ideally, we are able to use the trusted guix intrastructure
> to rebuild the same binary locally that the project has
> released... and with that double-check also the release process of the
> upstream.
> 
> 
> > > > Another problem is: if a go package has many (transitive) dependencies, how do
> > > > we check that it doesn't contain any malware or non-free components? That needs
> > >
> > > go-ethereum is a software that handles ethereum wallets with several
> > > zeroes. they are probably equally worried about this if not more.
> >
> > I'm sure the go-ethereum peope wouldn't mind if we double-check that no malware
> > slipt through their review process.
> 
> [...]
> 
> > > for projects like go-ethereum, it's not an option for the packager to
> > > make decisions about the version of any of its dependencies.
> >
> > Why isn't this an option? Choosing different versions from upstream is already
> > done in guix, see e.g. https://issues.guix.gnu.org/50217 (ok that's not merged
> > yet, not the best example --- note, editing the version requirement was
> > on my advice, see https://logs.guix.gnu.org/guix/2021-08-26.log).
> 
> 
> extra auditing on top of the official release is certainly welcome by
> each player. but, realisticly, i doubt that the hundreds of guix go
> packages will get comparable amount of security-auditing-attention any
> time soon. let alone -- and this is of key importance here -- auditing
> any possible cross-interactions of altered versions of the
> dependencies! who else is qualified to do that better than the
> developers themselves?
> 
> for a program like the ethereum client, any such semantic surprises
> may be disasterous. (losing money in various ways. e.g. their client
> forking off of the consensus of the ethereum network, and if e.g. they
> are paricipating in staking, then they will suffer penalties for not
> adhering to the rules of the network, etc.)
> 
> this is why i would prefer to have a solution that somehow reproduces,
> compares, and runs the same binary as the officially released one.
> 
> in the follwig days i'll play with reproducing the geth release binary
> on my guix, and also planning to mock up a binary downloader package,
> and see how it goes. i'll report back with anything worth mentioning.
> 
> 
> > I see a third viable option (3): treat the "go.sum" as a mere
> > ‘friendly suggestion’, and just use the latest version when feasible.
> > Again, I don't see much difference with, say, haskell, python, ruby,
> > guile, java ... packages.
> 
> 
> my point here is not that all go projects should be packaged like this
> (i.e. with exacly pinned dependencies), but that applications like
> geth should be, regardless of the language they are written in.
> 
> in case of a random image viewer written in go, i'm all in for a
> relaxed handling of dependencies.
> 
> 
> > Have there been any problems in practice with just using the latest
> > version (updating the version currently in guix where applicable)?
> 
> 
> not that i'm aware of, but the first such identified issue could turn
> out to be very expensive.
> 
> - attila
> 
> 

-- 
Regards,
Bengt Richter

Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Attila Lendvai]

> > will this have any pitfalls? is there anything i'm not aware of?
>
> For the paranoid, from man ldd, red emphasis mine (if it worked)


i included ldd's output only for reference.

what i meant to ask is this: is it a dirty hack to run a (trusted)
binary file like this:

$ ~/.guix-profile/lib/ld-linux-x86-64.so.2 some-foreign-elf-linux-binary

because this way go-ethereum's official binary release seems to be
running happily on guix.

or is this something "legal" that is meant to work?

- attila

Re: packaging go-ethereum, and ultimately bee (of ethswarm.org)

[Sarah Morgensen]

Hi,

Maxime Devos <maximedevos@telenet.be> writes:

>> > have uses outside go-ethereum, so they can be re-used as dependencies of
>> > new go packages, so over time, having to define many new packages when importing
>> > a go application should become less and less of a problem.
>> > 
>> > (About version pinning: I'm ignoring version incompatibilities here. I don't know
>> > how much of a problem that is in practice ...)
>> 
>> i'm not a go expert at all, but i think it's a rather frequent
>> situation in go land to move a dependency one git commit ahead, only
>> to pick up a bugfix that has just been pushed into its repo.
>
> ... unless you meant this here?  But this seems to imply that version
> incompatibilities typically don't happen (otherwise just picking up
> a bugfix with a single commit doesn't seem plausible), so we can
> typically just use the latest version of the go module in guix, no?

A clarification: a canonical Go module version is supposed to follow
semantic versioning guidelines [0], so anything except a major version
bump is supposed to be backwards-compatible.  This is further supported
by major version suffixes [1], which requires that "[s]tarting with
major version 2, module paths must have a major version suffix like /v2
that maches the major version.  For example, if a module has the path
example.com/mod at v1.0.0, it must have the path example.com/mod/v2 at
version v2.0.0."  Because each Guix package only represents one module
path, and that module path is used in the name of the package, we
*should* be able to update to (and use) any newer minor versions of a
module with no issues.

This doesn't address packages without any canonical versions, of
course...

[0] https://golang.org/ref/mod#versions
[1] https://golang.org/ref/mod#major-version-suffixes

--
Sarah