From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id cBmAN/SvL2HS/QAAgWs5BA (envelope-from ) for ; Wed, 01 Sep 2021 18:53:08 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id WCguM/SvL2GtdwAA1q6Kng (envelope-from ) for ; Wed, 01 Sep 2021 16:53:08 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 35ECF12038 for ; Wed, 1 Sep 2021 18:53:08 +0200 (CEST) Received: from localhost ([::1]:46922 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mLTU3-0000Kw-9T for larch@yhetil.org; Wed, 01 Sep 2021 12:53:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34590) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mLSvk-0006lr-Pt for guix-devel@gnu.org; Wed, 01 Sep 2021 12:17:42 -0400 Received: from xavier.telenet-ops.be ([2a02:1800:120:4::f00:14]:35416) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mLSvh-0001t2-AC for guix-devel@gnu.org; Wed, 01 Sep 2021 12:17:39 -0400 Received: from ptr-bvsjgyjmffd7q9timvx.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:aaf1:9810:a0b8:a55d]) by xavier.telenet-ops.be with bizsmtp id ogHY250070mfAB401gHYyg; Wed, 01 Sep 2021 18:17:32 +0200 Message-ID: Subject: Re: packaging go-ethereum, and ultimately bee (of ethswarm.org) From: Maxime Devos To: Attila Lendvai Date: Wed, 01 Sep 2021 18:17:12 +0200 In-Reply-To: References: <3a88b1abb853f8217e8dcc05810f1fb3f8468004.camel@telenet.be> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-RZuqNOW25MG1H6hlm2fh" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r21; t=1630513052; bh=wEQHDBJAjVsOtiXv3bYyNCUKfJTilH2RuyW5Y2sJIlY=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=XWWesAozD2rDIIT4DD6F6uHOEWkOb+4/m55Zs8FXMRhMVg1/Uh2ufxw4LwAEs/5/m iNJVDI4EIk2tf7vJfk/ZOCeqAMXE28bxLR2R838FdfdBARe086IiLm13IqBdPsuBSe WeInEJDSf82M+/PkmurEBXsUmKQjgvkzRqM+xh+x/bcOr0ngG2ryDwJ4LBDTxQiTrv 7ipmpeCmge21ol3qx0XvA62gcumb7cMq+MgMd/4JOmP1850nGSXu+LKMmdXfnyKfC/ UG/wL1Yl+jM+q7o0/CJAr4W0ILNS7kUkGmm6L4AGiIdxXlvPl23jrPcCoMm79bhfOs poxa6CNZwKdiA== Received-SPF: pass client-ip=2a02:1800:120:4::f00:14; envelope-from=maximedevos@telenet.be; helo=xavier.telenet-ops.be X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "guix-devel@gnu.org" Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1630515188; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=wEQHDBJAjVsOtiXv3bYyNCUKfJTilH2RuyW5Y2sJIlY=; b=MG1nvpSrbc5IExkSBGsLityCZ9rRKw8UPwHwOQP6gTc/gu8xD5yZXXk3FE7GL6VmiM6B5y eML0wSmD4EjWrbjxdqBvoGr6CcmMI57ezAliEeldKsO3Lzze4WMY8lhDwHCcLuZ/BvVaHh CiQZVA97lX40ihtutj4N0CIDbTFyLq+RD2DK+UEA4kAO9ha1bTbN6fUoYTfOdXfbpySFiw RljylUxoU0xdEOa/KS5vLVr/+vSHrtQkro+MTw6weJ3OIMnNuzc0s7elN32KUPmqMcUk3B 9b8P+UwGVyQR4TsQ3U9CIdSNTHpeMctFCcr3xBrCarZtTzr31TfGH0zJNYq5ew== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1630515188; a=rsa-sha256; cv=none; b=O3Pf5KqFzXFQVp9sbaxldMUq8pOOly3kgDIQdKikJa23TpBEdxlKoqJgdpwMThbCjWIW/d 9e1cLIUYJ4i5kXaWh1ifsnRlisIWhkRyBeDf9E3c7fuJjg9XISXCCaQet6Iqzpso9/MGzf /uc68lZ8Uidn5EyOh94n+HqkCH1rCjlHzWrWAdxCJn8KaWk9mAnEdE0MzDVVApIZ1i2Nte aN/zNWcZE2laHUXtBwLLNiWFuDZkwt4LFfkPi+GWjM28sfodRljzmOM+vo3Em8OduHdfxw NTZMS1u3xaaxPs5c/iwpzYHCjJVieapQMgwYmkIjAH4YfC3AbyXNC/wERg0S/w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=telenet.be header.s=r21 header.b=XWWesAoz; dmarc=pass (policy=none) header.from=telenet.be; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -5.22 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=telenet.be header.s=r21 header.b=XWWesAoz; dmarc=pass (policy=none) header.from=telenet.be; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 35ECF12038 X-Spam-Score: -5.22 X-Migadu-Scanner: scn1.migadu.com X-TUID: aBSEAFY8QQM5 --=-RZuqNOW25MG1H6hlm2fh Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Attila Lendvai schreef op wo 01-09-2021 om 14:29 [+0000]: > On Wednesday, September 1st, 2021 at 00:21, Maxime Devos wrote: >=20 > > Hi, > >=20 > > Warning: I haven't actually ever touched a go package. Take my mail > > with a huge grain of salt. > > Much of this you've probably already heard at > > https://logs.guix.gnu.org/guix/2021-08-31.log#024401. > >=20 > > Attila Lendvai schreef op ma 30-08-2021 om 21:52 [+0000]: > >=20 > > > [...] > > >=20 > > > so, regarding go-ethereum, i've seen this: > > > https://issues.guix.gnu.org/43872 > > > the initial conclusion was that the proper way to package a go projec= t is to package the pinned transitive > > > closure of every dependency. there's a go importer now, which is func= tional/hackable enough that this is not > > > a hopeless task, but... i'm doubtful that it's a good idea to multipl= y the number of Guix packages by such an endeavor... :) > >=20 > > This situation doesn't seem all that different from, say, importing 'ev= olution' > > (a GNOME e-mail program) in the hypothetical situation that guix doesn'= t have > > any GTK or GNOME library already packaged. I don't think you have to wo= rry > > about adding many guix packages. Presumably, the new guix packages woul= d >=20 > would you have the same opinion if some guix go packages would have > 10+ versions in parallel? (see later) Were is the =E2=80=98later=E2=80=99? I haven't seen anything about multipl= e versions in parallel below ... >=20 > > have uses outside go-ethereum, so they can be re-used as dependencies o= f > > new go packages, so over time, having to define many new packages when = importing > > a go application should become less and less of a problem. > >=20 > > (About version pinning: I'm ignoring version incompatibilities here. I = don't know > > how much of a problem that is in practice ...) >=20 > i'm not a go expert at all, but i think it's a rather frequent > situation in go land to move a dependency one git commit ahead, only > to pick up a bugfix that has just been pushed into its repo. ... unless you meant this here? But this seems to imply that version incompatibilities typically don't happen (otherwise just picking up a bugfix with a single commit doesn't seem plausible), so we can typically just use the latest version of the go module in guix, no? > > Maybe I'm spouting nonsense here though, (gnu packages golang) has been= around > > since 2016, and possibly go-ethereum has much more (indirect) dependenc= ies than > > 'evolution'. >=20 > go-ethereum has 70 dependencies listed in its go.mod file, but it has > 657 entries in the go.sum, which contains the hash of the transitive > closure of dependencies. I'm not sure if you mean that go-ethereum has very many or little dependenc= ies here. To compare, openttd has about 494 indirect dependencies (looking at guix graph --type=3Dpackage openttd). Many of these are python packages (8= 2), texlive packages (120), the X libraries (about 28 or so), the GTK+ stack, basic stuff like autoconf and automake ... Most of them packages that woul= dn't seem unique to openttd. > this mechanism is part of how go builds are reproducible. > building go-ethereum locally should yield the same binary that they > have released. FWIW, =E2=80=98reproducibility=E2=80=99 in guix means that, if the build is= repeated on another machine, with the same architecture and guix version, the binaries will be exactly the same. Bit-for-bit reproducibility isn't guaranteed when the bu= ilds are performed on different distro's. More practically speaking, if the guix package definition makes even a tiny= change, the binaries will be different. E.g., the guix package definition of go re= places "/etc/protocols" with "/gnu/store/[HASH-...]/etc/protocols, which changes t= he resulting hash. As a demonstration, consider the 'syncthing' package. $ guix build syncthing [... downloading ...] /gnu/store/9fmbnnjjhp2y578lgrg9s4ywwr4kn3xg-syncthing-1.16.1 /gnu/store/yg79kc20y8smjglxbd75w100ypbszj3c-syncthing-1.16.1-utils $ guix gc --references /gnu/store/9fmbnnjjhp2y578lgrg9s4ywwr4kn3xg-syncthin= g-1.16.1 /gnu/store/kl68v5mclwp511xgpsl2h1s9gmsdxpzh-tzdata-2021a /gnu/store/zfbbn61ij7w0bl4wbrwi87x5ghqx968c-net-base-5.3 When tzdata or net-base is updated, their hash will changes so the binaries= of syncthing will change. Another problem: looking at the definition of go@1.14, it appears go assume= s the loader to be at /lib/lib(64)/ld-linux.[...].so.[...], but that doesn't = even exist on Guix, so binaries created on another distro cannot be run on guix, and not the other way around (unless they are statically linked perhaps? But static linking prevents grafting of things like glibc, which sometimes needs security fixes ... Why are go programs statically linked in Guix any= ways?) Outside some very specific cases (e.g. https://reproducible-builds.org/news/2019/12/21/reproducible-bootstrap-of-m= es-c-compiler/), =E2=80=98cross-distro bit-for-bit reproducibility=E2=80=99 doesn't seem ver= y plausible. Also ... > building go-ethereum locally should yield the same binary that they > have released. ... why should it? > > [...] > > Another problem is: if a go package has many (transitive) dependencies,= how do > > we check that it doesn't contain any malware or non-free components? Th= at needs >=20 > go-ethereum is a software that handles ethereum wallets with several > zeroes. they are probably equally worried about this if not more. I'm sure the go-ethereum peope wouldn't mind if we double-check that no mal= ware slipt through their review process. > > Using the go importer (in --recursive mode I presume) seems good, but i= f with "pinned" > > you mean "multiple versions of the same go module in Guix", I would avo= id that if > > possible, due to the reasons I noted above. > > If the various dependents of a go package aren't to picky about the exa= ct version, > > you could use "guix refresh --type=3Dgo" to update the indirect depende= ncies of > > go-ethereum. (Note: guix refresh doesn't seem to support go yet.) >=20 > for projects like go-ethereum, it's not an option for the packager to > make decisions about the version of any of its dependencies. Why isn't this an option? Choosing different versions from upstream is alr= eady done in guix, see e.g. https://issues.guix.gnu.org/50217 (ok that's not mer= ged yet, not the best example --- note, editing the version requirement was on my advice, see https://logs.guix.gnu.org/guix/2021-08-26.log). > in fact, > i think the go.mod file contains the transitive closure of > dependencies exactly for this reason: to pin down the exact version of > every dependency. That's my understanding as well. However, we don't have to follow the choices made upstream. > with that in sight, i'm not sure anymore that it makes sense to > isolate go-ethereum and all its dependencies into a separate scheme > module and file. maybe every go library should be added to golang.scm > and toplevel applications like go-ethereum into their own file, or > into finance.scm (where bitcoin lives now). >=20 > to sum it up: with my limited experience, i only see two viable > options: 1) allow go to fetch, authenticate, and build everything, or > 2) use `guix import go` to mirror what is contained in the go.mod and > go.sum files on the guix side (but to me it feels to be an enormous > effort compared to what we gain with it). I see a third viable option (3): treat the "go.sum" as a mere =E2=80=98friendly suggestion=E2=80=99, and just use the latest version when= feasible. Again, I don't see much difference with, say, haskell, python, ruby, guile, java ... packages. Have there been any problems in practice with just using the latest version (updating the version currently in guix where applicable)? I hope my reply has been valueable. It seems like this discussion is going to end with =E2=80=98agreeing to disagree=E2=80=99 and I think I s= aid all I've got to say about the philosophical part of the matter, so I don't think I'll sending further replies on the philosophical parts. But if you have any technical questions, feel free to ask me. Greetings, Maxime --=-RZuqNOW25MG1H6hlm2fh Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYS+niBccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7gRaAQDh/K4Tso1ZW26YxR72urIUnii4 y2EQvAwdMDW1q4sMQQD/WsLAdfvbCquV3avLtnWRH9v+0rMAMwmTNbpDfRL8/wE= =kEmO -----END PGP SIGNATURE----- --=-RZuqNOW25MG1H6hlm2fh--