From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id eLQkLHKOL2GJuQAAgWs5BA (envelope-from ) for ; Wed, 01 Sep 2021 16:30:10 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id cBJ5J3KOL2HbKQAAbx9fmQ (envelope-from ) for ; Wed, 01 Sep 2021 14:30:10 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 361FAEE65 for ; Wed, 1 Sep 2021 16:30:09 +0200 (CEST) Received: from localhost ([::1]:58026 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mLRFg-0000eG-Bx for larch@yhetil.org; Wed, 01 Sep 2021 10:30:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38476) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mLRFT-0000de-BK for guix-devel@gnu.org; Wed, 01 Sep 2021 10:29:55 -0400 Received: from mail-40136.protonmail.ch ([185.70.40.136]:42868) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mLRFP-0000Fl-RR for guix-devel@gnu.org; Wed, 01 Sep 2021 10:29:54 -0400 Date: Wed, 01 Sep 2021 14:29:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lendvai.name; s=protonmail2; t=1630506586; bh=HJyliSPMX0N3DSYDzCEmIbyd1C5VT8FH6A6RdCdiJPg=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=vx5rQMdF/l8K0HcQnXpT8Pd+TXO1cNStBOUMV0PF38Ysss5RopTSw1KBaTr9xE4Vw TsNEChG9VqfCajzp7JbyxKPxC0DY3vk3dslYNRuSNHHSuIcV+XUKRZ60pQ7mXIyZjG ocRxBLhUO/J3ojjzx7DfaoyNjrBcQ7NtJ6CDr5K2H7+D0DUDN8XYpL09V3yp6DccHg BxfN0rub5rIZzog408CHXclOD68CVwvUTyT5Vl46+/vNetAOZzvyotRb+9vBT2DoK7 bOUCoD+elMkDwTwR0rzu4wkgL64xYS3XX3v7W4zwdLcIveJKrOno5f6GRmF2vt1hWR iOZr0IBDnwe2g== To: Maxime Devos From: Attila Lendvai Subject: Re: packaging go-ethereum, and ultimately bee (of ethswarm.org) Message-ID: In-Reply-To: <3a88b1abb853f8217e8dcc05810f1fb3f8468004.camel@telenet.be> References: <3a88b1abb853f8217e8dcc05810f1fb3f8468004.camel@telenet.be> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=185.70.40.136; envelope-from=attila@lendvai.name; helo=mail-40136.protonmail.ch X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Attila Lendvai Cc: "guix-devel@gnu.org" Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1630506609; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=HJyliSPMX0N3DSYDzCEmIbyd1C5VT8FH6A6RdCdiJPg=; b=nxDcvFMRBXUlO+DVubAJh6+53a53xEHWlAVL5+XwjHpz5XglP/bdu0VUmyNPSsPWkLJzLr atw5k/peNcFd1lAgnrUXRqZq41uNza/4vJpIX9Qmj/qOQZeMb9CchFvqQ3gYlOEahq35jq 55E6MnfneZsTZOyXmj1YemdxTMRsVdRXnMu1v0Y2SWR5pmrF9obMLRCknsh0YgNK6lIAjY zT8dGZfebHpAH6nqYBjfVOcYFo88qaAcpqXVxws8VUbawFrkjIgMJGLjhDrovlOxlwdKBl Sod/r5Y1KZHXJeri1GTdRjfMTOmkJtaKcLOJl9/m5AJGFwLc87SYGuikb6fQPQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1630506609; a=rsa-sha256; cv=none; b=D+i2h6nXd+Hep/JP5Zt9Yfb9dxPF95dsLrJXb2wXe4OlhSEGcvbdWZCuFFNk2QCQYxxrXK 071esQf2usbqD+TNTxGNuwiaFsrGxU+A8ILJhrnnsyTxkyW6fNvQyFktjXKZ5j3YPM6N4D E/Q5otAn//ls+U4FLil+ri9c92Y/wbXZDZBAarcrlwf6gUpYpVqmnsfOPfnf2LNJH5873X /02IplP1Fp0yiJGBrJtyjbP8HgS0u2P53o0P/dmnAFpj4eP2YrUvIIqD7sqg7w8jIR6D4O XLHaCFnox35aGC/3gRM+VMLW3lxe4NLUtBQQLs7S/S1POzXJrQMzmyLCVInNyg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=lendvai.name header.s=protonmail2 header.b=vx5rQMdF; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -2.62 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=lendvai.name header.s=protonmail2 header.b=vx5rQMdF; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 361FAEE65 X-Spam-Score: -2.62 X-Migadu-Scanner: scn1.migadu.com X-TUID: 3+IpLMFRanjn On Wednesday, September 1st, 2021 at 00:21, Maxime Devos wrote: > Hi, > > Warning: I haven't actually ever touched a go package. Take my mail > with a huge grain of salt. > Much of this you've probably already heard at > https://logs.guix.gnu.org/guix/2021-08-31.log#024401. > > Attila Lendvai schreef op ma 30-08-2021 om 21:52 [+0000]: > > > [...] > > > > so, regarding go-ethereum, i've seen this: > > https://issues.guix.gnu.org/43872 > > the initial conclusion was that the proper way to package a go project = is to package the pinned transitive > > closure of every dependency. there's a go importer now, which is functi= onal/hackable enough that this is not > > a hopeless task, but... i'm doubtful that it's a good idea to multiply = the number of Guix packages by such an endeavor... :) > > This situation doesn't seem all that different from, say, importing 'evol= ution' > (a GNOME e-mail program) in the hypothetical situation that guix doesn't = have > any GTK or GNOME library already packaged. I don't think you have to worr= y > about adding many guix packages. Presumably, the new guix packages would would you have the same opinion if some guix go packages would have 10+ versions in parallel? (see later) > have uses outside go-ethereum, so they can be re-used as dependencies of > new go packages, so over time, having to define many new packages when im= porting > a go application should become less and less of a problem. > > (About version pinning: I'm ignoring version incompatibilities here. I do= n't know > how much of a problem that is in practice ...) i'm not a go expert at all, but i think it's a rather frequent situation in go land to move a dependency one git commit ahead, only to pick up a bugfix that has just been pushed into its repo. > Maybe I'm spouting nonsense here though, (gnu packages golang) has been a= round > since 2016, and possibly go-ethereum has much more (indirect) dependencie= s than > 'evolution'. go-ethereum has 70 dependencies listed in its go.mod file, but it has 657 entries in the go.sum, which contains the hash of the transitive closure of dependencies. this mechanism is part of how go builds are reproducible. building go-ethereum locally should yield the same binary that they have released. > > then Helio Machado proposed something smarter in a later comment: > > https://issues.guix.gnu.org/43872#3 > > > > IIUC, he proposes a way to instead use the go module system to download= all the dependencies, > > and yet authenticate all the downloaded go code. > > (Parts of what I write below is written in the manual (guix)Submitting pa= tches, > search for =E2=80=986. Make sure the package does not use bundled copies = of software > already available as separate packages.=E2=80=99) > > One problem with this approach, is that a go package can be using very ol= d versions > (possibly with bugs that have long been fixed in the latest versions, or = with security > issues) of dependencies without any indication thereof in the package def= inition, > and "guix refresh -t go" and "guix lint -c cve" can't indicate problems (= ). > > () I don't know if these commands currently work on go packages. > > Another problem is: if a go package has many (transitive) dependencies, h= ow do > we check that it doesn't contain any malware or non-free components? That= needs go-ethereum is a software that handles ethereum wallets with several zeroes. they are probably equally worried about this if not more. > to be checked manually, per package. With the status quo (only have one c= opy > of everything whenever feasible), this only has to be done once per = =E2=80=98go software=E2=80=99 > (go module? I'm not familiar with the terminology). But when using someth= ing > like =E2=80=98https://issues.guix.gnu.org/43872#3=E2=80=99, if multiple = =E2=80=98guix go packages=E2=80=99 > use the same =E2=80=98go module(?)=E2=80=99, then both variants of the mo= dule need to be checked. > > So to conclude, I don't think this approach can scale safely, and this ap= proach > actually seems more work to me. > > (Also think of network traffic and build times, which would presumably be= much > increased by this approach. Disk space shouldn't be much of a problem due= to > the =E2=80=98content(/store?) deduplication=E2=80=99 feature of guix.) > > > his work is not merged yet, and i think it's not even ready for merging= yet. > > now, i'm rather motivated to work on this, maybe even willing to use th= e go importer > > and add countless pinned go packages... but is that desirable? is that = the ultimate > > solution/goal? > > Using the go importer (in --recursive mode I presume) seems good, but if = with "pinned" > you mean "multiple versions of the same go module in Guix", I would avoid= that if > possible, due to the reasons I noted above. > If the various dependents of a go package aren't to picky about the exact= version, > you could use "guix refresh --type=3Dgo" to update the indirect dependenc= ies of > go-ethereum. (Note: guix refresh doesn't seem to support go yet.) for projects like go-ethereum, it's not an option for the packager to make decisions about the version of any of its dependencies. in fact, i think the go.mod file contains the transitive closure of dependencies exactly for this reason: to pin down the exact version of every dependency. with that in sight, i'm not sure anymore that it makes sense to isolate go-ethereum and all its dependencies into a separate scheme module and file. maybe every go library should be added to golang.scm and toplevel applications like go-ethereum into their own file, or into finance.scm (where bitcoin lives now). to sum it up: with my limited experience, i only see two viable options: 1) allow go to fetch, authenticate, and build everything, or 2) use `guix import go` to mirror what is contained in the go.mod and go.sum files on the guix side (but to me it feels to be an enormous effort compared to what we gain with it). a variation of 1) could be to do "vendoring" (ala NixOS): build an authenticated cache of every dependency that then gets fetched from a substitute server as a single archive whenever someone wants to build any go project that has the same mod.go and mod.sums files. does this make sense? - attila