From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id qCToHdaG12HiywAAgWs5BA (envelope-from ) for ; Fri, 07 Jan 2022 01:18:30 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id COaHFtaG12HMlQAAG6o9tA (envelope-from ) for ; Fri, 07 Jan 2022 01:18:30 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0470738A66 for ; Fri, 7 Jan 2022 01:18:30 +0100 (CET) Received: from localhost ([::1]:33786 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n5cxg-0006jU-SJ for larch@yhetil.org; Thu, 06 Jan 2022 19:18:28 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43830) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n5cxG-0006jG-9k for bug-guix@gnu.org; Thu, 06 Jan 2022 19:18:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:49770) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n5cxG-00046Q-0C for bug-guix@gnu.org; Thu, 06 Jan 2022 19:18:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n5cxF-0002rx-OZ for bug-guix@gnu.org; Thu, 06 Jan 2022 19:18:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#34170: bitcoin-core bundles leveldb Resent-From: Carl Dong Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 07 Jan 2022 00:18:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 34170 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Maxime Devos Received: via spool by 34170-submit@debbugs.gnu.org id=B34170.164151465110971 (code B ref 34170); Fri, 07 Jan 2022 00:18:01 +0000 Received: (at 34170) by debbugs.gnu.org; 7 Jan 2022 00:17:31 +0000 Received: from localhost ([127.0.0.1]:42673 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n5cwk-0002qs-GQ for submit@debbugs.gnu.org; Thu, 06 Jan 2022 19:17:31 -0500 Received: from out5-smtp.messagingengine.com ([66.111.4.29]:35765) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n5cwh-0002qQ-HZ for 34170@debbugs.gnu.org; Thu, 06 Jan 2022 19:17:28 -0500 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id B51EC5C0138; Thu, 6 Jan 2022 19:17:21 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 06 Jan 2022 19:17:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=carldong.me; h= from:message-id:content-type:mime-version:subject:date :in-reply-to:cc:to:references; s=fm2; bh=qSVtEvNdenrW1qZuGUit6lF e0N4Itk7WdBXLmIvK+JM=; b=kj/ThT5HNrmgHHEm+j77yhBK1/ansZBZBuJrQdt NLxRSUzAxzrRzZ/Gh+yc3Tit31OYVU4KuTuUWDqkBb+01JrOBKv5zCr9z5t4m7LT kpc8YUQZZlFw1GiAA2toJeV4juJhmHYznX1LpY7l/q3WLv8acGS5MVmReVyuh2OV rxdurpLSz6lJCMTmykb52klhvhMnnxWZIzCsSB+b5QZSav8A7OIuXEDtX5c70+jD b9kUc5wN265eUMv9UyJ8L7RJ7m7xpT++LPvZA8M57v4mWrskVaRXi0TbC900DKra M3HmNTdB33B6JXqGFkaV5VH1iBwv0T7741rkgBoivJdemdg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=qSVtEv NdenrW1qZuGUit6lFe0N4Itk7WdBXLmIvK+JM=; b=KBWnk56uWEosaIVAVULv2r 86i2O7KBnEVvDiOVLRey28ifdfqk7oxHc4axnYkrA9zo639C3lV6x2It8ryD8riQ stshbfJ8lrl567537sPrYMTw0kuqpDgHroPYoU1hVbqTY1ypoWM0YHYIHmpZ6L9C NmYt2IBQQICh0GyWpH1JK5zBOw2b4UIp3zWX2DEhNeHjKlhaynSy0F0vqaYDXDxx v4wlpCM0sseyF5usa5XKCbEEtTvtLhyx7EXhMQryudQrj8mtT+WhdyDRdpwNYnV/ ypaHUdANxtTZT5f3z2HLn8hf8zTAJzCKxfOQ+3j07tnRba9M5/q3TYN6A9sYzazw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrudegtddgudejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffktgggufffjgfvfhfosehgtdhmrehhtdejnecuhfhrohhmpeevrghrlhcu ffhonhhguceotghonhhtrggtthestggrrhhlughonhhgrdhmvgeqnecuggftrfgrthhtvg hrnhepudduhfefgeeuleelueehgfekjeejjefgvdetleffueegleeifeejffelfeffveeh necuffhomhgrihhnpehgihhthhhusgdrtghomhdpuggvsghirghnrdhorhhgpdgsihhtmh gvgidrtghomhenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhr ohhmpegtohhnthgrtghtsegtrghrlhguohhnghdrmhgv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 6 Jan 2022 19:17:21 -0500 (EST) From: Carl Dong Message-Id: <70A07344-0DA9-4A8E-B3B0-FAE1721FE2D2@carldong.me> Content-Type: multipart/signed; boundary="Apple-Mail=_6138C549-0547-443F-A5D5-DA86153C4A4E"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Date: Thu, 6 Jan 2022 19:17:13 -0500 In-Reply-To: <563546ec57ff4726bf4138c16be6304c869119ce.camel@telenet.be> References: <20190122133124.7ca032d7@scratchpost.org> <865ys6m168.fsf@gmail.com> <86lezu8qj4.fsf@gmail.com> <26B84833-6F9F-4C5D-8448-715C67C16768@carldong.me> <563546ec57ff4726bf4138c16be6304c869119ce.camel@telenet.be> X-Mailer: Apple Mail (2.3693.40.0.1.81) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 34170@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1641514710; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=qSVtEvNdenrW1qZuGUit6lFe0N4Itk7WdBXLmIvK+JM=; b=TPymyosQJ1SFvvb9ojViVOJnQdg9pLxC7zeu8FRigpThwh0VAzYTD5k7nWK26TOxpIBQXs MHsE1AXejQdO/3K1iLyu6d7QsSJA+ZJwTkx2pLlfv8MXyESmD03h15atBftQC1fJfj5f7r Ma5hu0GHkeJo2BLussLd4Y8425qmTVCBZoGlH76YIG3L4mqL9tmoTzhyDKUQnmuKY2qoRJ ySchR6sOhqqYX9VUAcQyVFCt9Wgh41tcVB30Vf1ti7pVfPEHTj12JjGjHvTPj0vlNMIEfV 0tL1i9JkVcFKF7zs5bc4zUZEmbF4kum82lUXofrH3fxTTbpqnxcOzAk0YI6Fcg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1641514710; a=rsa-sha256; cv=none; b=gKYQJ4OZY3mKevFmrBmBoo7XKK892zFekvUR/e2AQ7NtXktRWV2/szmSDckh1pG1lBNEGE 6jWLRX/dqBkA/SOcF1CUSYOq4DQqPJQVUeVOyzdLFtM8U+pNVFseQUFZ4c0Is0eVfyIFC2 xASvCr7Bdq8m8Awo+0Sr4Svq9VKg+bdRxwbIQHlFHXSOe0fn5+zejrk1nDRC541MLN1zz6 mQ40oPGdOIr0pV/il0eYIJwiXocBCUdxyNP18IRv3WBxrORhBnVPpNGYxP5W+07pyjbilE p00Pe1X5NE4V7hjZ+T6eU+mxbT6rtCMnygCfTNTxAf0qKgfiMcndU0WlPt7xFA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=carldong.me header.s=fm2 header.b="kj/ThT5H"; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm1 header.b=KBWnk56u; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.50 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=carldong.me header.s=fm2 header.b="kj/ThT5H"; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm1 header.b=KBWnk56u; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 0470738A66 X-Spam-Score: -4.50 X-Migadu-Scanner: scn0.migadu.com X-TUID: mjqi1EhlEQ57 --Apple-Mail=_6138C549-0547-443F-A5D5-DA86153C4A4E Content-Type: multipart/alternative; boundary="Apple-Mail=_197A65E3-A5A8-4610-B7FB-FE262E4037E8" --Apple-Mail=_197A65E3-A5A8-4610-B7FB-FE262E4037E8 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > If desired, it would also be possible to do something in-between > unbundling and using bitcoin's leveldb: define a 'leveldb/bitcoin' > variant of the 'leveldb' package (using package/inherit or (package > (inherit ...) ...)), add it as input to the 'bitcoin' package and tell > and/or patch bitcoin's buid scripts to use that leveldb. Yes I think that would be a splendid idea! With regards to patching = bitcoin=E2=80=99s builds scripts, Bitcoin Knots follows Bitcoin Core = closely, but has a bunch of patches which allow for using system libs, = so that might be good to reference: = https://github.com/bitcoin/bitcoin/compare/master...bitcoinknots:21.x-sysl= ibs = > As source code, use an appropriate commit from > (and add a comment > to the definition of bitcoin-core to keep leveldb/bitcoin in-sync). FYI, according to https://github.com/bitcoin/bitcoin/pull/17398 = , we are currently using = the upstream LevelDB commit 0c40829872a9f00f38e11dc370ff8adb3e19f25b Cheers, Carl Dong > On Jan 5, 2022, at 2:13 PM, Maxime Devos = wrote: >=20 > Hi, >=20 > Carl Dong schreef op wo 05-01-2022 om 12:40 [-0500]: >> Simon, Maxime, Danny, >>=20 >> Thanks for CCing me on this message! The rationale for bundling >> leveldb in Bitcoin Core goes a bit beyond convenience, it is several >> things: >>=20 >> 1. The original reason for sub-treeing is that Bitcoin Core used to >> maintain its own version of leveldb with its own fixes >> here: https://github.com/bitcoin-core/leveldb-subtree, since then >> most of these fixes have been upstreamed as >> of: https://github.com/bitcoin/bitcoin/pull/17398 >=20 > Seems reasonable to me, but the bitcoin project is upstreaming the > changes it made and most are already upstream, so I would prefer > to use upstream's leveldb. >=20 >> 2. We also used to support using an external leveldb, however, it >> seems that it was fragile to rely on external projects to maintain >> ABI compatibility, see the quoted IRC bug report >> here: https://github.com/bitcoin/bitcoin/pull/23282. Reasonable minds >> may disagree on this point, especially coming from Guix where >> patching is convenient. >=20 > The quoted ABI incompatibility >=20 > bitcoind fails to start with undefined symbol: > _ZTIN7leveldb6LoggerE in Debian Sid after leveldb upgraded from 1.22 = to > 1.23: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D996486 >=20 > doesn't apply in Guix, because guix uses RUNPATH and multiple library > versions can exist in the same store (in different directories in the > store). >=20 >> In addition to the above, Bitcoin Core experienced a hard fork in >> 2013 due to database incompatibilities, which has predisposed >> maintainers towards a more stringent approach with pinning >> dependencies and their configure/build-time flags. >> See: https://blog.bitmex.com/bitcoins-consensus-forks/#was-the-2013- >> incident-a-hardfork >=20 > I doubt that Guix has sufficient Bitcoin Core users to cause > a hard fork, but yes, this is an understandable reason to bundle > things. But any such problem seems easy to resolve (at the guix = side): > we could simply temporarily switch to an older version of leveldb. >=20 > If desired, it would also be possible to do something in-between > unbundling and using bitcoin's leveldb: define a 'leveldb/bitcoin' > variant of the 'leveldb' package (using package/inherit or (package > (inherit ...) ...)), add it as input to the 'bitcoin' package and tell > and/or patch bitcoin's buid scripts to use that leveldb. >=20 > As source code, use an appropriate commit from > (and add a comment > to the definition of bitcoin-core to keep leveldb/bitcoin in-sync). >=20 > A benefit of this approach (if done properly, with (origin (inherit > ...) ...) such that patches of 'leveldb' are inherited) above the > status quo, is that is that if for some reason 'leveldb' is patched in > Guix, then 'leveldb/bitcoin' receives the patch as well. Another > benefit is that the dependency 'googletest' and 'benchmark' of leveldb > would remain unbundled. >=20 > Greetings, > Maxime. >=20 --Apple-Mail=_197A65E3-A5A8-4610-B7FB-FE262E4037E8 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
If desired, it would also be = possible to do something in-between
unbundling and using = bitcoin's leveldb: define a 'leveldb/bitcoin'
variant of = the 'leveldb' package (using package/inherit or (package
(inherit ...) ...)), add it as input to the 'bitcoin' package = and tell
and/or patch bitcoin's buid scripts to use = that leveldb.
Yes I think that would be a = splendid idea! With regards to patching bitcoin=E2=80=99s builds = scripts, Bitcoin Knots follows Bitcoin Core closely, but has a bunch of = patches which allow for using system libs, so that might be good to = reference: https://github.com/bitcoin/bitcoin/compare/master...bitcoinknot= s:21.x-syslibs

As source code, use an = appropriate commit from
<https://github.com/bitcoin-core/leveldb-subtree> (and = add a comment
to the definition of bitcoin-core to keep = leveldb/bitcoin in-sync).
FYI, = according to https://github.com/bitcoin/bitcoin/pull/17398, we are = currently using the upstream LevelDB commit 0c40829872a9f00f38e11dc370ff8adb3e19f25b

Cheers,
Carl Dong


On Jan = 5, 2022, at 2:13 PM, Maxime Devos <maximedevos@telenet.be> wrote:

Hi,

Carl Dong schreef op wo 05-01-2022 om 12:40 [-0500]:
Simon, Maxime, Danny,

Thanks for CCing me on this message! The = rationale for bundling
leveldb in Bitcoin Core goes a bit = beyond convenience, it is several
things:

1. The original reason for sub-treeing is that Bitcoin Core = used to
maintain its own version of leveldb with its own = fixes
here: https://github.com/bitcoin-core/leveldb-subtree, since = then
most of these fixes have been upstreamed as
of: https://github.com/bitcoin/bitcoin/pull/17398

Seems reasonable to me, but the = bitcoin project is upstreaming the
changes it made and = most are already upstream, so I would prefer
to use = upstream's leveldb.

2. We also used to support using an external leveldb, = however, it
seems that it was fragile to rely on external = projects to maintain
ABI compatibility, see the quoted IRC = bug report
here: https://github.com/bitcoin/bitcoin/pull/23282. Reasonable = minds
may disagree on this point, especially coming from = Guix where
patching is convenient.

The quoted ABI incompatibility

   <Talkless> bitcoind = fails to start with undefined symbol:
_ZTIN7leveldb6LoggerE = in Debian Sid after leveldb upgraded from 1.22 to
1.23: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D996486<= br class=3D"">
doesn't apply in Guix, because guix uses = RUNPATH and multiple library
versions can exist in the = same store (in different directories in the
store).

In = addition to the above, Bitcoin Core experienced a hard fork in
2013 due to database incompatibilities, which has = predisposed
maintainers towards a more stringent approach = with pinning
dependencies and their configure/build-time = flags.
See: https://blog.bitmex.com/bitcoins-consensus-forks/#was-the-2013-=
incident-a-hardfork

I doubt that Guix has sufficient Bitcoin Core users to = cause
a hard fork, but yes, this is an understandable = reason to bundle
things.  But any such problem seems = easy to resolve (at the guix side):
we could simply = temporarily switch to an older version of leveldb.

If desired, it would also be possible to do something = in-between
unbundling and using bitcoin's leveldb: define = a 'leveldb/bitcoin'
variant of the 'leveldb' package = (using package/inherit or (package
(inherit ...) ...)), = add it as input to the 'bitcoin' package and tell
and/or = patch bitcoin's buid scripts to use that leveldb.

As source code, use an appropriate commit from
<https://github.com/bitcoin-core/leveldb-subtree> (and = add a comment
to the definition of bitcoin-core to keep = leveldb/bitcoin in-sync).

A benefit of this = approach (if done properly, with (origin (inherit
...) = ...) such that patches of 'leveldb' are inherited) above the
status quo, is that is that if for some reason 'leveldb' is = patched in
Guix, then 'leveldb/bitcoin' receives the patch = as well. Another
benefit is that the dependency = 'googletest' and 'benchmark' of leveldb
would remain = unbundled.

Greetings,
Maxime.


= --Apple-Mail=_197A65E3-A5A8-4610-B7FB-FE262E4037E8-- --Apple-Mail=_6138C549-0547-443F-A5D5-DA86153C4A4E Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEECc3SW1JEo3Z49u6oDMUhUxl5kaUFAmHXhokACgkQDMUhUxl5 kaXf6Q/+Ia5jCiSY7hC0o1wR1B+ZfxMtxEPTrGk93P+7HOASskMZEcKDNFkoKbSK s/BZT22vfjo6yLle+JWRDXaR+9lhvyGtjQ1aiSCrRMtCPX8k1sBIiVFxYA/E0A4t WUVN1aS4B2h9wc5olT4vVWwwmjZpUBCpdiCXeSMLDhB6w85HHFxVAmmrPUaghxYb obvu24Pajrgvi8CndUJjPBUV0iWq0b6g/Ri6DiB9rUKbNghZp6S4twoKR3VqKbrd 6C/zeKWBBNrTPmrQMb13PezPTU3gK76tZ2r472W+ZRGWqLM5BTPmkJcV8UA8Ac57 yZkKkfYn5EDX8KU/2mssODwaLpURZfcTZEHsEbWzHje8+6uXbSRehnwXIbRdwvs7 P/doZTa5H9oau+IS07d6j7Z+/Oe5A801YPbuzzVxDSwAhcHZqI471WrcI6TbFixL TEdQTRqU8w0EjlD9LVJC2t+W7B3rNuWfeMRuS48yFkf7g2YtKInF2oK0hvR5LAem akbO4Fh1GGeIZ7cpLqq4V9HL3IodiZEUTBMBuF8P+x61DsW99mf/yjzUBW+RSS2j ABsKPYr37Bz+4I9OslPsEE9yRLclc1HTF3FuCZuQl1qEUWHOkJP/aeCZW/sdgMxi yp4GaPXdWeKacosCN/4xehlOkZyIeCya3PlIm9TnYQ0o48p5Hps= =vXwb -----END PGP SIGNATURE----- --Apple-Mail=_6138C549-0547-443F-A5D5-DA86153C4A4E--