From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id QAqmGTt8mF88HwAA0tVLHw (envelope-from ) for ; Tue, 27 Oct 2020 19:59:55 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id ED94FTt8mF8uIwAAB5/wlQ (envelope-from ) for ; Tue, 27 Oct 2020 19:59:55 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9FBB09402A8 for ; Tue, 27 Oct 2020 19:59:52 +0000 (UTC) Received: from localhost ([::1]:41110 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kXUnq-0002DN-Qd for larch@yhetil.org; Tue, 27 Oct 2020 15:38:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36896) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kXU3k-0005Cc-2d for guix-patches@gnu.org; Tue, 27 Oct 2020 14:51:05 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:34320) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kXU3i-0007Bp-6E for guix-patches@gnu.org; Tue, 27 Oct 2020 14:51:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kXU3i-0000Gw-4Y for guix-patches@gnu.org; Tue, 27 Oct 2020 14:51:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#44199] [PATCH 0/1] An origin method for GNUnet FS URI's Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 27 Oct 2020 18:51:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44199 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: zimoun , 44199@debbugs.gnu.org Received: via spool by 44199-submit@debbugs.gnu.org id=B44199.16038246511029 (code B ref 44199); Tue, 27 Oct 2020 18:51:02 +0000 Received: (at 44199) by debbugs.gnu.org; 27 Oct 2020 18:50:51 +0000 Received: from localhost ([127.0.0.1]:45866 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kXU3X-0000GV-7O for submit@debbugs.gnu.org; Tue, 27 Oct 2020 14:50:51 -0400 Received: from rhcavuit03.kulnet.kuleuven.be ([134.58.240.136]:37561) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kXU3S-0000GJ-W2 for 44199@debbugs.gnu.org; Tue, 27 Oct 2020 14:50:49 -0400 X-KULeuven-Envelope-From: maxime.devos@student.kuleuven.be X-KULeuven-Scanned: Found to be clean X-KULeuven-ID: 406E5120334.AD994 X-KULeuven-Information: Katholieke Universiteit Leuven Received: from icts-p-smtps-2.cc.kuleuven.be (icts-p-smtps-2e.kulnet.kuleuven.be [134.58.240.34]) by rhcavuit03.kulnet.kuleuven.be (Postfix) with ESMTP id 406E5120334 for <44199@debbugs.gnu.org>; Tue, 27 Oct 2020 19:50:35 +0100 (CET) Received: from butterfly.local (178-119-10-153.access.telenet.be [178.119.10.153]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by icts-p-smtps-2.cc.kuleuven.be (Postfix) with ESMTPSA id 0720C200A1; Tue, 27 Oct 2020 19:50:35 +0100 (CET) Message-ID: X-Kuleuven: This mail passed the K.U.Leuven mailcluster From: Maxime Devos Date: Tue, 27 Oct 2020 19:50:18 +0100 In-Reply-To: <86blgn4wk5.fsf@gmail.com> References: <5c72bcb9c86934deda97d952eb5cd459e615b313.camel@student.kuleuven.be> <86blgn4wk5.fsf@gmail.com> Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-GeyBALaEfrdfiy9v0H53" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -3.3 (---) X-Mailman-Approved-At: Tue, 27 Oct 2020 15:38:14 -0400 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 X-Spam-Score: 6.39 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam: Yes X-TUID: rL1P1Csd6S/w --=-GeyBALaEfrdfiy9v0H53 Content-Type: multipart/mixed; boundary="=-jIVA5jS/ZzYs+D3AM7cV" --=-jIVA5jS/ZzYs+D3AM7cV Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable zimoun schreef op di 27-10-2020 om 14:39 [+0100]: > Dear, >=20 > Thank you for the patch. My questions are totally naive since I do > not > know much about GNUnet. Conceptually, the GNUnet file-sharing system is not unlike IPFS, dat, or even torrents, although the details can vary wildly. I don't know much more about GNUnet either except for how to install it, and how to publish and search for files. > On Sat, 24 Oct 2020 at 21:47, Maxime Devos < > maxime.devos@student.kuleuven.be> wrote: > > This patch defines a `gnunet-fetch' method, allowing for > > downloading > > files from GNUnet by their GNUnet chk-URI. > >=20 > > This patch does not provide: > > - a service configuration > > - downloading substitutes from GNUnet > > - fall-back to non-P2P (e.g. http://) or other P2P (e.g. ipfs://) > > systems > > - downloading directories over GNUnet >=20 > This means it only works for archives as tarball, right? More-or-less, yes. I would assume a zip archive would work just fine as well, and in the rare case where the source code consists of a single file (e.g. search for the %boot-logo-patch in (gnu packages linux)), you don't need an archive at all. GNUnet has a special format for directories, presumably supporting deduplication (I haven't checked), conceptually not unlike ipfs and dat. It is probably not too much work to support this format, but I would like to keep the first patch simple. > > - Would it be possible somehow for url-fetch to support > > gnunet://fs/chk > > URIs? That way we could fall-back unto non-P2P URLs, which would > > be > > useful to bootstrap a P2P distribution from a non-P2P system. >=20 > Who is the =E2=80=9Cwe=E2=80=9D? What do you mean by =E2=80=9Curl-fetch = supports gnunet:// > and > fall-back unto non-P2P=E2=80=9D? Presuming this patch is accepted in some form, =E2=80=98we=E2=80=99 refers = to code in Guix responsible for downloading archives. I should have formulated that better. About the =E2=80=98fall-back unto non-P2P=E2=80=99: The ultimate goal is to not rely on any centralised servers at all (not even mirrors) (these are the non-P2P systems), and download all archives and substitutes from GNUnet FS (or IPFS or dat, but I prefer GNUnet). Of course, expecting everyone to stop using HTTP / ftp / etc. for source distribution and instead use GNUnet immediately is foolish, so ideally there would be a smooth transition path: 1. At first, maintainers would typically still publish the tarballs(*) on a centralised (possibly mirrored) servers, say . The package definition would use the url-fetch method. 2. If $MAINTAINER doesn't have access to or doesn't want to use any publicly-available distribution site (e.g. due to censorship, or for pseudonymity or simplicity), $MAINTAINER may instead use the gnunet-fetch method in the package definition. Alternatively, if the distribution site disappeared, Guix maintainers have the option to point to GNUnet. (Or refer to swh, I presume, but my goal is to decentralise.) 4. $MAINTAINER finds GNUnet convenient. However, GNUnet hasn't achieved world-domination yet, so $MAINTAINER still publishes tarballs on or similar. Actually, even if GNUnet *was* universally supported, a centralised server can still be useful: P2P file-sharing systems by design may have to throw away old data the local peer isn't interested in (due to limited disk space), and $MAINTAINER might stop publishing the source code over GNUnet. In such cases, a centralised server (and some mirrors) may be a useful back-up, although it would still be preferred to use a distributed P2P system (such as GNUnet or ipfs) when available to limit the cost of running the centralised server (and mirrors). (To limit the utilisation of the centralised server (and mirrors), sources downloaded from the server (and mirrors) should be published back into the GNUnet FS, but this can be as simple and crude as a cron job `gnunet-publish /gnu/store/*.tar.gz`.) My idea in this case is to specify the origin as: (origin (method url-fetch) (uri (list "gnunet://fs/chk/etcetera" (string-append "mirror://gnu/hello-" version ".tar.gz"))) etcetera) However, this patch defines gnunet-fetch as a separate method from url-fetch, url-fetch is left untouched, so url-fetch will just ignore the gnunet://fs/chk URI's. If I understand correctly, url-fetch is a special method, in that the downloading code of the Guix daemon is used, and not the code defined by the user. Nothing a `guix system reconfigure` cannot fix, but nevertheless rather inconvenient for testing. > Some recent discussions are about content-address and fallback. For > example, roughly speaking =E2=80=99git-fetch=E2=80=99 tries upstream, the= n the Guix > build farms, then Software Heritage (swh). For Git repo, it works > because the address from Guix side to SWH is straightforward. The 2 > other VCS =E2=80=93hg and svn=E2=80=93 supported by SWH should be impleme= nted soon=E2=80=A6 > who > knows! ;-) \me missed these discussions > The story about archives as tarball is a bit more complicated. The > main > issue =E2=80=93as I understand it=E2=80=93 can be summarized as: Guix kno= ws the URL, > the > integrity checksum and only at package time the content of the > tarball. > Later in time, it is difficult to lookup because of this very > address; > and some are around: nar, swh-id, ipfs, gnunet, etc. >=20 > Bridges to reassemble the content are currently discussed, e.g., >=20 > > >=20 > Well, today the fallback of tarball archive to SWH is not reliable. >=20 >=20 > What is your question? ;-) The question to which answer? I'm not quite which answer you're referring to, so some guesses: Q: How does Guix figure out the GNUnet URI from the Guix (or nar, I guess) hash? A: Not automatically. The gnunet-fetch method as defined in this patch needs to be passed the URI manually. However, an additional service for GNUnet can be written that uses the DHT to map Guix (or nar, or something else) hashes to corresponding GNUnet URI's. (I'm not volunteering (yet)) Q: What about automatically generated tarballs (e.g. from git repositories)? A: Not addressed by this patch. The intention is to be able to replace a http://*/*.tar.gz URL with a gnunet://fs/chk URI in package definitions; source code repositories aren't supported by this patch. (But perhaps a future patch could support this!) Q: Integration with bridges? A: Sounds nice, but I'm trying to keep things simple for the first patch! (And I haven't heard of disarchive before now.) > > Then publish the source tarball of the package to the GNUnet FS > > system: > > $ guix environment --ad-hoc wget -- wget=20 > > https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz > > $ gnunet-publish hello-2.10.tar.gz >=20 > Naive question: are packages only available on GNUnet? Naive interpretations of this question, and answers: * Is the only thing available on GNUnet, *packages*? While experimenting with GNUnet, I have seen some images published on GNUnet. GNUnet also has other systems that the FS system, but they aren't relevant to this path. * Are packages *definitions* only available, on *GNUnet*? I'm getting my Guix package definitions from the git repository on Savannah, and I have never encountered any on the GNUnet FS system. * Is package *source code* only available, on *GNUnet*? If someone published the source code (e.g. as a tarball) on GNUnet with `gnunet-publish hello-2.10.tar.gz`, it is only published(*) on GNUnet, and not somewhere else as well. (*) I don't know exactly when one can be reasonably sure the file will *remain* available for some time when published, except for keeping the GNUnet daemon running continuously. However, in practice, $MAINTAINER will publish the source code somewhere else as well (e.g. or perhaps ipfs). This patch doesn't automatically publish source code of built or downloaded packages on GNUnet, although that seems a useful service=20 to run as a daemon. * If the gnunet-fetch method is used, will Guix try to get the source code from GNUnet, and nowhere else? Ignoring substitutes, yes. This is a limitation of defining gnunet-fetch separately from url-fetch. I believe this has been addressed earlier in my response e-mail. > All the best, > simon Likewise, maxime --=-jIVA5jS/ZzYs+D3AM7cV Content-Type: application/pgp-encrypted; name="Maxime Devos.pgp" Content-Disposition: attachment; filename="Maxime Devos.pgp" Content-Transfer-Encoding: base64 mDMEX4ch6BYJKwYBBAHaRw8BAQdANPb/d6MrGnGi5HyvODCkBUJPRjiFQcRU5V+mxvMaAa+0L01h eGltZSBEZXZvcyA8bWF4aW1lLmRldm9zQHN0dWRlbnQua3VsZXV2ZW4uYmU+iJAEExYIADgWIQTB 8z7iDFKP233XAR9J4+4iGRcl7gUCX4ch6AIbAwULCQgHAwUVCgkICwUWAgMBAAIeAQIXgAAKCRBJ 4+4iGRcl7japAQC3opZ2KGWzWmRc/gIWSu0AAcfMwyinFEEPa/QhUt2CogD/e2RdF4CYAgaRHJJm Z9WU7piKbLZ7llB4LzgezVDHggy4OARfhyHoEgorBgEEAZdVAQUBAQdATpcyFoic4KGzBev3KsC2 i5O4yHBUmfvnNJPXAjU+mF8DAQgHiHgEGBYIACAWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCX4ch 6AIbDAAKCRBJ4+4iGRcl7jK9AQCXPHJeTxr1ZlbkkFSo5XjZeAytf1wGjwGD3t+ABH2STwD/SKiw kIR/xXiWpx6gT/IDXRjSNqp/EZ5y1X0SYFI2dQQ= --=-jIVA5jS/ZzYs+D3AM7cV-- --=-GeyBALaEfrdfiy9v0H53 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iJcEABYIAD8WIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCX5hr6iEcbWF4aW1lLmRl dm9zQHN0dWRlbnQua3VsZXV2ZW4uYmUACgkQSePuIhkXJe6rXwD/ZvuV2F0+CXOR aeeehho3jMT8oxE5ZmiF1oFILu6Qv7kA/1GCpvV0HKO7t3z++u7liNcCm/Ui7PBY 0DCP6YjQufYP =R3pG -----END PGP SIGNATURE----- --=-GeyBALaEfrdfiy9v0H53--