From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id VztsJe3EAmRIcQEAbAwnHQ (envelope-from ) for ; Sat, 04 Mar 2023 05:11:25 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id +KIIJe3EAmTlKgAA9RJhRA (envelope-from ) for ; Sat, 04 Mar 2023 05:11:25 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5606C112D2 for ; Sat, 4 Mar 2023 05:11:25 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pYJEZ-0003aB-Mj; Fri, 03 Mar 2023 23:10:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pYJEY-0003Zo-75 for guix-devel@gnu.org; Fri, 03 Mar 2023 23:10:58 -0500 Received: from mail-qv1-xf2d.google.com ([2607:f8b0:4864:20::f2d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pYJEW-0006Mp-H3 for guix-devel@gnu.org; Fri, 03 Mar 2023 23:10:57 -0500 Received: by mail-qv1-xf2d.google.com with SMTP id nv15so3127407qvb.7 for ; Fri, 03 Mar 2023 20:10:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=59xFOpGTi9GWhZylc0hisdHD5IBdFKI/M58+NkssL7w=; b=F915fNJ0Mj9M9EfSExKfMN3AH4KXKUEJwId1GVuRFUejbzF6BPcjt/FK1cIEoDSxX9 qEPcWivbmuRYRnWzTJIsQTMdJshrFXEkaX3/bpAsdUElV1IvoYd+V/h9XeWHhHIZx4hv 6gPdrhAluqZwniQk8+6tzMFeFFzwbHuyVuFyF/Ea/81ml5O1RyJfgIoRfyGYAbPbMA/n OmhjN1gJumK2RhVSi9+AZqRRdU7uEPW4eQV6LP5X3Swtp5Ib5k+tzqvha6NL7RkNf/wd h7wSXhGxLEvYWncf4Pr+Qxku4XztAzxTt/82+yz6a0+PANmGzH8r0amcL8l2+yq0PXa0 dVCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=59xFOpGTi9GWhZylc0hisdHD5IBdFKI/M58+NkssL7w=; b=DHZzZfwZq/BdjnrU3B5GaeG2YniCfC/Afiev4IemYOWutxUzsm+5t35ubOS2D+7wom sA6bgBG76rfNCu6VnUuAcKE9M2GwdMnFI2SmtjDELb1dUwI9PbhTKhXo1wYJvHwHaBvd LOCnp8nCqI9rXOdJDkxvjcmLFr/D+yiozq6gOtP120c/LYzb1dMTap+kogx9EBNMkC5D 4Na1EQK5EIcnZQ5Kluz1xs5lIFj3RHL1J0E0kvlhmL71LoDUV2m87qg4b3ZPBLCpnxf+ jE/51m1D0Lw1FiS5k8OcrkiwX4EDaMNtdYt7Gq6zuOzu0oM4PaHWl1muC5GWABdeRMco 0s4Q== X-Gm-Message-State: AO0yUKUPRZP/Xsenuotlt2AwDYRxoZ49EeT5MvdMDdEQDzkTKVvprBvr Mg7HG01qET9Aa5Irz1N1jtRHiH/0gvL3s+/W X-Google-Smtp-Source: AK7set9Tt7noZm4jO2Q5gkNFlPJEpIa/wbUQCl4gXpS8jnfgJ6gIxMerWOxEPZRQhXsMU2ShwEjUfw== X-Received: by 2002:a05:6214:19c7:b0:56e:a96a:2be1 with SMTP id j7-20020a05621419c700b0056ea96a2be1mr7965736qvc.48.1677903055159; Fri, 03 Mar 2023 20:10:55 -0800 (PST) Received: from hurd (dsl-10-129-180.b2b2c.ca. [72.10.129.180]) by smtp.gmail.com with ESMTPSA id s2-20020ae9de02000000b00706b09b16fasm3098052qkf.11.2023.03.03.20.10.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Mar 2023 20:10:54 -0800 (PST) From: Maxim Cournoyer To: Maxime Devos Cc: Simon Tournier , Guix Devel Subject: Re: intrinsic vs extrinsic identifier: toward more robustness? References: <87jzzxd7z8.fsf@gmail.com> <09d3d861-0390-3ce6-30c7-22a1e2685787@telenet.be> Date: Fri, 03 Mar 2023 23:10:53 -0500 In-Reply-To: <09d3d861-0390-3ce6-30c7-22a1e2685787@telenet.be> (Maxime Devos's message of "Sat, 4 Mar 2023 01:08:08 +0100") Message-ID: <87mt4t2m2a.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2607:f8b0:4864:20::f2d; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qv1-xf2d.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1677903085; a=rsa-sha256; cv=none; b=O03Fzy8nywEscwvWYWBupfaYdCddc2GVDwt/e67MoPdRALKKS28Po/6ueSLrD8FPQbvI2x +z+LoknXx9m1BTfU4fWrKwOYygAY6ek3+oKvJ7Gud+YvBf8D7OFIn2ytlA16o+zw4r2OD+ /2OtuguXQ3yTM6oy4gelgFzXVvZwBjPra1cqZhAkqnKB9jFWNo70U1Om4QTxhmHe0AVdQ1 jM2lW5gBeby8qiNMdZ1leRK+Zocf9r/7kJ5OYbAGa6zNQqItBDEQIr+GwefNIbf9RpEv8m IqnwRwLyXOZaRef8koIdTM8bEacbYlOyjP7+46vNrDpEMrGFZZZh4KUiG3yNsg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=F915fNJ0; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1677903085; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=59xFOpGTi9GWhZylc0hisdHD5IBdFKI/M58+NkssL7w=; b=Ry/qF6iD64Cin4nwIkm3LUKEiZYh4FjjAUoDQRxeU9Um8Ydf7Ykm60vsZqWlXpX1tGl9DJ jEPZcVxWvFpOjNusW9KNq3RI1e1yxkCHFeFqEHQZBTTwC9XAPA6iOqpwXUHKu0dC53btuN kQNYrq25ikvwMGMeALMf/dcW5yszfyAnETg+OtmGXeJdnNCiSkhvAKlnWlnWefiMMyTDrk igPMagDzZgcLEzjuflH1W1nicA6ISz3UYzS88eUMlqaeMAwifuQb12fTI6Z7hOwe6dqN2c Q7ndiEh4dmhHra5v4TELIH5Mgg1MBnNIJQug0+semSKc3TMXoHp09D30nq8RmA== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -5.42 X-Spam-Score: -5.42 X-Migadu-Queue-Id: 5606C112D2 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=F915fNJ0; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com X-TUID: wYARvcAeoiBw Hi Maxime (it's been some time, welcome back!) Maxime Devos writes: [...] > I think nar stuff should be kept outside SWH. It doesn't seem > scalable to me for SWH to support the format of every distribution. > Likewise, I think that SWH identifiers should _not_ become an > intrinsic identifier that is recorded in package definitions -- if > there are other archives that are somewhat SWH-like archives, then > Guix should support them too even if they don't use SWH identifiers > for whatever reason, and including the identifier of every single > archive seems unscalable to me. > > I believe I have a solution on how to solve the =E2=80=98everyone uses > different identifiers, how to map between them=E2=80=99 problem, but it w= ill > take some paragraphs: > > At some point in the past, when thinking about downloading source code > over GNUnet File-sharing (FS), I had the problem that Guix and GNUnet > uses different intrinsic identifiers -- Guix uses the NAR hash for > querying substitute servers, whereas FS has a system of its own that's > more convenient for P2P file-sharing stuff. > > The problem then was to somehow map the NAR hash to the FS identifier. > I couldn't do this the Disarchive way, because the point was to be > _P2P_ and Disarchive ... isn't. > > A straightforward solution would be to just replace the https:// by > gnunet:// in the origin (like in https://issues.guix.gnu.org/44199, > except that patch doesn't support fallbacks to other URLs like > url-fetch does). > > The problem was that people demanded that gnunet:// should only be > supported once there is actually source code on GNUnet and GNUnet is > stable, but why would people put source code on GNUnet when no > distribution supports it and how would GNUnet become stable without > any users? > > To work-around these circular demands, I started 'rehash': > > (current location: https://notabug.org/maximed/rehash). It is a > (P2P!) GNUnet service that maintains a 'SHA1512<->GNUnet FS URI' > mapping, or more generally, a 'this hash type<->that hash type' > mapping. > > (It is just a service on top of the DHT, so the same could easily be > done for BitTorrent or IPFS.) > > It's rather incomplete at the moment (there is no verification or > reputation mechanism at all so the network could be flooded with bogus > mappings, mappings are only in DHT, not stored on disk, so they are > lost on reboot, the POC Guix integretation is a bit limited), but the > basics are there -- the POC successfully downloaded a substitute over > GNUnet _without_ having to include FS URI in the narinfo (*)!. > > I'm writing about substitutes here, but the exact same approach could > be done for plain source code. > > (*) I might have misremembered; I can't find the POC on > issues.guix.gnu.org again, and I'm not sure if the POC used rehash or > if it just included the FS URI in the narinfo. > > (TBC, I haven't been working on Rehash lately, but rather > Scheme-GNUnet: a Scheme port of the GNUnet libraries that's less > limited than Guile-GNUnet. Idea is to make GNUnet-FS and rehash more > convenient to use from Scheme, and in particular, in Guix.) Thanks for sharing your efforts on the P2P in Guix/GNUnet front! P2P seems like it'd make substitutes mirroring easy and improve robustness as the network gets populated. It's very interesting; it'd definitely make an interesting summer internship :-). Keep up the good and inspiring hacks! --=20 Thanks, Maxim