Re: intrinsic vs extrinsic identifier: toward more robustness?

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

Hi Maxime (it's been some time, welcome back!)

Maxime Devos <maximedevos@telenet.be> writes:

[...]

> I think nar stuff should be kept outside SWH.  It doesn't seem
> scalable to me for SWH to support the format of every distribution.
> Likewise, I think that SWH identifiers should _not_ become an
> intrinsic identifier that is recorded in package definitions -- if
> there are other archives that are somewhat SWH-like archives, then
> Guix should support them too even if they don't use SWH identifiers
> for whatever reason, and including the identifier of every single
> archive seems unscalable to me.
>
> I believe I have a solution on how to solve the ‘everyone uses
> different identifiers, how to map between them’ problem, but it will
> take some paragraphs:
>
> At some point in the past, when thinking about downloading source code
> over GNUnet File-sharing (FS), I had the problem that Guix and GNUnet
> uses different intrinsic identifiers -- Guix uses the NAR hash for
> querying substitute servers, whereas FS has a system of its own that's
> more convenient for P2P file-sharing stuff.
>
> The problem then was to somehow map the NAR hash to the FS identifier.
> I couldn't do this the Disarchive way, because the point was to be
> _P2P_ and Disarchive ... isn't.
>
> A straightforward solution would be to just replace the https:// by
> gnunet:// in the origin (like in https://issues.guix.gnu.org/44199,
> except that patch doesn't support fallbacks to other URLs like
> url-fetch does).
>
> The problem was that people demanded that gnunet:// should only be
> supported once there is actually source code on GNUnet and GNUnet is
> stable, but why would people put source code on GNUnet when no
> distribution supports it and how would GNUnet become stable without
> any users?
>
> To work-around these circular demands, I started 'rehash':
> <https://lists.gnu.org/archive/html/guix-patches/2021-01/msg01067.html>
> (current location: https://notabug.org/maximed/rehash).  It is a
> (P2P!) GNUnet service that maintains a 'SHA1512<->GNUnet FS URI'
> mapping, or more generally, a 'this hash type<->that hash type'
> mapping.
>
> (It is just a service on top of the DHT, so the same could easily be
> done for BitTorrent or IPFS.)
>
> It's rather incomplete at the moment (there is no verification or
> reputation mechanism at all so the network could be flooded with bogus
> mappings, mappings are only in DHT, not stored on disk, so they are
> lost on reboot, the POC Guix integretation is a bit limited), but the
> basics are there -- the POC successfully downloaded a substitute over
> GNUnet _without_ having to include FS URI in the narinfo (*)!.
>
> I'm writing about substitutes here, but the exact same approach could
> be done for plain source code.
>
> (*) I might have misremembered; I can't find the POC on
> issues.guix.gnu.org again, and I'm not sure if the POC used rehash or
> if it just included the FS URI in the narinfo.
>
> (TBC, I haven't been working on Rehash lately, but rather
> Scheme-GNUnet: a Scheme port of the GNUnet libraries that's less
> limited than Guile-GNUnet.  Idea is to make GNUnet-FS and rehash more
> convenient to use from Scheme, and in particular, in Guix.)

Thanks for sharing your efforts on the P2P in Guix/GNUnet front!  P2P
seems like it'd make substitutes mirroring easy and improve robustness
as the network gets populated.  It's very interesting; it'd definitely
make an interesting summer internship :-).

Keep up the good and inspiring hacks!

-- 
Thanks,
Maxim