Re: Architecture to reduce download time when pulling multiple packages – historic success with magnet URLs, BTIHs, & Aria2c!

[JRHaigh+ML.GNU.Guix--- via <help-guix@gnu.org>]

[-- Attachment #1.1: Type: text/plain, Size: 6878 bytes --]

Hi Josh,

At Z-0400=2023-10-13Fri12:36:01, Josh Marshall sent:
> This is to parallelize connections which should never hurt downloading but can help.  Mirroring would be parallelizing for providing packages, what I want to implement is to parallelize obtaining packages.  Server side vs client side.

	Please, if you are going to do something like this, please use a torrent architecture like BitTorrent or GNUnet – I suggest Aria2c as a very good CLI download backend that can be daemonised and sent instructions over a socket to add, pause, remove downloads, etc., and it supports magnet URLs including the existing nontorrent servers (via ‘as’ parameters, iirc.).

	I actually implemented this in a local copy of APT Daemon many years ago (circa 2011), but the change was not accepted upstream to Launchpad (because I was not on bleeding-edge; I was too slow to keep-up with the upstream development).  My fork got forgotten about, because to get the full benefit the server would have had to have added a BitTorrent Info Hash (BTIH) to the metadata of each package, along with the MD5, SHA-256, etc. that it already did (not a big ask, really).  That said, without the full benefit of having the metadata, it did provide immediate benefit and I used it for many years, not upgrading my Ubuntu 11.04 Natty Narwhal that I was using back then until I really had to.

	The immediate benefit that it provided was exactly as you described: It allowed parallelisation of nontorrent downloads, be it from the same server or from multiple mirrors.  Iirc., I achieved this by simply passing the download list to Aria2c in daemon mode, I think I also converted all the HTTP URLs to ‘as’ parameters in magnet links, so that multiple mirrors could be passed using multiple ‘as’ parameters in each magnet link.  Then I simply relied on Aria2c being amazing at parallelising everything that I had given it!  I then also implemented progress updates such that APT Daemon could reflect where Aria2c was up to.

	The way I implemented this using Aria2c and magnet URLs meant that if additional hashes were known, they could be used as well, and so if the server metadata made the simple addition of adding BTIHs, it allows swarming to occur, which in-turn would massively reduce load on the central servers, and allow anyone who want to be a mirror to be a mirror simply by seeding indefinitely.  A default share ratio of 1.0 means that no user is a burden on the network, unless they deliberately change that.  Users can donate to the running costs of the project simply by increasing their share ratio, which adds another means of contribution that they may find easier than the others.

	Anyone keen to keep old packages online can simply seed them indefinitely, so this is also really great for archival purposes.  Even if the central project loses interest in the old packages and deletes them, anyone else can keep them up.  The hashes ensure that they have not been tampered with.

	There is also a really cool benefit that occurs, or can occur, on a LAN.  An entire network of computers can all swarm locally with each other, thus needing each package to only need downloading through the metered last mile bottleneck from the WAN precisely once – providing that local broadcasting is supported.  I think this requires Avahi, and I seem to remember that Aria2c supports this but I can't remember.  I don't ever remember getting this bit working but also I did not try hard because it would have required the metadata that I didn't have until after download, so even if I got it working it would not have been directly useful unless the APT repositories that I was using would include the BTIHs.

	So yeah, loads of great benefits to this architecture, and I highly-recommend it: convert all existing URLs to magnet links (can be done client-side as I did; or server-side); optionally add any additional mirrors as additional ‘as’ parameters (again client-side or server-side); add ‘btih’ parameters to the magnet links (the BTIH must be included in the server metadata to get the full benefit of the swarming, but conversion to magnet link format can be done client-side or server-side); then simply pass all this to a really good parallelising backend such as Aria2c; then update any progress data and relay pause, resume, cancel, etc. to the backend.

	One final note, as I am sure that there are a lot of GNUnet fans on this list, is that I would try Aria2c first to see how well it can work, and then try GNUnet or whatever else once you have a standard to benchmark against.  Both are Free Software, so no concern there.  Aria2c is an all-round download manager CLI that works with or without swarming, i.e. it is just as good at HTTPS as it is BitTorrent, and can do both at the same time.  GNUnet has the advantage of working from SHA-256 iirc., which is generally already included in the metadata of the repositories of various distributions, but I think it lacks a lot of other features and stability and ecosystem of alternative backends, compared to the BitTorrent network.

	Of course, there is no harm in including other hashes along with BTIH, to allow people to experiment with alternative backends, while always ensuring that what works works well.  Another hash that may be useful to include is the Tiger Tree Hash, which is structurally very similar to BTIH, but stronger, iirc..

	The first thing that the Guix project can do to signal interest in this architecture is to simply include the BTIH of each package in the repository metadata.  Be it in magnet URL form or not does not matter because the client can later convert that as needed.  The important thing is an authoritative statement in metadata that this version of this package has this BTIH.  Once that metadata is available, the game is on to implement swarming support, be it with Aria2c as a backend (as I recommend at least starting with) or otherwise.

	I know that this architecture works well out of first-hand experience with APT Daemon written in Python.  The only failure I had with it was lack of upstream support.  So I consider it important to first attain the upstream approval before really investing more time into this.  I seem to remember suggesting this to the Nix project many years ago and didn't get anywhere, and now I don't have the energy to try to improve upstream projects if they reject my ideas, so I'll be interested to see whether you have any success with your attempt to do the same.

	Good luck! ;-)

Kind regards,
James.
-- 
Wealth doesn't bring happiness, but poverty brings sadness.
Sent from Debian with Claws Mail, using email subaddressing as an alternative to error-prone heuristical spam filtering.
Postal: James R. Haigh, Middle Farm, Vennington, nr. Westbury, nr. Shrewsbury, Salop, SY5 9RG, Britain

[-- Attachment #1.2: JRHaigh+plz-sign4GPG@Runbox.com.asc --]
[-- Type: text/plain, Size: 3389 bytes --]

-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBGTp6SMBEADPQq1qjx0IpKf6nDxGeF9wRitXtkAxyDQ2y7DvJDfZ2L+vOjFR
InjGeD7FLK+AF5mTlsoRIg+kmE2j/1lIGj9sIybxQbcK0lxnQWMF7SXMu7e2BqPz
d/cTW8M0hdvni9FaNERL0LZkxClfHGSaQF8YKqGLuWD6q8s8YXyOK+mk+bKBjLMj
lRJGiYvSS/qCeyIRvYYPwLuSVhUybtvN/bvXyGP1Vaf39ssOyJYxSEzebsc70Doq
sS0g+47Rdl4ni24RDq/MNn5fNOy3/inP5OXxv+RgFMUwfzpx49znETMYnCGs8sbb
s7U/tTch5LTcZooBvo5HIwumnwg1RB6PjC0s59FUahoudJSDLYhZQjjZMDIjeiIT
CXjQLz2QXOdv5412c0jfWq8lLMos8tj6ACu4cTuCshljTZ+d1wG5l4H17YelgAHt
PCL6vaP4MdIpdHfeU2K0hM4EI8D+ucHpC/Hz3YGOC/vEqnX/G+NFzKK6aS469Ex/
YSbt84akd+LSW78ql7Tmw7NveWYU7wJalcMEPX0YMTRGTOsPA76KS956YCXabB6o
7xuWK9uSQzOEG8OVV9DTf8HPxc2C0dcqSDqKNyW9rOeh6nhqyHXw96W50fMx4qmp
dl1eg2Zd22Sea/uv/xyxhfI3Bq3WamaWztPMzn983qgDc50E0RDDOBe4SQARAQAB
tKtKYW1lcyBSLiBIYWlnaCAoUGxlYXNlIHNpZ24geW91ciBlbWFpbCB0byB0aGlz
IGFkZHJlc3Mgd2l0aCBHTlUgUHJpdmFjeSBHdWFyZCBvciBjb21wYXRpYmxlOyBl
bHNlIG1heSBiZSBhdXRvbWF0aWNhbGx5IHJlZ2FyZGVkIGFzIHNwYW0uKSA8SlJI
YWlnaCtwbHotc2lnbjRHUEdAUnVuYm94LmNvbT6JAlQEEwEIAD4WIQSW9hOM3nLo
AXpJ6nMHFMjblIE24wUCZOnpIwIbAwUJE0/ZAAULCQgHAgYVCAkKCwIEFgIDAQIe
AQIXgAAKCRAHFMjblIE24xwDD/4gREdchv8MpVz4DSpDkvrXJfZJknmL5IJS2DLh
mVFQ/NYgFQZa2IdcMEdE7BkwTmL0czxgjHOo9hjHP70kkSGfwsuChaNoAXMj89R4
p1Sv61FC24cbrdaT4bS2CYQqhhzbOcTozAyGlP7AaijDPc4RkrjLqepjdlEvp1tk
dFwdusDpQZ/YE8Sri4GFVPh56ZvbJW1cfoAyVaOPvnE1z5XsW6fFnRfZ1/Ll3XIL
5nNCbXrXvTJ79bNvETmznvH7Cn+4c19YicrNFjlhrf2Sv+8/q/ljts2nny3TPX1/
nh8NlgHFRYcYza7YeBK6P3twmf2tNLXIDdKZoD49rsPBVTRGkV2u5pIBpEcgKZBr
cvnT07sOEnQrdz23gEfnwzNFHpW3XOp03bF8by99p9sfdFywundb3I3d/8Xs4/DD
RZl39Xh+HSMH/Vv6jDsrH+AKp3ItO0cg44Rt8I7l/FlIfekeZ5m6QMNiCtWE4Jk5
+bQVY4j3mRFxW1vNNxv4GRpyrAHBvjqc4CFZTiFvNil/iAKnCOfy9VNv39yq2ZUB
lDFoehrf2aHifp6KsR32tw91zjIoneY1Kl8b81IKYsf2w8GlkEks1hzpSNbQqLJt
e9fCDef10NuZJv0bYjUPVjwHHLwy3R8GER3BcJkytNg/MgtlQ/xFkfC6n9qDUf4G
QLLNPbkCDQRk6ekjARAAn6+xhvuz3uxTLsF3KX9f7ADl7ekwQRKNiZsn4H7nBQ2F
Rw5ByjxTNTWbmKkKNDVBtcgkdQAs1X+9rNaG2xrFqqjhQ3ND8Q9H8jw4m2S2zX7X
T+CbZJJxQ46yuyw1m/+P9PQu4I6dkzqEcld4F1OWhJFljLpIzrOJnhQI+d8DR0ky
3tLN88/MT1ps9/gHrHj54uNO+IkGQiebBxYP2jpq+h2K1DZHLIiNSe3YUjTkRFjs
oORnBinw0KVSnltkXn4CQ1iVSCDJWgAdiI7yLmoWP2nFDuRnFLhLwymjahpr6nGO
gJLBag6JmGDM2+AxKBaBpwShsNjfygwv1joAM5YIct9iK/CTbEgzvY5Q//t9mMV9
iz2tOmOOkEfRs8h7BSzvjVD6/VAYstIPg0EXnXk9DABPy7OeLcGPyOPULvCvbF41
gu+pv3TkJ9jsfLZdaZwxVAvcxkOJ0ZkeM/Nm/c3za/rpvK7H8j8GZkXyDjST2t7/
byn3Jf9yBtr1lg4O/6nm7TU09/W48Wg7LRPxp7hK4B87eBcIAO1WHOwNb5Hcrfc4
ggW8UfLmeFVEiT1kWFW77yuw/cgz7T2TMe4wRjVDHq9pvyqvWiRzq29wtbozDWVc
VAND4+oh3OX35pUrJLIwF27NfSWCiwYu+m4Vpd+ARpv7gFBg/XUujcriYDCwLiEA
EQEAAYkCPAQYAQgAJhYhBJb2E4zecugBeknqcwcUyNuUgTbjBQJk6ekjAhsMBQkT
T9kAAAoJEAcUyNuUgTbjkwkQAKUNSgHJLpaQA5rOV5fw8ddU1VeHVeBojVCWab9L
6AopnQ98tHaO1EQlsoMJ4R3n3iZG7r0Ek1Tb2o3QmAuQg9RgiHi2+ccXCs9VTy99
pQhguCZrTj3xrNCr5D4SvicAzSuwd5ALPZq4sdEpRA0ZIaUU3FBH2W4NZ5qex3G5
VvM1hVfXnBNTPeN/rT6KwF8BpKrrVjuiB4/2NY61i7WX6/Hb9gIwZqeHMGwI/wUp
n1hiIv0xrIcfu6/7zda3yAwmHTBO3foaETZHEj7eyK7a/qH5yPEN6f5srb9TNWds
cx/yCscTZdGtBJKoYYA/dQtwal2GuOrbq8jVTdtWk/VINn+XV0Twkoer3HD7IH+/
XbPReGS1X6+Q1XmGcc0CWQ+yvsUipowtj2Z8EGefscEHLcDOdcStW9HV8pOewGlQ
PbBz9QPQTnrqhDp6ia3oKEBuxuVCOMX7meNVzBycHvqbNGV1oGJbPkTJeRcEFwO/
NSBFTlXc+euH37ZFobTnDHj7wKsbKZ+GWI7QQmcbBCeuDaFjujhQ8AgUcp4nPUYI
Op+nhMZW6LHpE0JSFtMpSYAw3Lf+6dP4hVFOXLrqs1zTaM69g0tvQJOXbF1I9hJ7
0GPnLtYhj5xx+FJBcchp3ipY0dwEVW6jqRRPJWS6xfm42JggKmpRPG+1IW4wWmEK
Q9XT
=jwvA
-----END PGP PUBLIC KEY BLOCK-----

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]