unofficial mirror of help-guix@gnu.org 
 help / color / mirror / Atom feed
* Architecture to reduce download time when pulling multiple packages
@ 2023-10-12  3:27 Josh Marshall
  2023-10-12  8:04 ` Christopher Baines
  0 siblings, 1 reply; 10+ messages in thread
From: Josh Marshall @ 2023-10-12  3:27 UTC (permalink / raw)
  To: help-guix

Presently, I am waiting until the end of global warming to finish
pulling down texlive packages.  I see that there are a few servers
from which packages are provided.  Is the following feasible as a
feature to improve effective download speed?

List the base information for what packages there are and where those
packages are located.
1) For each package identifier, list the locations from which they may
be obtained.

List the top level status for each location so the first level of
scheduling packages across servers.  Something simple and sane.
Prioritize uniquely pulling down packages and more specifically
getting packages which are available from the fewest locations first
so as to not bottleneck later on.
2) For each location, have a mapped value for a presently downloading a package

It would be simpler to leave it at step 2 for downloading, but we can
do better.  If we're running into a situation where we have a package
which can be sourced from multiple locations and those locations
cannot be given unique packages (typically, more locations than
packages) then downloading a package can be interleaved between
multiple locations.
3) For each actively downloading package, list locations actively
assigned to obtain data for the package alongside information to
interleave data coming in from each location.

If someone is willing to do a bit of mentoring, this might be a good
project to work on.  Any thoughts on this?  Is this re-hashing old
ground?


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2023-11-07 17:30 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-12  3:27 Architecture to reduce download time when pulling multiple packages Josh Marshall
2023-10-12  8:04 ` Christopher Baines
2023-10-12 18:09   ` Josh Marshall
2023-10-13 10:14     ` Christopher Baines
2023-10-13 16:36       ` Josh Marshall
2023-10-13 18:05         ` Architecture to reduce download time when pulling multiple packages – historic success with magnet URLs, BTIHs, & Aria2c! JRHaigh+ML.GNU.Guix--- via
2023-10-15 18:21           ` Josh Marshall
2023-10-18  1:44             ` Josh Marshall
2023-10-18  8:11             ` Christopher Baines
2023-11-07 17:29               ` JRHaigh+ML.GNU.Guix--- via

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).