From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id UKqWALhnJ2X0agAA9RJhRA:P1 (envelope-from ) for ; Thu, 12 Oct 2023 05:27:52 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id UKqWALhnJ2X0agAA9RJhRA (envelope-from ) for ; Thu, 12 Oct 2023 05:27:52 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id AB9C76000C for ; Thu, 12 Oct 2023 05:27:51 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=CbhR7t7S; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "help-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="help-guix-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1697081271; a=rsa-sha256; cv=none; b=eJk6HUqzIinCWZ7Z2KM5HW76jBiMrpFErJKpv/ihyaSAPjlDJIkJhw+MYPE7EfUhzH4jjK QBNkGTGTdVyRl2RxEHIcJwjTSYpCpq3rroYRkhm9Fy+unMXjzTitzk1bvj7X+jCmCu/gPO nuR26FoOjPv2UPamj/KM2TFfRh+mxAXiUoACNZxhDX4DeLye1mTUnGQbZFMiKQVgPU8s1Q cKOknL6vHKt94K6KZo/7Hf4gnHICBjdQUJEbfe7tPhB8/dZWrwPA9Gjt43NGrFEGPCEG+m hFEdflUuy8Eq0vGlf66Urz+sjFhnW4M/ZJBRgat4JHLmHlgN6PEeEFuYkVT/bg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=CbhR7t7S; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "help-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="help-guix-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1697081271; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=c4xbEl1K2D/URPQcv54dX2d4omlczHBrWkL6Acc1eEw=; b=KlxaqQIXaO+EmbiNH9X1uGqG3I2pUf8gKe0fwfF3B7ofbwAMVMJH0wMZ159gVkFzdZNDXs iqYP7GRU2iAY1w1uSkxWbirhFYLrEdonYvy5QWBViC3+0rJibSASnCi6bD93pjO3xIuS9H Cjgkr8GHwl/ytRsL8U7b1gSoJ24XFMVNc5UJ+AJTV8GBDUP7dpqtkZMpZLgvdft+m6IlnA 1NgiVqg9J9CkATT/Y1X5SBcpKwcTlzead/wPwMpIqHHfOF//1O9LKFkEV2163fHJLgIxQa ueFWqo7ppZxjddLVqrj9txRjrlZvdi/uCBRNtdIvtF8VD2tBwhV5ivdAQjypgg== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qqmM1-00048N-2Q; Wed, 11 Oct 2023 23:27:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qqmLy-00048A-SH for help-guix@gnu.org; Wed, 11 Oct 2023 23:27:14 -0400 Received: from mail-wm1-x32c.google.com ([2a00:1450:4864:20::32c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qqmLx-0004ru-B3 for help-guix@gnu.org; Wed, 11 Oct 2023 23:27:14 -0400 Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-40572aeb73cso5842825e9.3 for ; Wed, 11 Oct 2023 20:27:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697081231; x=1697686031; darn=gnu.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=c4xbEl1K2D/URPQcv54dX2d4omlczHBrWkL6Acc1eEw=; b=CbhR7t7Sl3TAKQIo2Ym31XIAWysrajgomxvY06QThZjYNScg3OWuyiogGE7WgMup2Z SeRgBRHPviffW2NZ0fvzzDXK+NLrvMjjTueW0a95/pcOrhZbHr2GdKtI6EDNfKob1YzH TorfsVC1egHd9fszni4vH5Z8BFjowImn9/1moWnZoYFVN9Lx80GiD9GqjaeOugTxfbWO F73M8+lIMhm7aT+owkFmgrPe2+5OTY8GevTmamLpGgGy5CaKUQ9i+wgfiGTOatkvHuiu 8pdAsz8/gW8l0oYwReef9e69V4iE4l5NW72u57xE/2zI9WUHpqJpzE9drDLRXMGUYqSH vu6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697081231; x=1697686031; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=c4xbEl1K2D/URPQcv54dX2d4omlczHBrWkL6Acc1eEw=; b=LbzQfmcTCoa+9bobl7T0lzsxZ+QH8vW7+kXt970ZrZkDcMoszggullAt6lza0qiBX1 6wEMgRP8RczR/6MOREHpHlQHtiS/67ULjpUkhdP0yjtTR+n5Pr3dtiNjLQITNepZjs7S bnZEFeAFybX28avZN+3gxEzEPYBU3ThoUYxTx+wban2fMyH33IfG0kIZhFE9GsjSoHHC nRqFCuQXmTKPIMgyP4Piq0orMkEe8Uzw7xDXreVRH+KA1/h9lC/EAuY1ctDxzjpptt/i oQzF/DBTa5ynAgzksq/HAl472uGPVQ///wbppb2f5eqXjoaq+u7vw/g9oyuaGhAikMt7 AJIA== X-Gm-Message-State: AOJu0Yz4tuEKyW97PAav2eg1h7Xocz/8i6mDa6bG4jhy47aPZE3Tv/AA M8IPlKzZ7e4otM0ltMbrLWC/fbb293NJGCeRaMDQpPvc X-Google-Smtp-Source: AGHT+IHXYtWTCZx9PbKNneMIrR72+ICaqC0CV7Ns8ivhAotrk2JJtcR16Z/o5QdXG45EMeUQUEqeZYzlcKi4xXASz0A= X-Received: by 2002:a1c:7917:0:b0:401:d803:624f with SMTP id l23-20020a1c7917000000b00401d803624fmr20567598wme.4.1697081231078; Wed, 11 Oct 2023 20:27:11 -0700 (PDT) MIME-Version: 1.0 From: Josh Marshall Date: Wed, 11 Oct 2023 23:27:00 -0400 Message-ID: Subject: Architecture to reduce download time when pulling multiple packages To: help-guix@gnu.org Content-Type: text/plain; charset="UTF-8" Received-SPF: pass client-ip=2a00:1450:4864:20::32c; envelope-from=joshua.r.marshall.1991@gmail.com; helo=mail-wm1-x32c.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: help-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Scanner: mx1.migadu.com X-Migadu-Spam-Score: -6.53 X-Spam-Score: -6.53 X-Migadu-Queue-Id: AB9C76000C X-TUID: +kxnsByaD7E7 Presently, I am waiting until the end of global warming to finish pulling down texlive packages. I see that there are a few servers from which packages are provided. Is the following feasible as a feature to improve effective download speed? List the base information for what packages there are and where those packages are located. 1) For each package identifier, list the locations from which they may be obtained. List the top level status for each location so the first level of scheduling packages across servers. Something simple and sane. Prioritize uniquely pulling down packages and more specifically getting packages which are available from the fewest locations first so as to not bottleneck later on. 2) For each location, have a mapped value for a presently downloading a package It would be simpler to leave it at step 2 for downloading, but we can do better. If we're running into a situation where we have a package which can be sourced from multiple locations and those locations cannot be given unique packages (typically, more locations than packages) then downloading a package can be interleaved between multiple locations. 3) For each actively downloading package, list locations actively assigned to obtain data for the package alongside information to interleave data coming in from each location. If someone is willing to do a bit of mentoring, this might be a good project to work on. Any thoughts on this? Is this re-hashing old ground?