From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 8KpBGWwMFl+LAgAA0tVLHw (envelope-from ) for ; Mon, 20 Jul 2020 21:28:12 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 6EoIFWwMFl+3aQAA1q6Kng (envelope-from ) for ; Mon, 20 Jul 2020 21:28:12 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0BBCD9401CF for ; Mon, 20 Jul 2020 21:28:12 +0000 (UTC) Received: from localhost ([::1]:47504 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jxdKV-0002rl-1p for larch@yhetil.org; Mon, 20 Jul 2020 17:28:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48852) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jxdKN-0002py-8V for bug-guix@gnu.org; Mon, 20 Jul 2020 17:28:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:53230) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jxdKM-0003Zq-UP for bug-guix@gnu.org; Mon, 20 Jul 2020 17:28:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jxdKM-0004uP-QO for bug-guix@gnu.org; Mon, 20 Jul 2020 17:28:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#42162: Recovering source tarballs Resent-From: zimoun Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 20 Jul 2020 21:28:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 42162 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Christopher Baines , Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 42162-submit@debbugs.gnu.org id=B42162.159528046218822 (code B ref 42162); Mon, 20 Jul 2020 21:28:02 +0000 Received: (at 42162) by debbugs.gnu.org; 20 Jul 2020 21:27:42 +0000 Received: from localhost ([127.0.0.1]:36539 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jxdK1-0004tW-U5 for submit@debbugs.gnu.org; Mon, 20 Jul 2020 17:27:42 -0400 Received: from mail-wr1-f50.google.com ([209.85.221.50]:38265) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jxdJz-0004tI-IE for 42162@debbugs.gnu.org; Mon, 20 Jul 2020 17:27:40 -0400 Received: by mail-wr1-f50.google.com with SMTP id a14so4352625wra.5 for <42162@debbugs.gnu.org>; Mon, 20 Jul 2020 14:27:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=2+SnLNwqdv1rxWYvVaEUhogUq7I92bNwB0b3aUGnnZs=; b=HCQ1Z4wBrBN7aA1e9DM52gB+xgK+FfkWbvo7Z9Y4gVsmH3OormHdu+LgCzrpE3MSvz rEaYgij0anUzafOeq2cUP7DVBfimByVXXdeTYAvMi+pZHHuNj/imxD+tdzAw3QMnNs+K N5XNUU9srZrEjPM19GPUZrSXLtByshUeP7cm8nH6lAsdHJiFOno/ArCqpG6GoF1YKs4h XaBNfob3mZYZpGefPNKv0iDb61ejHoAlkO+0APyvpWAzwutAtgRlQULKNrBzSTyV97/v G0lOnDnw1PLeRGf8K8ES2Mh0Namo13fJ3wwXATyJ4xcgvAsGg1ln2iVwVEJT3M8ze0hm GXbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=2+SnLNwqdv1rxWYvVaEUhogUq7I92bNwB0b3aUGnnZs=; b=P7rcW1ghqLwTD0TVpwwXD0VbW9ZLZi/uTUYMh97IcV1Rprq71F1/hODqXCMmqC3pO+ q+NffoAt60d+byr4L0bTyKuHx8KSgIo+MDSpUyILFpQquxoZlC9PKsks5F+Bh/Q0JoEP KzYLKqP1wL51+qKFn8gU1j2zKJw8QXu2RwvZuS/BI+LmaORWoBfsTYd4ukyCGSvGQGti SlvX31XFtnp0duGK0HO4rOJ6ZxRPU5k+EbT1xM030udwf10ZGSdk7JldFekhQ2enbt+i R+10HhAOJevTsQavFo8IY/CJzWFkHuk5ywO2SSEyQW11AYSnIfjBu5OrjRs+mhJ1pKlY 0nFQ== X-Gm-Message-State: AOAM531TgEa9nv1hgt94M3Q4tL4F0okEhfBgAmn9IqS13AMf+f555zo/ QZlcBfuyED928J89PMOKNl8= X-Google-Smtp-Source: ABdhPJyJK0i2JsiKiAXirf/MP1XnruW3q/ZCFjUMe16X4+96sZTwbsIyx/y//HOO/THdvnet9OhscQ== X-Received: by 2002:adf:dfd1:: with SMTP id q17mr22565951wrn.94.1595280453723; Mon, 20 Jul 2020 14:27:33 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id p25sm489073wma.39.2020.07.20.14.27.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 14:27:33 -0700 (PDT) From: zimoun In-Reply-To: <87a703jk78.fsf@cbaines.net> References: <87mu4iv0gc.fsf@inria.fr> <86h7uq8fmk.fsf@gmail.com> <87d05etero.fsf@gnu.org> <87r1tit5j6.fsf_-_@gnu.org> <87a703jk78.fsf@cbaines.net> Date: Mon, 20 Jul 2020 23:27:32 +0200 Message-ID: <865zahev23.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 42162@debbugs.gnu.org, Maurice =?UTF-8?Q?Br=C3=A9mond?= Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (rsa verify failed) header.d=gmail.com header.s=20161025 header.b=HCQ1Z4wB; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Spam-Score: 0.09 X-TUID: mXyE7xqcYYFs Hi Chris, On Mon, 13 Jul 2020 at 20:20, Christopher Baines wrote: > Going forward, being methodical as a project about storing the tarballs > and source material for the packages is probalby the way to ensure it's > available for the future. I'm not sure the data storage cost is > significant, the cost of doing this is probably in working out what to > store, doing so in a redundant manor, and making the data available. A really rough estimate is 120KB on average* per raw tarball. So if we consider 14000 packages and 70% of them are url-fetch, then it leads to 14k*0.7*120K= 1.2GB; which is not significant. Moreover, if we extrapolate the numbers, between v1.0.0 and now it is 23 commits per day modifying gnu/packages/ so 0.7*23*120K*365= 700MB per year. However, the 120KB of metadata to re-assemble the tarball have to be compared to the 712KB of raw compressed tarball; both about the hello package. *based on the hello package. And it depends on the number of files in the tarball. File stored not compressed: plain sexp. Therefore, in addition to what to store, redundancy and availability, one question is how to store? Git-repo? SQL database? etc. > The Guix Data Service knows about fixed output derivations, so it might > be possible to backfill such a store by just attempting to build those > derivations. It might also be possible to use the Guix Data Service to > work out what's available, and what tarballs are missing. Missing from where? The substitutes farm or SWH? Cheers, simon