From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id wIdvFim0DF9xbwAA0tVLHw (envelope-from ) for ; Mon, 13 Jul 2020 19:21:13 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id wMA/Eim0DF9mXwAAB5/wlQ (envelope-from ) for ; Mon, 13 Jul 2020 19:21:13 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D26289400B1 for ; Mon, 13 Jul 2020 19:21:12 +0000 (UTC) Received: from localhost ([::1]:34666 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jv40k-0003Xv-Nh for larch@yhetil.org; Mon, 13 Jul 2020 15:21:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42680) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jv40c-0003Vj-GJ for bug-guix@gnu.org; Mon, 13 Jul 2020 15:21:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:37778) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jv40c-0006RQ-5d for bug-guix@gnu.org; Mon, 13 Jul 2020 15:21:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jv40c-0001i2-22 for bug-guix@gnu.org; Mon, 13 Jul 2020 15:21:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#42162: Recovering source tarballs Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 13 Jul 2020 19:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 42162 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 42162-submit@debbugs.gnu.org id=B42162.15946680356516 (code B ref 42162); Mon, 13 Jul 2020 19:21:02 +0000 Received: (at 42162) by debbugs.gnu.org; 13 Jul 2020 19:20:35 +0000 Received: from localhost ([127.0.0.1]:49324 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jv40A-0001h1-PF for submit@debbugs.gnu.org; Mon, 13 Jul 2020 15:20:35 -0400 Received: from mira.cbaines.net ([212.71.252.8]:39750) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jv408-0001gt-VD for 42162@debbugs.gnu.org; Mon, 13 Jul 2020 15:20:33 -0400 Received: from localhost (unknown [46.237.175.173]) by mira.cbaines.net (Postfix) with ESMTPSA id C870727BBE1; Mon, 13 Jul 2020 20:20:31 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id b0041d41; Mon, 13 Jul 2020 19:20:29 +0000 (UTC) References: <87mu4iv0gc.fsf@inria.fr> <86h7uq8fmk.fsf@gmail.com> <87d05etero.fsf@gnu.org> <87r1tit5j6.fsf_-_@gnu.org> User-agent: mu4e 1.4.10; emacs 26.3 From: Christopher Baines In-reply-to: <87r1tit5j6.fsf_-_@gnu.org> Date: Mon, 13 Jul 2020 20:20:27 +0100 Message-ID: <87a703jk78.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 42162@debbugs.gnu.org, Maurice =?UTF-8?Q?Br=C3=A9mond?= Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Spam-Score: 0.19 X-TUID: /yZlOZo+5mAr --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi, > > Ludovic Court=C3=A8s skribis: > >> There=E2=80=99s this other discussion you mentioned, which I hope will h= ave a >> positive outcome: >> >> https://forge.softwareheritage.org/T2430 > > This discussion as well as discussions on #swh-devel have made it clear > that SWH will not archive raw tarballs, at least not in the foreseeable > future. Instead, it will keep archiving the contents of tarballs, as it > has always done=E2=80=94that=E2=80=99s already a huge service. > > Not storing raw tarballs makes sense from an engineering perspective, > but it does mean that we cannot rely on SWH as a content-addressed > mirror for tarballs. (In fact, some raw tarballs are available on SWH, > but that=E2=80=99s mostly =E2=80=9Cby chance=E2=80=9D, for instance becau= se they appear as-is in > a Git repo that was ingested.) In fact this is one of the challenges > mentioned in > . > > So we need a solution for now (and quite urgently), and a solution for > the future. > > For the now, since 70% of our packages use =E2=80=98url-fetch=E2=80=99, w= e need to be > able to fetch or to reconstruct tarballs. There=E2=80=99s no way around = it. > > In the short term, we should arrange so that the build farm keeps GC > roots on source tarballs for an indefinite amount of time. Cuirass > jobset? Mcron job to preserve GC roots? Ideas? Going forward, being methodical as a project about storing the tarballs and source material for the packages is probalby the way to ensure it's available for the future. I'm not sure the data storage cost is significant, the cost of doing this is probably in working out what to store, doing so in a redundant manor, and making the data available. The Guix Data Service knows about fixed output derivations, so it might be possible to backfill such a store by just attempting to build those derivations. It might also be possible to use the Guix Data Service to work out what's available, and what tarballs are missing. Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl8Ms/tfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XfAtw/6AtEyqRcimef5NTFchcAigC6fT6DJLcGnyJNUXlfZn6nHU9ao/ev33D5d MFfKl1YljKf+fA848fZSIe0eBERbkZ+D1oed6SD6Xx8fG9ekCSgGtbmysNEcDDKK qO5kg/QUbKYODpRW8iZIDMPUQZ0yNQu9KQdvVKIhHIZJnSGNt2XVjRdoCkW+H19m QVPVdgqZIarkZctOzPegA8FFEi8O/GO7gK4gbizewecgsl1qL0yWBDyUJ9tsWeAH +EsVykk91y9tHDPfQYfKqik7A0WrK75oeNOqs5QtEqRPjcMzwsDkIO13e5Y3Z5Yl M7zTs7R/OLSyiSlT5z/1S5RrbMyMMryt0S4uvqjZfFDtgaOHxhVhBg/1kya/H5v1 cB3jq8WpvL6sDYFbSqI9vWPJnQDq5EpIvI16Ri0ygnMAffiz6hhtdn/pCGV7GG5U 7H6ED7gz5FB8YovGED1C9l8dh7h3Hi+1P+JL3KheJyF5bU829wqL9r2l5sOprad0 PEsq52RCwPBuNu8agTbobICimqFnp3B5wySDNEvkXZ4FFlMR6ZdW0BjBnLF0ZRU4 v8FCf+w81lAIksF9UWusZTzb++aMPXsdlHfelyWtOUi5mc1GMNRfCIW/VLIYyZIP aqVPHoFkTWb6q6XK5tjC302Di/BD9qDEr5g9qFU16Yeq7ywcAjs= =hgnz -----END PGP SIGNATURE----- --=-=-=--