From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id OHdUEG8/aGFXIAEAgWs5BA (envelope-from ) for ; Thu, 14 Oct 2021 16:32:15 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 2G/eC28/aGEQOQAAbx9fmQ (envelope-from ) for ; Thu, 14 Oct 2021 14:32:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 927D21590A for ; Thu, 14 Oct 2021 16:32:14 +0200 (CEST) Received: from localhost ([::1]:53680 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mb1mH-0007wc-JJ for larch@yhetil.org; Thu, 14 Oct 2021 10:32:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55108) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mb1m3-0007uU-Br for guix-devel@gnu.org; Thu, 14 Oct 2021 10:31:59 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:49590) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mb1m3-0003Ye-3E for guix-devel@gnu.org; Thu, 14 Oct 2021 10:31:59 -0400 Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:63780 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mb1m2-0002dC-M1 for guix-devel@gnu.org; Thu, 14 Oct 2021 10:31:58 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Subject: Re: Disarchive update References: <87r1cu1pj5.fsf@inria.fr> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 23 =?utf-8?Q?Vend=C3=A9miaire?= an 230 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 14 Oct 2021 16:31:56 +0200 In-Reply-To: <87r1cu1pj5.fsf@inria.fr> ("Ludovic =?utf-8?Q?Court=C3=A8s=22?= =?utf-8?Q?'s?= message of "Sat, 09 Oct 2021 12:05:50 +0200") Message-ID: <87lf2voexv.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1634221934; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=+CusdSxE/dCb9QXjBmjfDPhRJR7xW2CEz0v4oKH+B6s=; b=EOG7HvJf5OpVdv0El3fa1ATtywFhi5m582ktVnK5c0KHPRQaaDXzLVs/1gC+bOg5TYorDI dFZb/JHoJKmQ12qy8rFCilQrsqDtPtpbzqgXRI7+GZLBiQ8AfZzTGmQUFO6FV0fp58H7RI KtaeCrbcTsUTWeW6VWB8Wk4iGzt1pOHW0DkfKfXSWd1SG8TVIs4Hi+cjSdIhy3eXEJFYi2 0ewiz9tJY2vBnB7WUW6wTimaFtZDFc4GknWZdLUUAH5UPlKbOyJyklEaN28xUtSMCXw7RY DLt2Qt/681kfTzhHN0Bz8Wa+0hATrphMZgDkb//95uE4XZq1Gb7PxOVr+Bioqw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1634221934; a=rsa-sha256; cv=none; b=CqkgR7UZMpIHUno+J2ZJxUfdSTiEi6NG+nVMp+eWkEGa60Qi8QhZBR+zfAh7rb40NbYWaB /3pEJAQuCUCjW87W7CHcy9hKpPqc/MR5LS+Vqj3mkm7mucyjD/AAGhEk4M3aVXn9+nf8aL +hpfQEGH7Ng3BrlyYMOg1UW3C0ZH2tFkzj/ZYhyp3NnWtpFfZtug1fz/h0+zK1N4vIfUf5 CbeI3yPmQcOtr+JZNKn22lVj4CDfSJy6fOjCMacZ6zJpYYBcSFEmE9HRwxl/udkHGK4M2Y ShraGZOFKtcoKF7oNN2E6GGq/uRkCcMmd9Qyc7YC4xunT199D9yZ8IFnLaT5pg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -2.91 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 927D21590A X-Spam-Score: -2.91 X-Migadu-Scanner: scn0.migadu.com X-TUID: erKZw4PV/Wq+ Hello Guix! Ludovic Court=C3=A8s skribis: > This job is disassembling all the .tar.gz files packages refer to, using > the recently-added =E2=80=98etc/disarchive-manifest.scm=E2=80=99 file: > > https://ci.guix.gnu.org/jobset/disarchive [...] > A plan we can already deploy would be: > > 1. Add the disarchive.guix.gnu.org DNS entry, pointing to berlin. Done: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=3Ddf9e9= b7f51abceb5999aabc9a7b71396600cffa4 https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=3D12195= 160432871b80d0e1eac996a9aa7d8500697 Sample URLs: https://disarchive.guix.gnu.org/sha256/53cf3e14c71f3a149f29d13a0da64120b3= c1d3334fba39c4af3e520be053982a https://disarchive.guix.gnu.org/sha256/39052f59ff474a4a69cefc25cf3caf8429= 400889deba010ee6403ca188f8b311 https://disarchive.guix.gnu.org/sha256/03a71d53055bd9ec528d55e07afaf15c09= dec9856cba734904bfd05acbc6cf12 Aren=E2=80=99t those Disarchive sexps really cute? :-) > 2. On berlin, add an mcron job that periodically copies the output of > the latest =E2=80=9Cdisarchive-collection=E2=80=9D build to a direct= ory, say > /srv/disarchive. Thus, the database would accumulate tarball > metadata over time. First, there=E2=80=99s a script to populate the database; it copies files f= rom the latest successful =E2=80=9Cdisarchive-collection=E2=80=9D build to a sp= ecified directory, gzipping them on their way. It=E2=80=99s atomic, so the directo= ry in question can be directly served by nginx or similar: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=3Dfb83b= 3d8de189c6d6c33c4cdc2ebabf6eae1463e If you want to try it at home, just run: ./sync-disarchive-db.scm /tmp/db It=E2=80=99s pretty fast! The output is only 70=C2=A0MiB, now that individ= ual files are gzipped. Then there=E2=80=99s the mcron job that runs it once a day on berlin: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=3D27dc7= 4fbe33a9d929b37994e825dc202385f87c0 We could run it as well on bayfront so we have a backup. > 3. Add an nginx route so that /srv/disarchive is served at > https://disarchive.guix.gnu.org. Done here: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=3D9ffb2= db81a2fbee67b99c76217be874ec0fd6bde > 4. Add disarchive.guix.gnu.org to (guix download). Done: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=3Df9a506aa6a5aaeb2c= 06c97d5b663d01d2103db69 As I was once again modifying files by hand to test the download fallback mechanisms, I figured we could just as well add a variable to enable testing, which is what I did here: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=3Dc4a7aa82e25503133= a1bd33148d17968c899a5f5 So you can do, say: GUIX_DOWNLOAD_FALLBACK_TEST=3Ddisarchive-mirrors guix build -S r-ebimage = --check or: GUIX_DOWNLOAD_FALLBACK_TEST=3Dcontent-addressed-mirrors guix build -S r-e= bimage --check to check whether these fallback mechanisms work as expected. (They do, but I=E2=80=99ll update the =E2=80=98guix=E2=80=99 package because the curr= ent one has a bug that breaks the Disarchive/SWH fallback.) I think we=E2=80=99re making progress! :-) Thanks, Ludo=E2=80=99.