From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id AF3IJV+PzWE/OgAAgWs5BA (envelope-from ) for ; Thu, 30 Dec 2021 11:52:15 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id UANmHl+PzWHdygAAG6o9tA (envelope-from ) for ; Thu, 30 Dec 2021 11:52:15 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EEB732BB6B for ; Thu, 30 Dec 2021 11:52:11 +0100 (CET) Received: from localhost ([::1]:50698 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n2t2Y-0004nG-NQ for larch@yhetil.org; Thu, 30 Dec 2021 05:52:10 -0500 Received: from eggs.gnu.org ([209.51.188.92]:32834) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n2t2Q-0004n8-1w for bug-guix@gnu.org; Thu, 30 Dec 2021 05:52:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:39944) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n2t2P-0000O2-OX for bug-guix@gnu.org; Thu, 30 Dec 2021 05:52:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n2t2P-0002FA-N9 for bug-guix@gnu.org; Thu, 30 Dec 2021 05:52:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#51787: Disk performance on ci.guix.gnu.org Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 30 Dec 2021 10:52:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51787 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Mathieu Othacehe Received: via spool by 51787-submit@debbugs.gnu.org id=B51787.16408614898565 (code B ref 51787); Thu, 30 Dec 2021 10:52:01 +0000 Received: (at 51787) by debbugs.gnu.org; 30 Dec 2021 10:51:29 +0000 Received: from localhost ([127.0.0.1]:51490 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n2t1s-0002E5-Kt for submit@debbugs.gnu.org; Thu, 30 Dec 2021 05:51:28 -0500 Received: from sender3-of-o51.zoho.com ([136.143.184.51]:21190) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n2t1q-0002Dx-7A for 51787@debbugs.gnu.org; Thu, 30 Dec 2021 05:51:27 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1640861479; cv=none; d=zohomail.com; s=zohoarc; b=gwPGqev3qu6m1/3FCeQg1Mj+gGvm/8ltXj+yOyh8bAdBtROHxfUjOO6yhktpk43b4VVxvKL5IyFVqWYA9ScyWZRsgHctk+TOaiBKlQRl7HB61iWXKRbtk/pP3H4tsd2ClNSs2WFZxlljF316yHuX5ZjJ6TflHX1I6fiiFUS+Ykk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1640861479; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=I1qXWDnDytSB/eXEp3XZadeU6+Gn52FI6FDMfUWW0rM=; b=BGH9zcVaKIDi15jGSiEMKsrKX3e0Dml0Qyi7+I0OcQ/NyXY1lCIpIYnCbkjxAQAKnjnYP31Hmfdq9AekuWnCDVCDvH+c+SLqUI1WJ3YQqIfNcs00l6xG5PLGHcqkQ708ykyHnCcsO+hjGADKDpbEH0HruVvlsQXM6KLUeUX7Zko= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1640861479; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=I1qXWDnDytSB/eXEp3XZadeU6+Gn52FI6FDMfUWW0rM=; b=S1lDDHcmL35Q5bxzsx2vmZG4HmunQauar6FEHqwS7nDuD+AWeRLVOF+ECVuCWrwQ smve3hMEkkrylwVhu6GpkYWtgKGNNYFNLgtTFyRWAL9kqb0pFB7x+1vjVilk7Vlqj79 MDSTJq4kou4Ru8NDv9uatyTrWHiJx7WL8EcSV+UY= Received: from localhost (p54ad4d28.dip0.t-ipconnect.de [84.173.77.40]) by mx.zohomail.com with SMTPS id 1640861478017408.4799745377054; Thu, 30 Dec 2021 02:51:18 -0800 (PST) References: <875yrjpi1y.fsf@elephly.net> <87o85bjjpm.fsf@gnu.org> <871r27p5jq.fsf@elephly.net> <87a6gv3pue.fsf@gnu.org> <87v8zhn9m1.fsf@elephly.net> <87wnjsjpdh.fsf@elephly.net> <87fsqfd9xm.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 27.2 From: Ricardo Wurmus Date: Thu, 30 Dec 2021 11:44:12 +0100 In-reply-to: <87fsqfd9xm.fsf@gnu.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <87wnjmgycd.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 51787@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1640861535; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=I1qXWDnDytSB/eXEp3XZadeU6+Gn52FI6FDMfUWW0rM=; b=QWJs0IfZhecz05Ah7kNdoaJQmMtQtryhGN3GY3DKNs9s/SZqrF/6I0NOcYh9RgVrO8UKX1 s1J5XReIiH8gACINUhr7o459mbMkQcu2rfIqg1L1paTli2s4koiUixkbGlOY5N9BQHAReg 1d+BwXPQy0XV3keQRCgqu3y/ZQTnQm43Se/ZYHUsgCGmxej6pQ9Laq0ebMXNfI+vgwq0K+ SIZNmSLDjHPqBDgenRYQtOLYMjUEPC36UMQNpcRY831mDUtd7E6GIyf16HzufNcifSAp3Y 7Vqf90lPu96WFb/X52zqXu0fMtmyDPNSd425zfhGzwyFH6syDVBxObzImaV52A== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1640861535; a=rsa-sha256; cv=fail; b=L+Vi5OhyRaJP2nxk2U2E1S2yNgr2CeF4kf49eh1X/jtUm1MBVkcimskJRS2ZqnateVjl3A om3NiF2ELYVoviEINILtQjRtAU9tfNriwG6LDe5uNBWATBIEujTWv/vy4cAlvV2XSZt6Rh 3ju54hKpUTYy66wQRXzwdAxqdji+dN58k6I8On1lVQWLtD4/t6/TW2KUw4A/sI8eyIQ5vo +m5Au/0aQybTfZnDakGK/Bkhb8BvIMJQWIPIRkGSinQmGICW9lSqXERshHZM0Sh5VC4SOA pHTGquCnh2HLrCYHD1WcyyNeHIW4rz17n8UNZ3QR7PkG80LAdw+kfbdkPiWb1Q== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=S1lDDHcm; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.07 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=S1lDDHcm; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: EEB732BB6B X-Spam-Score: -1.07 X-Migadu-Scanner: scn1.migadu.com X-TUID: pJy0+xNBCFC8 Mathieu Othacehe writes: > Hello Ricardo, > >> Should we attempt to overwrite /gnu/store and rely exclusively on >> substitutes from the cache? > > Yes, I don't see any other options. Before that, what might be nice > could be: > > 1. Ensure that all Berlin /var/cache/guix/publish directory is > synchronized on Bordeaux. We are now at 117G out of X. We could then > start a publish server on Bordeaux. As Bordeaux is already part of the > default substitute servers list, the transition could be smooth I guess. I had the SAN slice extended from 5TB to 10TB. This is now also full (at 9.2TB due to SAN configuration). I suggest doing the rsync to Bordeaux from /mnt_test/var/cache/guix/publish instead of the much slower /var/cache/guix/publish. It doesn=E2=80=99t hold *all* files, but 9= +TB should be enough to fuel the transfer to Bordeaux for a while. > Regarding the /gnu/store re-creation, I wonder how can we do it without > reinstalling completely Berlin. Maybe we could save the system store > closure somewhere and restore it on the shining new file-system? I don=E2=80=99t know. I would want to take a copy of the root file system = as a backup of state (like the Lets Encrypt certs), and copy the closure of the current operating system configuration somewhere. We could copy it to a dedicated build node (after stopping the GC cron job) and set it up as an internal substitute server. Then =E2=80=9Cguix system init=E2=80=9D = while fetching the substitutes from that server. But I guess we=E2=80=99d have to boot the installer image anyway so that we= can safely erase /gnu/store, or else we=E2=80=99d erase files that are currentl= y in use. --=20 Ricardo