From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:403:4ea1::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 6MY/OQuax2ErggAAgWs5BA (envelope-from ) for ; Sat, 25 Dec 2021 23:24:11 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id 2Po/Mwuax2ERcgEAG6o9tA (envelope-from ) for ; Sat, 25 Dec 2021 23:24:11 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6DC8DD604 for ; Sat, 25 Dec 2021 23:24:11 +0100 (CET) Received: from localhost ([::1]:44040 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n1FSU-00080a-9p for larch@yhetil.org; Sat, 25 Dec 2021 17:24:10 -0500 Received: from eggs.gnu.org ([209.51.188.92]:37398) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n1FSM-00080Q-8y for bug-guix@gnu.org; Sat, 25 Dec 2021 17:24:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:56688) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n1FSL-0003aN-Vd for bug-guix@gnu.org; Sat, 25 Dec 2021 17:24:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n1FSL-0008Cb-LU for bug-guix@gnu.org; Sat, 25 Dec 2021 17:24:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#51787: Disk performance on ci.guix.gnu.org Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 25 Dec 2021 22:24:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51787 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Mathieu Othacehe Received: via spool by 51787-submit@debbugs.gnu.org id=B51787.164047100431476 (code B ref 51787); Sat, 25 Dec 2021 22:24:01 +0000 Received: (at 51787) by debbugs.gnu.org; 25 Dec 2021 22:23:24 +0000 Received: from localhost ([127.0.0.1]:40001 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n1FRj-0008Bc-Px for submit@debbugs.gnu.org; Sat, 25 Dec 2021 17:23:24 -0500 Received: from sender4-of-o52.zoho.com ([136.143.188.52]:21282) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n1FRf-0008BQ-1o for 51787@debbugs.gnu.org; Sat, 25 Dec 2021 17:23:22 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1640470993; cv=none; d=zohomail.com; s=zohoarc; b=DT/OMetXxde1isvecV4fU0rPqKKa8PGLsESvTYC9p2uNHSSshHlBrUI4B0QqwO2gumLSI91S0DhJa/PCQm+KpKAB54VdexcT+3k8M2sZWn4cEOaQ39nduOQ6RcDguzVzTsbwXGZubjoXbmLhW4OBTheF3BE7bPoXpQbYNzm1cy8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1640470993; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=VY+1RNnbRdxSzBwvgxcH39KfMkZE/8/MXaRN2E99tno=; b=PcgyoR8S2hrNYNLonDRGzJRpbPc0vOH6QkDQBGYkEyj+Ap5dX401xxaHrxCbLkHJNCqbSCkCCTQ111mNHzyoDSqLZUxIQV1csbH1RgdqfZOjYeRIB7Z9R7eRYBTwaedDcGKJEM37Ej82ZWTCNaxRQSAEWrnCjrQHkI0vPKOep2s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1640470993; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=VY+1RNnbRdxSzBwvgxcH39KfMkZE/8/MXaRN2E99tno=; b=FJJldJl9Jcu0ZRzKjFwYTaZD8p6Y81JveYnmXIsbtI8MWqPh1u8IUDnMlN/CC4QL oRn8/pgHNl7rGQX3572P2Q+MhMrCLsotxpbLc5cDedePRPgecBoqnZSbZgEkUg6UuiH XeykWOSKAWLYum9UwzTNIteuNQ7qcfL7/lpyu6rk= Received: from localhost (p4fd5a901.dip0.t-ipconnect.de [79.213.169.1]) by mx.zohomail.com with SMTPS id 1640470990202742.1489485710027; Sat, 25 Dec 2021 14:23:10 -0800 (PST) References: <875yrjpi1y.fsf@elephly.net> <87o85bjjpm.fsf@gnu.org> <871r27p5jq.fsf@elephly.net> <87a6gv3pue.fsf@gnu.org> <87v8zhn9m1.fsf@elephly.net> User-agent: mu4e 1.6.10; emacs 27.2 From: Ricardo Wurmus Date: Sat, 25 Dec 2021 23:19:23 +0100 In-reply-to: <87v8zhn9m1.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <87wnjsjpdh.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 51787@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1640471051; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=VY+1RNnbRdxSzBwvgxcH39KfMkZE/8/MXaRN2E99tno=; b=iIWsYU4qGe5244bjocQsgx1z8deMP5y2HiAzPCewCfZYqy2UAnI+JazBUViYWUFTTaQL0h GZCjj2HPHex4a6P9MjnuEYQj6Ma3zZW74pqeR7F8DGrqhesOdZm+63sn3pBFlHVHiZ6vop un0bxBkMj6N1/p6CEDTL4xPmWDCdhqNxNpsix1epY3d6qQ6ISzdw9pGa9iso8ccZ4qb8Ik b0jCNkN2JPNwObaYMObNMJiw9GQUVe8hCl6MFg5QNrcYc/FMMWPLfQ+HTQqTUDGp60mTTk UqRoyXYjunz9xWHjum4yr+V36Ylph0MHH09iNdv57Fk8J7uQmxThwnAVXAYHaA== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1640471051; a=rsa-sha256; cv=fail; b=GX36Nvw768NBXuPM7XRn24KMVle83d6+fjCb2XGadktWluAgxUfUun3o+r5JY3kS/6/KkI NZ+p0m3RzLGg3Ew5k+LRl3ZZO9VpPws8nkkETMuodISswi9Ra2kIBI+U0Fa60ynTJeClLm f9KqDolYyoguU3Kc9JnLOyN8DFjd27WUaSrbXCiLbR6VQKMB02SqB/7q9/J1W2Cunkpe6E 6Ec4eoN3v020Er4bAuR+1EFh9//6ukq7XRL7/hip5a3GaJ2/iG/rcaeejTYUdO8QosjrS2 j1w2OsVdWw7U9km+GmB84SSobTEaVnAo5N/09q/ex3CA6/SxYC5uUgJPk1qoyA== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=FJJldJl9; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.06 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=FJJldJl9; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 6DC8DD604 X-Spam-Score: -1.06 X-Migadu-Scanner: scn1.migadu.com X-TUID: UqcTuK556QUP Ricardo Wurmus writes: > Today we discovered a few more things and discussed them on IRC. Here=E2= =80=99s > a summary. > > /var/cache sits on the same storage as /gnu. We mounted the 5TB ext4 > file system that=E2=80=99s hosted by the SAN at /mnt_test and started cop= ying > over /var/cache to /mnt_test/var/cache. Transfer speed was considerably > faster (not *great*, but reasonably fast) than the copy of > /gnu/store/trash to the same target. Turns out that space on the SAN is insufficient for a full copy of /var/cache. We=E2=80=99ve hit ENOSPC after 4.2TB. The SAN enforces some headroom to remain free, so it denies us full access to the 5TB slice. Bummer. I guess we=E2=80=99ll have to wait for the SAN extension some time early 20= 22 before we can relocate the substitutes cache. Should we attempt to overwrite /gnu/store and rely exclusively on substitutes from the cache? No matter how we look at it, the huge store is a performance problem for us. Today I had to kill =E2=80=99guix gc=E2=80=99 after the GC lock had be= en held for about 24 hours. We will keep having this problem. --=20 Ricardo