From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 4ICeFNQQwmFt8AAAgWs5BA (envelope-from ) for ; Tue, 21 Dec 2021 18:37:24 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id yKFJENQQwmFpcwAAbx9fmQ (envelope-from ) for ; Tue, 21 Dec 2021 17:37:24 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0E0BF2AD0 for ; Tue, 21 Dec 2021 18:37:24 +0100 (CET) Received: from localhost ([::1]:54584 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mzj4k-0002Uh-2w for larch@yhetil.org; Tue, 21 Dec 2021 12:37:23 -0500 Received: from eggs.gnu.org ([209.51.188.92]:38558) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mzj4Q-0002Tt-MT for bug-guix@gnu.org; Tue, 21 Dec 2021 12:37:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:43935) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mzj4Q-0005R3-32 for bug-guix@gnu.org; Tue, 21 Dec 2021 12:37:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mzj4Q-0005ju-1q for bug-guix@gnu.org; Tue, 21 Dec 2021 12:37:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#51787: Disk performance on ci.guix.gnu.org Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 21 Dec 2021 17:37:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51787 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Mathieu Othacehe Received: via spool by 51787-submit@debbugs.gnu.org id=B51787.164010820321995 (code B ref 51787); Tue, 21 Dec 2021 17:37:02 +0000 Received: (at 51787) by debbugs.gnu.org; 21 Dec 2021 17:36:43 +0000 Received: from localhost ([127.0.0.1]:55472 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzj46-0005ig-Pu for submit@debbugs.gnu.org; Tue, 21 Dec 2021 12:36:43 -0500 Received: from sender3-of-o51.zoho.com ([136.143.184.51]:21738) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzj45-0005iU-Ii for 51787@debbugs.gnu.org; Tue, 21 Dec 2021 12:36:42 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1640108191; cv=none; d=zohomail.com; s=zohoarc; b=JDqgyzsAypaq9CMTlFtILwmZ30LLNHb20sJHGPab7Qpu8Lm2oJGc1P6t4XAQD8XieoJQz5IbvAJ4ra9mbRWHlfLAc17K0yX/zdZ3E7vLIT3s8/FEATEhU0L8SLuWA0sS5yFP5TiZ0yzsB8B9rK9u0K/mT77OWw9MqsJzvo1NXa4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1640108191; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=jH1Dkp4lD2bICXPSZf+fwlKJ828YTBcl8g3NEzCp1q0=; b=jgubDGrk5DkNle81PDiUxbQWfJIlvJltupTNvAmnlkQgFsokvBECnmfHStkhdInNuVuwGahATwHwKXyDzPx1qcoRYcUX5Gy/AUFelDAr4brQlWL0hSg/yRiF+HKaCIlF1CFqlD2Ks+k5OL8wxb9eYIaeiL/IXZ9Z+0NCeCIv/3o= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1640108191; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=jH1Dkp4lD2bICXPSZf+fwlKJ828YTBcl8g3NEzCp1q0=; b=B+2PHfRBTtb7URbYVoHkYqVeXfSCxvGJp1UKt7NnuF2OPkf8lUsghJMJ20LWuHRk KawcQHxr8i0me0yjKWZLU8IwP4XKCQoLTvwS9ZZVXWAAjr3loVYejWlLsXK5P5OEIcs m0BWehO2IRsSIoZ5aYm3+4cs3pjqtNR+hgnlfKOE= Received: from localhost (p54ad4c64.dip0.t-ipconnect.de [84.173.76.100]) by mx.zohomail.com with SMTPS id 1640108187448384.47795051636797; Tue, 21 Dec 2021 09:36:27 -0800 (PST) References: <875yrjpi1y.fsf@elephly.net> <87o85bjjpm.fsf@gnu.org> <871r27p5jq.fsf@elephly.net> <87a6gv3pue.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 27.2 From: Ricardo Wurmus Date: Tue, 21 Dec 2021 18:26:03 +0100 In-reply-to: <87a6gv3pue.fsf@gnu.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <87v8zhn9m1.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 51787@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1640108244; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=jH1Dkp4lD2bICXPSZf+fwlKJ828YTBcl8g3NEzCp1q0=; b=jNtiVIMCvD/wrDI6AzeHwaLJ5hUKZIeMpVnhcpEx9rsayVCxx8ORWQa8Tf6spAC8Ukt81D oNA24B+153bh5Xrw+4sZdjFepm/u275JtC7O4a5QtjBnSC/vQacRhGPHwBYztzWpBjZiI6 JyXq9ji70VzwtljMKum8twB/OM7EBvXe1aXWy3mIbu+KgvZrDVd3Kk7VgZQmJ8LM0XaP7b Nk0CUgQ/wQhL8p1fRayxT1F2S6suzyfKnd3t6wCbYLXmpdXT+MprfE3W3P1k3M1ITWhuxd 9iWYtXIYUzbR9wMyhR/3NMLVGzzDsHlpdATlBFsZvdsUqqnLOWLUFe09jYZgCw== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1640108244; a=rsa-sha256; cv=fail; b=tHDepOILacD/So724zETJzFMWqjzh6bpy2QMjsdPAcNqQvwZGc6W8kCBayys95xQo+LXDj kr6AwBeX6+JBo9jqtBh9r4XFvWhoKIMRurmqriUrzWGxREhZKxvoq4XjZ+Rknybc/EftMd viYFjxLA9ovZ+OwWKW6Nr0vmuSSld67WrIGxaAcIh33OAICSvieINzzbXU+fDwtZrEcz1A PLCu6P7D4ny3zGmM3qLNzofM0pnJe5xF50gQMdqzgCQv9tYngCsCR0S4OrzF5q4IQz8E16 bosTmAi3HnJKv2KK+KdOZQzts3bFP9q6gu2enz4qMImo2oLV7hs6ptkXNK/QEA== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=B+2PHfRB; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.83 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=B+2PHfRB; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 0E0BF2AD0 X-Spam-Score: -2.83 X-Migadu-Scanner: scn0.migadu.com X-TUID: JbVVVTkUvhww Today we discovered a few more things and discussed them on IRC. Here=E2= =80=99s a summary. /var/cache sits on the same storage as /gnu. We mounted the 5TB ext4 file system that=E2=80=99s hosted by the SAN at /mnt_test and started copyi= ng over /var/cache to /mnt_test/var/cache. Transfer speed was considerably faster (not *great*, but reasonably fast) than the copy of /gnu/store/trash to the same target. This confirmed our suspicions that the problem is not with the storage array but due to the fact that /gnu/store/trash (and also /gnu/store) is an extremely large, flat directory. /var/cache is not. Here=E2=80=99s what we do now: continue copying /var/cache to the SAN, then remount to serve substitutes from there. This removes some pressure from the file system as it will only be used for /gnu. We=E2=80=99re considering to dump the file system completely (i.e. reinstall the server), thereby emptying /gnu, but leaving the stash of built substitutes in /var/cache (hosted from the faster SAN). We could take this opportunity to reformat /gnu with btrfs, which performs quite a bit more poorly than ext4 but would be immune to defragmentation. It=E2=80=99s not clear that defragmentation matters here.= It could just be that the problem is exclusively caused by having these incredibly large, flat /gnu/store, /gnu/store/.links, and /gnu/store/trash directories. A possible alternative for this file system might also be XFS, which performs well when presented with unreasonably large directories. It may be a good idea to come up with realistic test scenarios that we could test with each of these three file systems at scale. Any ideas? --=20 Ricardo