From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id KDjaJI9hwmEONAAAgWs5BA (envelope-from ) for ; Wed, 22 Dec 2021 00:21:51 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 2KeqII9hwmHzYAAAB5/wlQ (envelope-from ) for ; Tue, 21 Dec 2021 23:21:51 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 049F3125EF for ; Wed, 22 Dec 2021 00:21:51 +0100 (CET) Received: from localhost ([::1]:33970 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mzoS5-0003dW-RE for larch@yhetil.org; Tue, 21 Dec 2021 18:21:49 -0500 Received: from eggs.gnu.org ([209.51.188.92]:47240) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mzoRL-0003bj-Ma for bug-guix@gnu.org; Tue, 21 Dec 2021 18:21:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44620) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mzoRJ-0007io-P6 for bug-guix@gnu.org; Tue, 21 Dec 2021 18:21:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mzoRJ-0002js-MF for bug-guix@gnu.org; Tue, 21 Dec 2021 18:21:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#51787: Disk performance on ci.guix.gnu.org Resent-From: Bengt Richter Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 21 Dec 2021 23:21:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51787 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ricardo Wurmus Received: via spool by 51787-submit@debbugs.gnu.org id=B51787.164012884410471 (code B ref 51787); Tue, 21 Dec 2021 23:21:01 +0000 Received: (at 51787) by debbugs.gnu.org; 21 Dec 2021 23:20:44 +0000 Received: from localhost ([127.0.0.1]:56160 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzoR2-0002io-2r for submit@debbugs.gnu.org; Tue, 21 Dec 2021 18:20:44 -0500 Received: from imta-36.everyone.net ([216.200.145.36]:51576 helo=imta-38.everyone.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzoQz-0002ic-Hr for 51787@debbugs.gnu.org; Tue, 21 Dec 2021 18:20:42 -0500 Received: from pps.filterd (omta002.sj2.proofpoint.com [127.0.0.1]) by imta-38.everyone.net (8.16.0.43/8.16.0.43) with SMTP id 1BLNItAo029320; Tue, 21 Dec 2021 15:20:39 -0800 X-Eon-Originating-Account: i9_lel_50rLmNMFnfTwMF5dgksd13XQGCXzUboYiffs X-Eon-Dm: m0117124.ppops.net Received: by m0117124.mta.everyone.net (EON-AUTHRELAY2 - 53b92f98) id m0117124.6195d1ae.28b54c; Tue, 21 Dec 2021 15:20:37 -0800 X-Eon-Sig: AQMHrIJhwmFFNJJjkwIAAAAD,9402e946dd9158a37aa9c87c691908ec X-Eip: pK5W_qA8Izdfz3RPTY6r50YbJvK_DaTYHrb3kbvsItQ Date: Wed, 22 Dec 2021 00:20:24 +0100 From: Bengt Richter Message-ID: <20211221232024.GA41746@LionPure> References: <875yrjpi1y.fsf@elephly.net> <87o85bjjpm.fsf@gnu.org> <871r27p5jq.fsf@elephly.net> <87a6gv3pue.fsf@gnu.org> <87v8zhn9m1.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <87v8zhn9m1.fsf@elephly.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-Proofpoint-GUID: 3bv8ji_1oRleO6-mugcqY5_-fmQGCT6f X-Proofpoint-ORIG-GUID: 3bv8ji_1oRleO6-mugcqY5_-fmQGCT6f X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.425, 18.0.790 definitions=2021-12-21_07:2021-12-21, 2021-12-21 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 impostorscore=0 lowpriorityscore=0 adultscore=0 bulkscore=0 spamscore=0 phishscore=0 mlxlogscore=804 mlxscore=0 malwarescore=0 clxscore=1034 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2112210116 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Bengt Richter Cc: Mathieu Othacehe , 51787@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1640128911; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=iZm3MI+HoWPxAjo4ckgAasVyBxSN/rMPwnhOJJjkFUA=; b=CUXcd3kPoXz6/eBaSuE58URWQu8j9hdimOfZ2HZm23nKvlNMvZTsiyp3v3JkdclL5ebUPF +5Wp5z89Rug2PGHURllTehiHYnvNlUmsAfjH4mKTCKORlsJwG6yrT5YTG+GKyWF+TlJcfu 19s7yjtouq39l2N+Yq2RVfa9K5ZBV90YgrbvbCLRai2THsRrTN5WLnipFIWLybStgBeSXF 0uSBy151Ar4WmAB03K7OIERwvHMR0a0GNK4jubHh8GHIxyYYQTxL7HYVCkdpwvFndUKzOY bqho2SNY0Vb5VKvpgNnjKVIWrEf3BSebwd/TaM/nxIILFfq2cspkQ85YzDvLxg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1640128911; a=rsa-sha256; cv=none; b=MFEryvypbvllE+4lgLy0S7oEgQ7KwBGtcKjkMiEiADtyOEFZqtDaVhPkNZYJ0p+EIoD/5I iK1Av1Dyjx9Bc7LLxZVdqmNKiR6NFUfcDOL/LxHBKWU3Yl1v2+1dNa/cOIyGMpFaD42vQt gKEOCl0YuRUSIKCnF8IIEM62IO2paIEL59uiMqBFsFO/mq1aG9tmML60SHsn9UhhrEI1kE VKIw/G/YSUnzx9f/Ks99uUYgGU3GjqP04gxLc1+wU0lAcJyihjMY6LQOQ9JXSvCp5aEACe aBZIDPFZ4zGkiB1pWEVzUhAGzfAbhTY+9gTDNWfSZXAtE4+qupZfFnVyrXc81g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.13 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 049F3125EF X-Spam-Score: -2.13 X-Migadu-Scanner: scn0.migadu.com X-TUID: gVCutSpMRfHr Hi Ricardo, TL;DR: re: "Any ideas?" :) Read this [0], and consider how file systems may be interacting with with SSD wear-leveling algorithms. Are some file systems dependent on successful speculative transaction continuations, while others might slow down waiting for signs that an SSD controller has committed one of ITS transactions, e.g. in special cases where the user or kernel file system wants to be sure metadata is written/journaled for fs structural integrity, but maybe cares less about data? I guess this difference might show up in copying a large file over-writing the same target file (slower) vs copying to a series of new files (faster). What happens if you use a contiguous file as swap space? Or, if you use anonymous files as user data space buffers, passing them to wayland as file handles, per its protocol, can you do better than ignoring SSD controllers and/or storage hardware altogether? Reference [0] is from 2013, so probably much has happened since then, and the paper mentions (which has probably not gotten better), the following, referring to trade secrets giving one manufacturer ability to produce longer-lasting SSDs cheaper and better than others ... --8<---------------cut here---------------start------------->8--- This means that the SSD controller is dedicated to a single brand of NAND, and it means that the SSD maker can’t shop around among NAND suppliers for the best price. Furthermore, the NAND supplier won’t share this information unless it believes that there is some compelling reason to work the SSD manufacturer. Since there are hundreds of SSD makers it’s really difficult to get these companies to pay attention to you! The SSD manufacturers that have this kind of relationship with their flash suppliers are very rare and very special. --8<---------------cut here---------------end--------------->8--- Well, maybe you will have to parameterize your file system tuning with manufacturer ID and SSD controller firmware version ;/ Mvh, Bengt [0] https://www.snia.org/sites/default/files/SSSITECHNOTES_HowControllersMaximizeSSDLife.pdf On +2021-12-21 18:26:03 +0100, Ricardo Wurmus wrote: > Today we discovered a few more things and discussed them on IRC. Here’s > a summary. > > /var/cache sits on the same storage as /gnu. We mounted the 5TB ext4 > file system that’s hosted by the SAN at /mnt_test and started copying > over /var/cache to /mnt_test/var/cache. Transfer speed was considerably > faster (not *great*, but reasonably fast) than the copy of > /gnu/store/trash to the same target. > > This confirmed our suspicions that the problem is not with the storage > array but due to the fact that /gnu/store/trash (and also /gnu/store) > is an extremely large, flat directory. /var/cache is not. > > Here’s what we do now: continue copying /var/cache to the SAN, then > remount to serve substitutes from there. This removes some pressure > from the file system as it will only be used for /gnu. We’re > considering to dump the file system completely (i.e. reinstall the > server), thereby emptying /gnu, but leaving the stash of built > substitutes in /var/cache (hosted from the faster SAN). > > We could take this opportunity to reformat /gnu with btrfs, which > performs quite a bit more poorly than ext4 but would be immune to > defragmentation. It’s not clear that defragmentation matters here. It > could just be that the problem is exclusively caused by having these > incredibly large, flat /gnu/store, /gnu/store/.links, and > /gnu/store/trash directories. > > A possible alternative for this file system might also be XFS, which > performs well when presented with unreasonably large directories. > > It may be a good idea to come up with realistic test scenarios that we > could test with each of these three file systems at scale. > > Any ideas? > > -- > Ricardo > > > (sorry, the top-post grew) -- Regards, Bengt Richter