From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id WLr6BheBwmElfwAAgWs5BA (envelope-from ) for ; Wed, 22 Dec 2021 02:36:23 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 6B3AAheBwmF0IQAA1q6Kng (envelope-from ) for ; Wed, 22 Dec 2021 01:36:23 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A9EF8C09C for ; Wed, 22 Dec 2021 02:36:22 +0100 (CET) Received: from localhost ([::1]:33668 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mzqYH-0007Rm-Rg for larch@yhetil.org; Tue, 21 Dec 2021 20:36:21 -0500 Received: from eggs.gnu.org ([209.51.188.92]:39934) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mzqXH-0007PR-2w for bug-guix@gnu.org; Tue, 21 Dec 2021 20:35:19 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44736) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mzqX0-0005hj-8K for bug-guix@gnu.org; Tue, 21 Dec 2021 20:35:13 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mzqX0-0008Dz-4F for bug-guix@gnu.org; Tue, 21 Dec 2021 20:35:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#51787: Disk performance on ci.guix.gnu.org Resent-From: Thiago Jung Bauermann Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 22 Dec 2021 01:35:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51787 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 51787@debbugs.gnu.org Cc: Ricardo Wurmus , Mathieu Othacehe , Bengt Richter , Leo Famulari Received: via spool by 51787-submit@debbugs.gnu.org id=B51787.164013687431573 (code B ref 51787); Wed, 22 Dec 2021 01:35:02 +0000 Received: (at 51787) by debbugs.gnu.org; 22 Dec 2021 01:34:34 +0000 Received: from localhost ([127.0.0.1]:56282 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzqWY-0008DB-AA for submit@debbugs.gnu.org; Tue, 21 Dec 2021 20:34:34 -0500 Received: from mx.kolabnow.com ([212.103.80.153]:24362) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzqWV-0008Cr-UY for 51787@debbugs.gnu.org; Tue, 21 Dec 2021 20:34:32 -0500 Received: from localhost (unknown [127.0.0.1]) by mx.kolabnow.com (Postfix) with ESMTP id B7888410ED; Wed, 22 Dec 2021 02:34:25 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolabnow.com; h= content-transfer-encoding:content-type:content-type:mime-version :message-id:in-reply-to:date:date:subject:subject:from:from :references:received:received:received; s=dkim20160331; t= 1640136863; x=1641951264; bh=3i63tiFp+CoNc/GuP9QPl1lU2yVYTFDvyF2 sJxmBOu0=; b=y6lWu7Pkj1WxW9gmZFp0PWS5LXAjz9+gnSFUifbB8n0eaGZuMIU S2MwHeis1SdzMI9j8rcS7hV39ZyjFo9J4FHLq5fN1xQ2E8fuSBEUYKjEFLx4/1d4 IVMx++PCYa4fHPhGp9g3OXJe7gDxaymob7V+89dnE/9BH7+ZdmOyKtlRFIMIFfT+ /0JLAudAG91n30PXfeSznSDt3F8A5XaYxCvW0ROcjP9DqkgTGM9nG6QvMjJ3um6h UiglYVFnx87HF85A9BUZrFHRroC+Fj98TBhZqgxDMFFTMiOXzwKPTQaQTCscWNfI xOFoHt3rvoUuooYNGG80vx7Qes2wwdRurU6egwHB9VXw9F/O/nNmdv1ne3IIRJZV vkxI0nxNPHFIwxla1Ky5On7U9MlaG38Qy6p8GG3uOOAA4qk/kArGBytOezn71yby E0B7iH3Fqnv7FGKlhHikR4zkVbfoknO4ZOdHlnlVfLI2CrcAW4hTGLDzmaIa+on3 fMA3QTpeOE0Ar807JaKpzU1wVq4iabbq+BqRf4lBk/lOd6s3gONxwKL7UzKfh5M0 dh+J1QpT4gec+vHONtawFuR+BsWDoBotVUnchApK1jL/Ja28pTtLCbRN3ACqi8hT cag4fUmeuCcjU31/7S8q2R2FZLWqqMrXE9kaR1IgoqwPL4vLvknfVtmg= X-Virus-Scanned: amavisd-new at mykolab.com Received: from mx.kolabnow.com ([127.0.0.1]) by localhost (ext-mx-out003.mykolab.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HB6ko0HtvW3e; Wed, 22 Dec 2021 02:34:23 +0100 (CET) Received: from int-mx001.mykolab.com (unknown [10.9.13.1]) by mx.kolabnow.com (Postfix) with ESMTPS id 56A15408FC; Wed, 22 Dec 2021 02:34:22 +0100 (CET) Received: from ext-subm003.mykolab.com (unknown [10.9.6.3]) by int-mx001.mykolab.com (Postfix) with ESMTPS id D796921AC; Wed, 22 Dec 2021 02:34:20 +0100 (CET) References: <875yrjpi1y.fsf@elephly.net> <87o85bjjpm.fsf@gnu.org> <871r27p5jq.fsf@elephly.net> <87a6gv3pue.fsf@gnu.org> <87v8zhn9m1.fsf@elephly.net> Date: Tue, 21 Dec 2021 21:27:46 -0300 In-reply-to: <87v8zhn9m1.fsf@elephly.net> Message-ID: <87lf0dzalt.fsf@kolabnow.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" Reply-to: Thiago Jung Bauermann From: Thiago Jung Bauermann via Bug reports for GNU Guix X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1640136982; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=3i63tiFp+CoNc/GuP9QPl1lU2yVYTFDvyF2sJxmBOu0=; b=FcxyTeL39BotCCkyWfVyzO4hgvFYFhSWXWHlWvR8GA4p+c7OxWow/YfbiUOHMoMuMH4pwc FJi+nBCGRUWVHRPP3ecf4gGqI9YKjv+s6hlM/2FT5gvtHRW86887hLg79OYBurZumzjMYS fTXkG02MF/G8N/fAjBWyMgawg+Q41tfbZ/xLnn1i0/Bs5Gq3WNoE4lAE2GQ89ERfug7BeE gf7/kVBKv4V0PhB59dR/ck6AuAIhiJkRNtQtnoZIw6zdZNx8qc0MPXpIL14WCV46M7kB4S Q4a9qyxU7NQ4Q9gKtOjShkt7uuKydbFNhRonkxFl8wMW5hl9xr38+WQlVgT3yw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1640136982; a=rsa-sha256; cv=none; b=rgoBSjaczTxyjDR8+zPpSgAZxS1bNWISC5huWCPRC0ldQ7TycAoc48WYnS34biEzk7HLKW cM9K0qn1QXD7rr3P32ymIp1+cVYEsBQLmpnDgN/JScp0EimFdcnzIe6T3u+3X7BanJAFYp w0P8yDNwvLa+Wa07V4gt1t2NyderfDfHLdOlw2/SddLQjXwbFHiB69EnraSnyd0l5qL61X Ni5WepHdAZsgM5JZVEPdKBr9Vpvx5I9IiZEqZ6P1mDsInuDrfXqNTZ+ttxUHEkMxykZRmd eqg77BCJBgxRwj8QQqI1jrwXPFWuJ36iFGJPwusNuHbZ5T32k/3PEsqiR7pJfQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=kolabnow.com header.s=dkim20160331 header.b=y6lWu7Pk; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.83 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=kolabnow.com header.s=dkim20160331 header.b=y6lWu7Pk; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: A9EF8C09C X-Spam-Score: -2.83 X-Migadu-Scanner: scn0.migadu.com X-TUID: G4Y+w1R87dSt Hello, Ricardo Wurmus writes: > Today we discovered a few more things and discussed them on IRC. Here=E2= =80=99s > a summary. > > /var/cache sits on the same storage as /gnu. We mounted the 5TB ext4 > file system that=E2=80=99s hosted by the SAN at /mnt_test and started cop= ying > over /var/cache to /mnt_test/var/cache. Transfer speed was considerably > faster (not *great*, but reasonably fast) than the copy of > /gnu/store/trash to the same target. > > This confirmed our suspicions that the problem is not with the storage > array but due to the fact that /gnu/store/trash (and also /gnu/store) > is an extremely large, flat directory. /var/cache is not. There was an interesting thread in the Linux kernel mailing lists about this very issue earlier this year: https://lore.kernel.org/linux-fsdevel/206078.1621264018@warthog.procyon.org= .uk/ I=E2=80=99m not sure I completely understood all of the concerns discussed = there, but my understanding of it is that for workloads which don=E2=80=99t concurrent= ly modify the huge directory, it=E2=80=99s size isn=E2=80=99t a problem for btrfs and= XFS and in fact it=E2=80=99s even more efficient to have one big directory rather than subdirectories=C2=B9. It=E2=80=99s should also be well handled even by ext4= , IIUC=C2=B2. The problem for all filesystems is concurrently modifying the directory (e.g., adding or removing files), because the kernel serializes directory operations at the VFS layer. Also in that case XFS can also have allocation issues when adding new files if one isn=E2=80=99t careful.=C2=B3 --=20 Thanks Thiago =C2=B9 https://lore.kernel.org/linux-fsdevel/20210517232237.GE2893@dread.di= saster.area/ =C2=B2 https://lore.kernel.org/linux-fsdevel/6E4DE257-4220-4B5B-B3D0-B67C7B= C69BB5@dilger.ca/ =C2=B3 https://lore.kernel.org/linux-fsdevel/20210519125743.GP2893@dread.di= saster.area/