From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id eM5EHx31wGFuEQEAgWs5BA (envelope-from ) for ; Mon, 20 Dec 2021 22:26:53 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id GCXpGh31wGGlXAAAB5/wlQ (envelope-from ) for ; Mon, 20 Dec 2021 21:26:53 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 136A19487 for ; Mon, 20 Dec 2021 22:26:53 +0100 (CET) Received: from localhost ([::1]:33506 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mzQBI-0002Td-8C for larch@yhetil.org; Mon, 20 Dec 2021 16:26:52 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56924) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mzQAV-0002QT-GI for bug-guix@gnu.org; Mon, 20 Dec 2021 16:26:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:40045) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mzQAV-0000X9-6Y for bug-guix@gnu.org; Mon, 20 Dec 2021 16:26:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mzQAT-0004va-Vk for bug-guix@gnu.org; Mon, 20 Dec 2021 16:26:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#51787: GC takes more than 9 hours on berlin References: <87o86pegr3.fsf@gnu.org> In-Reply-To: <87o86pegr3.fsf@gnu.org> Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 20 Dec 2021 21:26:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51787 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 51787@debbugs.gnu.org Received: via spool by 51787-submit@debbugs.gnu.org id=B51787.164003552318886 (code B ref 51787); Mon, 20 Dec 2021 21:26:01 +0000 Received: (at 51787) by debbugs.gnu.org; 20 Dec 2021 21:25:23 +0000 Received: from localhost ([127.0.0.1]:51591 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzQ9q-0004uX-Rl for submit@debbugs.gnu.org; Mon, 20 Dec 2021 16:25:23 -0500 Received: from sender3-of-o52.zoho.com ([136.143.184.52]:21817) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mzQ9o-0004uO-IK for 51787@debbugs.gnu.org; Mon, 20 Dec 2021 16:25:21 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1640035516; cv=none; d=zohomail.com; s=zohoarc; b=QHTqoZxDLzm2QLuJfLpwYtzrCl7TeciaKgkQDNvQ/XN60dEF2R6HnzKzYPsmOy1crw+bNXIJ8RLMDJktiXAh/b6qsp5BEIzArmBYlc418a4wPWtl4MLiZaVWnqPACwe8qifQB3EzpDQurn+kG6g1iTnLtvYht3jDEqBiMKBxawY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1640035516; h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To; bh=xIbwBFNy3bt2yzc9rgHFU2pVXQ9mHZioEMGDDMM6YgY=; b=DcyRK+jd/Be3+x/fvXnv5jWGMFu+ySaY/WEqjPBtuaqW1Eqn4aEj9puTPNKX8ActpwuiGjpkRl3CR+2vM7SVYl+qulaLNitHrFVrBqvtf3TidIpS7uW55IgbXvC2di9aKeafIlXadkWa6ap9PypIUDRwT+B3IexpR+N/9PiZVZo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1640035516; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=xIbwBFNy3bt2yzc9rgHFU2pVXQ9mHZioEMGDDMM6YgY=; b=cSvmmhusJJIJ6kSr2f9y2LXKpc9lJC7xs+Mwu/TDCqMWdaChaZwjTmvp/Aop2h0y dtN7UmruJJIJTTeaP1hNxyc3Vsx3DEZtsctl/JTdHD6c5ds8PRw6nOmcRjoRHztQN3q YKf+f10E1tbpD6inQ1I4oRQ5vs1gBFmD8PAMFGh8= Received: from localhost (p54ad4ec1.dip0.t-ipconnect.de [84.173.78.193]) by mx.zohomail.com with SMTPS id 1640035513433995.1509856250896; Mon, 20 Dec 2021 13:25:13 -0800 (PST) User-agent: mu4e 1.6.10; emacs 27.2 From: Ricardo Wurmus Date: Mon, 20 Dec 2021 22:12:46 +0100 X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <87czlrnf4a.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1640035613; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=xIbwBFNy3bt2yzc9rgHFU2pVXQ9mHZioEMGDDMM6YgY=; b=iJLFV9NwslVdZsWI17HFfNqRisT4pGYPHCL7RmQrQc6zfajCUnNhwQGmpFJsHDdTkt51c/ fquAIrx3o1ZE9jCazG+ybNcVQvmdJMu1O0WxFITKnfzZRCUJlRQPLF/F11DteTK1eG3v1p y92JjUsDeHe+Fx588yj5Pzd1B9A7hMlodItGjGIEJMqJWUTrm0BiLQO5/U/ype0OIgNWpK tlhwLXxdEfMEVkvv5xQiXQoqKuxCKqpIye1WF0CWN/B/YiAN2ZuuRMhTwIDrZuF0foT6/t +Q6acEqi99GhPpzIchCCfsyrQGLcqsQNB/JpSPmqWnTXXHSZ68CudpWsYCymaA== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1640035613; a=rsa-sha256; cv=fail; b=U6D8nl6Kd6n3H2TFzYi7HsvQYFhJLQf533mXElgDwB8pSWdXa9dGl+NqUgdWfRt5rJ+3r0 lFgW2PaNyVDJFitWNALSLKVDTpvJ7C1MZhtarHvd3hGmFM0ZsOeagUsKgocaVn5fu8XyPS dywPKH/4eLmM3+8380piitsR+z7g2ogl3y+Ra5i696AqZhtYyH+yv/zzOv/LvChhu4QqKJ BAc0e9HdLEL6Ir8VznldUeskNPh6KaS2XleLzliH+ZesGAlsQODlvbFHOD2gRF+xL6/8c6 2GjU7QKiSCbm5VTiI5euIcPlo8wHDcJbm4A5d5Sb+ACmiOwAX6KnvuX/Zy3Gzg== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=cSvmmhus; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 0.58 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=cSvmmhus; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 136A19487 X-Spam-Score: 0.58 X-Migadu-Scanner: scn0.migadu.com X-TUID: l1Ilql2JJezs My colleague extended the SAN slice to 5TB for more realistic testing. I formatted the disk with btrfs, and mounted it like this: mount /dev/sdd /mnt_test/ Then I ran the test with block size 512k: --8<---------------cut here---------------start------------->8--- root@berlin ~# fio --randrepeat=3D1 --ioengine=3Dlibaio --direct=3D1 --gtod= _reduce=3D1 --name=3Dtest --filename=3D/mnt_test/test --bs=3D512k --iodepth= =3D64 --size=3D4G --readwrite=3Drandrw --rwmixread=3D75 test: (g=3D0): rw=3Drandrw, bs=3D(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) = 512KiB-512KiB, ioengine=3Dlibaio, iodepth=3D64 fio-3.6 Starting 1 process test: Laying out IO file (1 file / 4096MiB) Jobs: 1 (f=3D1): [m(1)][100.0%][r=3D802MiB/s,w=3D274MiB/s][r=3D1603,w=3D547= IOPS][eta 00m:00s] test: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D16949: Mon Dec 20 22:18:28 2= 021 read: IOPS=3D1590, BW=3D795MiB/s (834MB/s)(3055MiB/3842msec) bw ( KiB/s): min=3D747520, max=3D857088, per=3D99.83%, avg=3D812763.43,= stdev=3D44213.07, samples=3D7 iops : min=3D 1460, max=3D 1674, avg=3D1587.43, stdev=3D86.35, sa= mples=3D7 write: IOPS=3D542, BW=3D271MiB/s (284MB/s)(1042MiB/3842msec) bw ( KiB/s): min=3D262144, max=3D297984, per=3D100.00%, avg=3D278820.57= , stdev=3D15115.88, samples=3D7 iops : min=3D 512, max=3D 582, avg=3D544.57, stdev=3D29.52, sam= ples=3D7 cpu : usr=3D1.98%, sys=3D96.28%, ctx=3D1096, majf=3D0, minf=3D6 IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.2%, 32=3D0.= 4%, >=3D64=3D99.2% submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64= =3D0.0%, >=3D64=3D0.0% complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64= =3D0.1%, >=3D64=3D0.0% issued rwts: total=3D6109,2083,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0 latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D64 Run status group 0 (all jobs): READ: bw=3D795MiB/s (834MB/s), 795MiB/s-795MiB/s (834MB/s-834MB/s), io= =3D3055MiB (3203MB), run=3D3842-3842msec WRITE: bw=3D271MiB/s (284MB/s), 271MiB/s-271MiB/s (284MB/s-284MB/s), io= =3D1042MiB (1092MB), run=3D3842-3842msec --8<---------------cut here---------------end--------------->8--- Because this is fun I reran it with the same arguments: --8<---------------cut here---------------start------------->8--- root@berlin ~# fio --randrepeat=3D1 --ioengine=3Dlibaio --direct=3D1 --gtod= _reduce=3D1 --name=3Dtest --filename=3D/mnt_test/test --bs=3D512k --iodepth= =3D64 --size=3D4G --readwrite=3Drandrw --rwmixread=3D75 test: (g=3D0): rw=3Drandrw, bs=3D(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) = 512KiB-512KiB, ioengine=3Dlibaio, iodepth=3D64 fio-3.6 Starting 1 process Jobs: 1 (f=3D0): [f(1)][-.-%][r=3D756MiB/s,w=3D260MiB/s][r=3D1511,w=3D519 I= OPS][eta 00m:00s] test: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D17488: Mon Dec 20 22:18:56 2= 021 read: IOPS=3D1647, BW=3D824MiB/s (864MB/s)(3055MiB/3708msec) bw ( KiB/s): min=3D738304, max=3D929792, per=3D99.28%, avg=3D837485.71,= stdev=3D73710.05, samples=3D7 iops : min=3D 1442, max=3D 1816, avg=3D1635.71, stdev=3D143.96, s= amples=3D7 write: IOPS=3D561, BW=3D281MiB/s (295MB/s)(1042MiB/3708msec) bw ( KiB/s): min=3D234496, max=3D320512, per=3D99.79%, avg=3D287012.57,= stdev=3D29009.60, samples=3D7 iops : min=3D 458, max=3D 626, avg=3D560.57, stdev=3D56.66, sam= ples=3D7 cpu : usr=3D1.38%, sys=3D96.47%, ctx=3D1394, majf=3D0, minf=3D16= 420 IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.2%, 32=3D0.= 4%, >=3D64=3D99.2% submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64= =3D0.0%, >=3D64=3D0.0% complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64= =3D0.1%, >=3D64=3D0.0% issued rwts: total=3D6109,2083,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0 latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D64 Run status group 0 (all jobs): READ: bw=3D824MiB/s (864MB/s), 824MiB/s-824MiB/s (864MB/s-864MB/s), io= =3D3055MiB (3203MB), run=3D3708-3708msec WRITE: bw=3D281MiB/s (295MB/s), 281MiB/s-281MiB/s (295MB/s-295MB/s), io= =3D1042MiB (1092MB), run=3D3708-3708msec --8<---------------cut here---------------end--------------->8--- Then I mounted with compression and space cache: mount /dev/sdd -o compress-force=3Dzstd,space_cache=3Dv2 /mnt_test/ The numbers don=E2=80=99t differ much at all. --8<---------------cut here---------------start------------->8--- Run status group 0 (all jobs): READ: bw=3D882MiB/s (925MB/s), 882MiB/s-882MiB/s (925MB/s-925MB/s), io= =3D3055MiB (3203MB), run=3D3464-3464msec WRITE: bw=3D301MiB/s (315MB/s), 301MiB/s-301MiB/s (315MB/s-315MB/s), io= =3D1042MiB (1092MB), run=3D3464-3464msec --8<---------------cut here---------------end--------------->8--- I then erased the file system and again put on a big ext4: --8<---------------cut here---------------start------------->8--- root@berlin ~# fio --randrepeat=3D1 --ioengine=3Dlibaio --direct=3D1 --gtod= _reduce=3D1 --name=3Dtest --filename=3D/mnt_test/test --bs=3D512k --iodepth= =3D64 --size=3D4G --readwrite=3Drandrw --rwmixread=3D75 test: (g=3D0): rw=3Drandrw, bs=3D(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) = 512KiB-512KiB, ioengine=3Dlibaio, iodepth=3D64 fio-3.6 Starting 1 process test: Laying out IO file (1 file / 4096MiB) Jobs: 1 (f=3D1): [m(1)][-.-%][r=3D1539MiB/s,w=3D526MiB/s][r=3D3078,w=3D1052= IOPS][eta 00m:00s] test: (groupid=3D0, jobs=3D1): err=3D 0: pid=3D20672: Mon Dec 20 22:23:29 2= 021 read: IOPS=3D3077, BW=3D1539MiB/s (1614MB/s)(3055MiB/1985msec) bw ( MiB/s): min=3D 1530, max=3D 1548, per=3D100.00%, avg=3D1539.33, st= dev=3D 9.02, samples=3D3 iops : min=3D 3060, max=3D 3096, avg=3D3078.67, stdev=3D18.04, sa= mples=3D3 write: IOPS=3D1049, BW=3D525MiB/s (550MB/s)(1042MiB/1985msec) bw ( KiB/s): min=3D533504, max=3D557056, per=3D100.00%, avg=3D546133.33= , stdev=3D11868.39, samples=3D3 iops : min=3D 1042, max=3D 1088, avg=3D1066.67, stdev=3D23.18, sa= mples=3D3 cpu : usr=3D2.17%, sys=3D11.24%, ctx=3D4787, majf=3D0, minf=3D8 IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.2%, 32=3D0.= 4%, >=3D64=3D99.2% submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64= =3D0.0%, >=3D64=3D0.0% complete : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64= =3D0.1%, >=3D64=3D0.0% issued rwts: total=3D6109,2083,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0 latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D64 Run status group 0 (all jobs): READ: bw=3D1539MiB/s (1614MB/s), 1539MiB/s-1539MiB/s (1614MB/s-1614MB/s)= , io=3D3055MiB (3203MB), run=3D1985-1985msec WRITE: bw=3D525MiB/s (550MB/s), 525MiB/s-525MiB/s (550MB/s-550MB/s), io= =3D1042MiB (1092MB), run=3D1985-1985msec Disk stats (read/write): sdd: ios=3D5926/2087, merge=3D1/0, ticks=3D119183/3276, in_queue=3D122460= , util=3D94.87% --8<---------------cut here---------------end--------------->8--- No idea why btrfs performs so much worse in comparison. I=E2=80=99ll copy over /gnu/store/trash next. --=20 Ricardo