bug#51787: GC takes more than 9 hours on berlin

[Mathieu Othacehe <othacehe@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]


Hey,

New day, new benchmark. Berlin has two hard drives, which are roughly
used this way:

/dev/sda -> / (916G)
/dev/sdb -> /gnu (37T)

I ran the fio benchmark tool on both of them. See the reports
attached, and the following summary:

--8<---------------cut here---------------start------------->8---
|       | sda       | sdb       |
|-------+-----------+-----------|
| read  | 1565KiB/s | 9695KiB/s |
| write | 523KiB/s  | 3240KiB/s |
--8<---------------cut here---------------end--------------->8---

I'm not sure how slow those figures are relatively to the hard drives
technologies. Ricardo, any idea about that?

My own NVME hard drive has 294MiB/s read and 98.1MiB/s write with the
same test in comparison.

I also tried to benchmark file removal this way, but this is not really
conclusive:

--8<---------------cut here---------------start------------->8---
# 3.8s on sdb
time for i in {0..100000}; do echo 'test' > "fs_test/test${i}.txt"; done

# 2.7s on sdb
time rf -rf fs_test
--8<---------------cut here---------------end--------------->8---

Thanks,

Mathieu


[-- Attachment #2: fio_sda --]
[-- Type: application/octet-stream, Size: 2264 bytes --]

mathieu@berlin ~$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=1437KiB/s,w=428KiB/s][r=359,w=107 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=71924: Fri Dec 17 11:17:12 2021
  read: IOPS=391, BW=1565KiB/s (1602kB/s)(3070MiB/2008868msec)
   bw (  KiB/s): min=  304, max= 3190, per=100.00%, avg=1566.30, stdev=323.23, samples=4014
   iops        : min=   76, max=  797, avg=391.48, stdev=80.84, samples=4014
  write: IOPS=130, BW=523KiB/s (536kB/s)(1026MiB/2008868msec); 0 zone resets
   bw (  KiB/s): min=  120, max= 1146, per=100.00%, avg=523.43, stdev=124.21, samples=4014
   iops        : min=   30, max=  286, avg=130.80, stdev=31.06, samples=4014
  cpu          : usr=0.16%, sys=0.58%, ctx=1039425, majf=0, minf=589
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1565KiB/s (1602kB/s), 1565KiB/s-1565KiB/s (1602kB/s-1602kB/s), io=3070MiB (3219MB), run=2008868-2008868msec
  WRITE: bw=523KiB/s (536kB/s), 523KiB/s-523KiB/s (536kB/s-536kB/s), io=1026MiB (1076MB), run=2008868-2008868msec

Disk stats (read/write):
  sda: ios=805734/364174, merge=7143/37100, ticks=137586969/41308, in_queue=137628277, util=99.71%

mathieu@berlin ~$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/var/cache/test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
fio: pid=0, err=13/file:filesetup.c:174, func=open, error=Permission denied

[-- Attachment #3: fio_sdb --]
[-- Type: application/octet-stream, Size: 2329 bytes --]

mathieu@berlin ~$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/var/cache/test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
fio: pid=0, err=13/file:filesetup.c:174, func=open, error=Permission denied


Run status group 0 (all jobs):
mathieu@berlin ~$ sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/var/cache/test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
Password: 
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=6610KiB/s,w=2190KiB/s][r=1652,w=547 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=49274: Fri Dec 17 11:30:38 2021
  read: IOPS=2423, BW=9695KiB/s (9927kB/s)(3070MiB/324270msec)
   bw (  KiB/s): min= 2392, max=17808, per=100.00%, avg=9704.06, stdev=2476.08, samples=648
   iops        : min=  598, max= 4452, avg=2425.92, stdev=619.02, samples=648
  write: IOPS=809, BW=3240KiB/s (3318kB/s)(1026MiB/324270msec); 0 zone resets
   bw (  KiB/s): min= 1080, max= 5968, per=100.00%, avg=3243.27, stdev=833.97, samples=648
   iops        : min=  270, max= 1492, avg=810.71, stdev=208.49, samples=648
  cpu          : usr=0.55%, sys=2.84%, ctx=1004749, majf=0, minf=129
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=9695KiB/s (9927kB/s), 9695KiB/s-9695KiB/s (9927kB/s-9927kB/s), io=3070MiB (3219MB), run=324270-324270msec
  WRITE: bw=3240KiB/s (3318kB/s), 3240KiB/s-3240KiB/s (3318kB/s-3318kB/s), io=1026MiB (1076MB), run=324270-324270msec

Disk stats (read/write):
  sdb: ios=790930/270846, merge=2539/10683, ticks=21982026/264629, in_queue=22246666, util=100.00%