From: Ricardo Wurmus <rekado@elephly.net>
To: Mathieu Othacehe <othacehe@gnu.org>
Cc: 51787@debbugs.gnu.org
Subject: bug#51787: GC takes more than 9 hours on berlin
Date: Fri, 17 Dec 2021 14:06:51 +0100 [thread overview]
Message-ID: <87ilvnjr49.fsf@elephly.net> (raw)
In-Reply-To: <8735mrec0x.fsf@gnu.org>
Hi Mathieu,
> New day, new benchmark. Berlin has two hard drives, which are roughly
> used this way:
>
> /dev/sda -> / (916G)
> /dev/sdb -> /gnu (37T)
sda consists of two local hard disks that are combined to a RAID.
Here are the disk details:
--8<---------------cut here---------------start------------->8---
Disk.Bay.0:Enclosure.Internal.0-1:RAID.Slot.3-1
Status = Ok
DeviceDescription = Disk 0 in Backplane 1 of RAID Controller in Slot 3
RollupStatus = Ok
Name = Physical Disk 0:1:0
State = Online
OperationState = Not Applicable
PowerStatus = Spun-Up
Size = 931.000 GB
FailurePredicted = NO
RemainingRatedWriteEndurance = Not Applicable
SecurityStatus = Not Capable
BusProtocol = SATA
MediaType = HDD
UsedRaidDiskSpace = 931.000 GB
AvailableRaidDiskSpace = 0.001 GB
Hotspare = NO
Manufacturer = SEAGATE
ProductId = ST1000NX0443
Revision = NB33
SerialNumber = W470QK7K
PartNumber = CN08DN1YSGW0076S00L8A00
NegotiatedSpeed = 6.0 Gb/s
ManufacturedDay = 0
ManufacturedWeek = 0
ManufacturedYear = 0
ForeignKeyIdentifier = null
SasAddress = 0x4433221106000000
FormFactor = 2.5 Inch
RaidNominalMediumRotationRate = 7200
T10PICapability = Not Capable
BlockSizeInBytes = 512
MaxCapableSpeed = 6 Gb/s
RaidType = None
SystemEraseCapability = CryptographicErasePD
SelfEncryptingDriveCapability = Not Capable
EncryptionCapability = Not Capable
CryptographicEraseCapability = Capable
Disk.Bay.1:Enclosure.Internal.0-1:RAID.Slot.3-1
Status = Ok
DeviceDescription = Disk 1 in Backplane 1 of RAID Controller in Slot 3
RollupStatus = Ok
Name = Physical Disk 0:1:1
State = Online
OperationState = Not Applicable
PowerStatus = Spun-Up
Size = 931.000 GB
FailurePredicted = NO
RemainingRatedWriteEndurance = Not Applicable
SecurityStatus = Not Capable
BusProtocol = SATA
MediaType = HDD
UsedRaidDiskSpace = 931.000 GB
AvailableRaidDiskSpace = 0.001 GB
Hotspare = NO
Manufacturer = SEAGATE
ProductId = ST1000NX0443
Revision = NB33
SerialNumber = W470SYTP
PartNumber = CN08DN1YSGW0077F00FQA00
NegotiatedSpeed = 6.0 Gb/s
ManufacturedDay = 0
ManufacturedWeek = 0
ManufacturedYear = 0
ForeignKeyIdentifier = null
SasAddress = 0x4433221107000000
FormFactor = 2.5 Inch
RaidNominalMediumRotationRate = 7200
T10PICapability = Not Capable
BlockSizeInBytes = 512
MaxCapableSpeed = 6 Gb/s
RaidType = None
SystemEraseCapability = CryptographicErasePD
SelfEncryptingDriveCapability = Not Capable
EncryptionCapability = Not Capable
CryptographicEraseCapability = Capable
--8<---------------cut here---------------end--------------->8---
sdb is an external storage array (Dell MD3400) filled with 10 hard disks
(SAS) in a RAID 10 configuration (36.36 TB effective capacity). There
are two hot spares that are currently unassigned. They are used
automatically when the RAID is degraded. The two RAID controllers have
read and write caches enabled. The enclosure has two redundant host
interfaces.
Berlin has two host based adapter cards of which *one* is connected to
the array. Why only one? Because we don’t have multipathd configured
so that the system could *boot* off the external array with multipath.
Without multipath the storage would appear as one disk device per card,
but it would not be safe to mount them both at the same time.
If we wanted to make use of the redundant connection here: figure out
how to add multipathd to the initrd and set up multipath *before*
handing off control to Linux. This would effectively double our
bandwidth to the storage.
My guess is that we’re not even close to saturating the available
bandwidth.
> I ran the fio benchmark tool on both of them. See the reports
> attached, and the following summary:
>
> | | sda | sdb |
> |-------+-----------+-----------|
> | read | 1565KiB/s | 9695KiB/s |
> | write | 523KiB/s | 3240KiB/s |
>
>
> I'm not sure how slow those figures are relatively to the hard drives
> technologies. Ricardo, any idea about that?
It seems awfully slow. Especially performance of sda is abysmal: this
is a local disk. sdb is the fancy external disk array that’s hooked up
to two HBA cards. It should not perform *better* than sda.
I’ll run this on a few of the build nodes to get some more comparisons.
--
Ricardo
next prev parent reply other threads:[~2021-12-17 13:33 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-12 11:49 bug#51787: GC takes more than 9 hours on berlin Mathieu Othacehe
2021-11-12 19:17 ` Mathieu Othacehe
2021-11-22 9:16 ` zimoun
2021-11-23 17:48 ` Ludovic Courtès
2021-11-25 13:24 ` Christopher Baines
2021-11-27 11:23 ` Ludovic Courtès
2021-12-03 9:45 ` Mathieu Othacehe
2021-12-10 6:24 ` Mathieu Othacehe
2021-12-10 10:21 ` Ludovic Courtès
2021-12-10 17:11 ` Mathieu Othacehe
2021-12-10 21:54 ` Ludovic Courtès
2021-12-11 9:42 ` Mathieu Othacehe
2021-12-12 17:09 ` Mathieu Othacehe
2021-12-13 16:13 ` Christian Thäter
2021-12-14 3:31 ` Christian Thäter
2021-12-17 10:55 ` Mathieu Othacehe
2021-12-17 13:06 ` Ricardo Wurmus [this message]
2021-12-17 14:08 ` Ricardo Wurmus
2021-12-10 17:11 ` Mathieu Othacehe
2021-11-27 11:11 ` Ludovic Courtès
2021-12-10 19:38 ` Ricardo Wurmus
2021-12-20 21:12 ` Ricardo Wurmus
2023-08-16 10:53 ` Maxim Cournoyer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87ilvnjr49.fsf@elephly.net \
--to=rekado@elephly.net \
--cc=51787@debbugs.gnu.org \
--cc=othacehe@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).