From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
To: Eric Wong <e@80x24.org>
Cc: meta@public-inbox.org
Subject: Re: --batch-size and --jobs combination
Date: Thu, 29 Jul 2021 17:24:44 -0400 [thread overview]
Message-ID: <20210729212444.3jnmaq4vo4dnudk3@nitro.local> (raw)
In-Reply-To: <20210729211321.GA23521@dcvr>
On Thu, Jul 29, 2021 at 09:13:21PM +0000, Eric Wong wrote:
> jobs will be bound by I/O capability for your case. SATA-2 vs
> SATA-3 vs NVME will have a notable difference, as does the
> quality of the device (MLC, TLC, QLC; cache/controller).
So, on these systems with large lvmcache disks, large --jobs number isn't
really going to help, right? Disk performance is not great, since these
systems are continuously serving traffic for mirrors.kernel.org, but writes
are spread across RAID-6, so not sure --sequential-shard will do much
difference.
Let me try "--jobs=2 --batch-size=1g --no-fsync" for the initial index.
There's 128G of RAM on this system, so it should be able to take that easily.
I didn't use --no-fsync due to the admonition in the manpages that it probably
will result in worse performance with a large --batch-size, but if you say
it's going to likely help, I'm willing to try that out.
Thanks for your help!
-K
next prev parent reply other threads:[~2021-07-29 21:24 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-29 20:28 --batch-size and --jobs combination Konstantin Ryabitsev
2021-07-29 21:13 ` Eric Wong
2021-07-29 21:24 ` Konstantin Ryabitsev [this message]
2021-07-29 22:06 ` Eric Wong
2021-08-01 20:40 ` Konstantin Ryabitsev
2021-08-05 11:05 ` Eric Wong
2021-08-05 20:59 ` Konstantin Ryabitsev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210729212444.3jnmaq4vo4dnudk3@nitro.local \
--to=konstantin@linuxfoundation.org \
--cc=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).