From: Eric Wong <e@80x24.org>
To: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: meta@public-inbox.org
Subject: Re: --batch-size and --jobs combination
Date: Thu, 29 Jul 2021 21:13:21 +0000 [thread overview]
Message-ID: <20210729211321.GA23521@dcvr> (raw)
In-Reply-To: <20210729202836.7qdwxojjel6jmxh6@nitro.local>
Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
>
> Is there any specific logic for mixing --batch-size and --jobs? On a system
> with plenty of CPUs and lots of RAM, does it make sense to have more --jobs,
> larger --batch-size, or some balance of both?
jobs will be bound by I/O capability for your case. SATA-2 vs
SATA-3 vs NVME will have a notable difference, as does the
quality of the device (MLC, TLC, QLC; cache/controller).
Xapian seems to do better with bigger batch-sizes up to a point.
I'm not sure I have enough RAM to accurately test >8m batch
sizes (since we also need to account for kernel caching).
batch-size * (jobs - 1) = rough total batch size
If it's the initial index creation, I would definitely use
--no-fsync, too. Perhaps that should be the default for new
indices.
Also note: the recent RFC for --sequential-commit doesn't seem to
be working out performance-wise on my SATA-2 system; but I'm also
not sure about SSD life/degradation.
next prev parent reply other threads:[~2021-07-29 21:13 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-29 20:28 --batch-size and --jobs combination Konstantin Ryabitsev
2021-07-29 21:13 ` Eric Wong [this message]
2021-07-29 21:24 ` Konstantin Ryabitsev
2021-07-29 22:06 ` Eric Wong
2021-08-01 20:40 ` Konstantin Ryabitsev
2021-08-05 11:05 ` Eric Wong
2021-08-05 20:59 ` Konstantin Ryabitsev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210729211321.GA23521@dcvr \
--to=e@80x24.org \
--cc=konstantin@linuxfoundation.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).