performance problems with notmuch new

performance problems with notmuch new

[Don Zickus]

Hi,

I have noticed my 'notmuch new' command seems awfully slow, maybe 10-20
emails / second on large batches.  It goes quick for the first few hundred
(maybe close to 100/second), then quickly slows down to about 10/second
after processing the first 500 or so.

I am guessing that isn't an expected behaviour.  So I am trying to figure
out a good way to analyze and debug this?  This could be a problem with my
fedora distro or laptop.  I just don't know where to look.

Tips for debugging this?

I ran the notmuch performance/time-test, but after 15 minutes of waiting for
the initial notmuch new to finish, I gave up and aborted.

I am using 'glass' for my xapian storage if that helps.

Help?

Cheers,
Don

Re: performance problems with notmuch new

[David Bremner]

Don Zickus <dzickus@redhat.com> writes:

>
> Tips for debugging this?
>
> I ran the notmuch performance/time-test, but after 15 minutes of waiting for
> the initial notmuch new to finish, I gave up and aborted.

You can try one of the smaller corpus sizes

$ make OPTIONS=--small time-test

runs in about 30s here (i7 4770 / SSD).  Replacing --small with --medium
takes about 10M (so a superlinear slowdown in wall clock time, since
that represents a 10x scale-up in the corpus size.).

d

Re: performance problems with notmuch new

[Don Zickus]

On Wed, Apr 15, 2020 at 01:01:52PM -0300, David Bremner wrote:
> Don Zickus <dzickus@redhat.com> writes:
> 
> >
> > Tips for debugging this?
> >
> > I ran the notmuch performance/time-test, but after 15 minutes of waiting for
> > the initial notmuch new to finish, I gave up and aborted.
> 
> You can try one of the smaller corpus sizes
> 
> $ make OPTIONS=--small time-test
> 
> runs in about 30s here (i7 4770 / SSD).  Replacing --small with --medium
> takes about 10M (so a superlinear slowdown in wall clock time, since
> that represents a 10x scale-up in the corpus size.).

Hmm, for me --small was 35s and --medium was 32 minutes.  This is on a
i7-9750H / nvme.  I would expect numbers similar to yours.

Cheers,
Don

Re: performance problems with notmuch new

[David Bremner]

Don Zickus <dzickus@redhat.com> writes:

>> runs in about 30s here (i7 4770 / SSD).  Replacing --small with --medium
>> takes about 10M (so a superlinear slowdown in wall clock time, since
>> that represents a 10x scale-up in the corpus size.).
>
> Hmm, for me --small was 35s and --medium was 32 minutes.  This is on a
> i7-9750H / nvme.  I would expect numbers similar to yours.

I did another few tests test for --medium and they all take 7-9 minutes,
depending what else is going on on the machine.
Here's my breakdown of times (unfortunately a bit of hand editing is
needed to clean up the warnings)


performance-test/notmuch-time-test --medium

T00-new.sh: Testing notmuch new                         [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  Initial notmuch new   66.29	62.22	2.82	241148	0/1089784
  notmuch new #2        0.03	0.00	0.00	9864	0/160
  notmuch new #3        0.00	0.00	0.00	9292	0/8
  notmuch new #4        0.00	0.00	0.00	9556	0/8
  notmuch new #5        0.00	0.00	0.00	9396	0/8
  notmuch new #6        0.00	0.00	0.00	9316	0/8
  new (7500 mv)         49.02	35.88	12.59	185152	0/392720
  new (7500 mv back)    59.41	43.04	15.88	185888	0/413824
  new (7500 cp)         36.09	26.55	9.02	182160	0/411840

T01-dump-restore.sh: Testing dump and restore           [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  load nmbug tags       5.37	2.09	1.63	12172	0/31864
  dump *                0.63	0.58	0.04	11756	0/4344
  restore *             0.72	0.65	0.06	9572	0/0

T02-tag.sh: Testing tagging                             [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  tag * +new_tag        54.45	31.93	20.26	86380	8/250512
  tag * +existing_tag   0.00	0.00	0.00	9396	0/0
  tag * -existing_tag   46.73	26.07	20.08	20580	0/284248
  tag * -missing_tag    0.00	0.00	0.00	9316	0/0

T03-reindex.sh: Testing reindexing                      [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  reindex *             78.39	63.06	14.31	229204	0/546136
  reindex *             70.52	56.68	13.12	223980	0/333608
  reindex *             78.02	62.61	14.92	225160	0/374424

T04-thread-subquery.sh: Testing thread subqueries       [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  search thread:{} ...  0.37	0.33	0.04	26508	0/24
  search thread:{} ...  0.38	0.33	0.04	23592	0/24
  search thread:{} ...  0.37	0.34	0.03	26612	0/24

415.28user 128.24system 9:13.07elapsed 98%CPU (0avgtext+0avgdata 241148maxresident)k
8inputs+7294896outputs (0major+458590minor)pagefaults 0swaps

Re: performance problems with notmuch new

[Franz Fellner]

I also suffer from bad performance of notmuch new.
I used notmuch some years ago and notmuch new always felt instantanious.
Had to stop using it because internet was too slow to sync my mails :/
Now (with better internet and a completely new setup using mbsync) indexing one mail takes at least 10 seconds, sometimes even more.
It can go into minutes when I get lots of mail (~30...).
When I run it after a reboot I can have breakfast while notmuch starts up...
This is all on spinning rust. I thought of getting an SSD but not in the near future.

What I observe during that time: notmuch doesn't really need much CPU.
iotop shows constant read and write with extremely low rates, under 1MB/sec.
So I think it might be an issue in xapian?

If there is anything I can do to help debug this please tell me

Franz 

P.S.:
@David: Sorry for writing only to you.
GMail web interface only added you as recipient and not the list...

On Wed Apr 15 11:08:01 2020, Don Zickus <dzickus@redhat.com> wrote:
> Hi,
> 
> I have noticed my 'notmuch new' command seems awfully slow, maybe 10-20
> emails / second on large batches.  It goes quick for the first few hundred
> (maybe close to 100/second), then quickly slows down to about 10/second
> after processing the first 500 or so.
> 
> I am guessing that isn't an expected behaviour.  So I am trying to figure
> out a good way to analyze and debug this?  This could be a problem with my
> fedora distro or laptop.  I just don't know where to look.
> 
> Tips for debugging this?
> 
> I ran the notmuch performance/time-test, but after 15 minutes of waiting for
> the initial notmuch new to finish, I gave up and aborted.
> 
> I am using 'glass' for my xapian storage if that helps.
> 
> Help?
> 
> Cheers,
> Don
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch
> 
> 
-- 

Re: performance problems with notmuch new

[David Bremner]

Don Zickus <dzickus@redhat.com> writes:

>
> Hmm, for me --small was 35s and --medium was 32 minutes.  This is on a
> i7-9750H / nvme.  I would expect numbers similar to yours.
>

In addition to the breakdown of numbers that I posted, it would be
potentially useful to know if it is I/O bound or CPU bound, and what
kind of file system you are using.

d

Re: performance problems with notmuch new

[David Bremner]

Franz Fellner <alpine.art.de@gmail.com> writes:

> I also suffer from bad performance of notmuch new.  I used notmuch
> some years ago and notmuch new always felt instantanious.  Had to stop
> using it because internet was too slow to sync my mails :/ Now (with
> better internet and a completely new setup using mbsync) indexing one
> mail takes at least 10 seconds, sometimes even more.  It can go into
> minutes when I get lots of mail (~30...).  When I run it after a
> reboot I can have breakfast while notmuch starts up...  This is all on
> spinning rust. I thought of getting an SSD but not in the near future.

I do have at least one spinning rust configuration with about 300k
messages, and notmuch new is still fast there.

> What I observe during that time: notmuch doesn't really need much CPU.
> iotop shows constant read and write with extremely low rates, under
> 1MB/sec.  So I think it might be an issue in xapian?
>

Just in case one of the xapian experts can suggest some kind of test for
why you might be seeing this behaviour, I've included the xapian list in
CC.

> If there is anything I can do to help debug this please tell me

What kind of filesystem do you have on your spinning rust?

d

Re: performance problems with notmuch new

[Kim Minh Kaplan]

Franz Fellner writes:

> I also suffer from bad performance of notmuch new.
> I used notmuch some years ago and notmuch new always felt instantanious.
> Had to stop using it because internet was too slow to sync my mails :/
> Now (with better internet and a completely new setup using mbsync) indexing one mail takes at least 10 seconds, sometimes even more.
> It can go into minutes when I get lots of mail (~30...).
> When I run it after a reboot I can have breakfast while notmuch starts up...
> This is all on spinning rust. I thought of getting an SSD but not in the near future.
>
> What I observe during that time: notmuch doesn't really need much CPU.
> iotop shows constant read and write with extremely low rates, under 1MB/sec.
> So I think it might be an issue in xapian?

Hello,

I had a similar performance issue working with a very large email set
(in the million count). Amongst those, I have crons that run about every
minute and email back a report, mostly identical every time with some
thousands words.

Apparently indexing many times mostly identical emails triggers a
pathological behaviour. Moving these email out of my mail directories
brought back acceptable performances. We are speaking in the 10,000 to
100,000 files here.

Kim Minh.


> If there is anything I can do to help debug this please tell me
>
> Franz 
>
> P.S.:
> @David: Sorry for writing only to you.
> GMail web interface only added you as recipient and not the list...

Re: performance problems with notmuch new

[Don Zickus]

On Wed, Apr 15, 2020 at 05:29:13PM -0300, David Bremner wrote:
> Don Zickus <dzickus@redhat.com> writes:
> 
> >> runs in about 30s here (i7 4770 / SSD).  Replacing --small with --medium
> >> takes about 10M (so a superlinear slowdown in wall clock time, since
> >> that represents a 10x scale-up in the corpus size.).
> >
> > Hmm, for me --small was 35s and --medium was 32 minutes.  This is on a
> > i7-9750H / nvme.  I would expect numbers similar to yours.
> 
> I did another few tests test for --medium and they all take 7-9 minutes,
> depending what else is going on on the machine.
> Here's my breakdown of times (unfortunately a bit of hand editing is
> needed to clean up the warnings)

Sorry for the delay.  I re-ran the test today and see similar numbers to
yours now. Roughly 6m30s per run (3 runs).

T00-new.sh: Testing notmuch new                         [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  Initial notmuch new   61.86	59.13	2.53	228124	2024/1087672
  notmuch new #2        0.01	0.00	0.00	9136	0/160
  notmuch new #3        0.00	0.00	0.00	9068	0/8
  notmuch new #4        0.00	0.00	0.00	8940	0/8
  notmuch new #5        0.00	0.00	0.00	9060	0/8
  notmuch new #6        0.00	0.00	0.00	8932	0/8
  new (7500 mv)         44.76	32.67	12.02	186156	0/541024
  new (7500 mv back)    47.82	34.81	12.91	187240	0/500000
  new (7500 cp)         26.41	20.06	6.28	183448	0/378192

T01-dump-restore.sh: Testing dump and restore           [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
3.35	1.82	1.31	11992	624/31976
  dump *                0.60	0.57	0.02	11356	0/4344
  restore *             0.69	0.62	0.06	9064	0/0

T02-tag.sh: Testing tagging                             [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  tag * +new_tag        39.96	24.99	14.74	80048	16/282944
  tag * +existing_tag   0.00	0.00	0.00	8968	0/0
  tag * -existing_tag   29.98	18.18	11.70	20040	0/208264
  tag * -missing_tag    0.00	0.00	0.00	9036	0/0

T03-reindex.sh: Testing reindexing                      [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  reindex *             40.56	33.46	7.02	228908	0/459240
  reindex *             39.91	33.10	6.75	223768	112/378824
  reindex *             40.04	33.10	6.89	223724	0/334824

T04-thread-subquery.sh: Testing thread subqueries       [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  search thread:{} ...  0.29	0.26	0.02	23256	0/24
  search thread:{} ...  0.27	0.25	0.02	23296	0/24
  search thread:{} ...  0.29	0.27	0.02	23192	0/24


The only thing I can think of is my fstrim service runs 1x / week on Monday
at midnight and maybe that helped clean things up??  Perhaps I should
increase that frequency or run it manually when things go bad.

Thanks for the feedback!

Cheers,
Don

Re: performance problems with notmuch new

[Don Zickus]

On Mon, Apr 20, 2020 at 11:31:09AM -0300, David Bremner wrote:
> Don Zickus <dzickus@redhat.com> writes:
> 
> >
> > Hmm, for me --small was 35s and --medium was 32 minutes.  This is on a
> > i7-9750H / nvme.  I would expect numbers similar to yours.
> >
> 
> In addition to the breakdown of numbers that I posted, it would be
> potentially useful to know if it is I/O bound or CPU bound, and what
> kind of file system you are using.

I believe it is cpu bound as the cpu hits 95-100% while iotop should around
250k /sec reading and 50k/sec writing.  The filesystem is ext4 using device
mapper to mount it.

The other numbers are in my other reponse.

Hope that helps!

Cheers,
Don

Re: performance problems with notmuch new

[Franz Fellner]

On Mon Apr 20 11:36:36 2020, David Bremner <david@tethera.net> wrote:
> What kind of filesystem do you have on your spinning rust?

It's ext4 on both HDDs.

Re: performance problems with notmuch new

[Olly Betts]

On Mon, Apr 20, 2020 at 11:36:36AM -0300, David Bremner wrote:
> Franz Fellner <alpine.art.de@gmail.com> writes:
> 
> > I also suffer from bad performance of notmuch new.  I used notmuch
> > some years ago and notmuch new always felt instantanious.  Had to stop
> > using it because internet was too slow to sync my mails :/ Now (with
> > better internet and a completely new setup using mbsync) indexing one
> > mail takes at least 10 seconds, sometimes even more.  It can go into
> > minutes when I get lots of mail (~30...).

First question: what version of Xapian are you using?

And second thing to check, are you committing each message separately?

The commit operation tries to ensure that the data has actually been
written out to disk, so the time to index one message by itself isn't
indicative as it'll often mostly just be waiting for fdatasync() or
similar to return.

If you index 30 messages but commit each separately (i.e. run "notmuch
new" 30 times picking up one new message each time) that'll probably
scale something like linearly, but indexing a batch of 30 messages
should be much quicker per message.

> > When I run it after a
> > reboot I can have breakfast while notmuch starts up...  This is all on
> > spinning rust. I thought of getting an SSD but not in the near future.

After reboot the disk cache won't have any of the database in, so the
first operation will typically be slower, especially with a spinning
drive where seeks are relatively slow.

> > What I observe during that time: notmuch doesn't really need much CPU.
> > iotop shows constant read and write with extremely low rates, under
> > 1MB/sec.  So I think it might be an issue in xapian?
> 
> Just in case one of the xapian experts can suggest some kind of test for
> why you might be seeing this behaviour, I've included the xapian list in
> CC.

It sounds like you're seek-limited in this "cold cache" phase.  That is
not necessarily related to the slow indexing, but it could be.

I'd check the SMART diagnostics for the drive first (e.g. with
smartctl).  It's not the most likely cause, but it's quick to check and
if the drive is starting to fail it's better to find out sooner rather
than later.

Then I'd try compacting the database (I think there's a "notmuch
compact" subcommand to do this).
 
If that doesn't help, profiling the I/O would probably be my next
suggestion - there are some tools in the xapian git repo to help with
this (in xapian-maintainer-tools/profiling).  Under Linux I'd suggest
the strace ones (there's also an LD_PRELOAD library but it may need
tweaking for 32 vs 64 bit).

Cheers,
    Olly

Re: performance problems with notmuch new

[David Bremner]

Don Zickus <dzickus@redhat.com> writes:

>
> The only thing I can think of is my fstrim service runs 1x / week on Monday
> at midnight and maybe that helped clean things up??  Perhaps I should
> increase that frequency or run it manually when things go bad.
>

Is notmuch new still slow on your mail store, or did that mysteriously
improve as well?

d

Re: performance problems with notmuch new

[Franz Fellner]

On Thu Apr 23 00:21:30 2020, Olly Betts <olly@survex.com> wrote:
> First question: what version of Xapian are you using?

On my laptop it's 1.4.15 (arch linux) and the desktop runs 1.4.14 (Gentoo linux)

> And second thing to check, are you committing each message separately?

No, I sync with mbsync which dosnloads a bunch of mails,
then I run notmuch new which indexes all in one go.

> After reboot the disk cache won't have any of the database in, so the
> first operation will typically be slower, especially with a spinning
> drive where seeks are relatively slow.

Yes, I know that, I just wanted to mention the number, which IMO is insane.
I want to setup notmuch for my dad on the desktop PC.
5 minutes to wait for his mail in the morning would have made notmuch a no-go.
 
> It sounds like you're seek-limited in this "cold cache" phase.  That is
> not necessarily related to the slow indexing, but it could be.
> 
> I'd check the SMART diagnostics for the drive first (e.g. with
> smartctl).  It's not the most likely cause, but it's quick to check and
> if the drive is starting to fail it's better to find out sooner rather
> than later.

HDDs are healthy. I actually checked quite recently when converting 
the laptop from Gentoo to arch.

> 
> Then I'd try compacting the database (I think there's a "notmuch
> compact" subcommand to do this).

And there we go. Cured the issues.
Dropped the very first indexing from several minutes to 1.5 seconds on the desktop.
?!?!
This is a really new setup and I suffered from bad performance from the
very first notmuch new after the initial indexing.
Is it really needed to run notmch compact directly after the initial notmuch new?
Desktop currently has 38502 messages indexed, in case that matters.

Regards
Franz

Re: performance problems with notmuch new

[Don Zickus]

On Fri, Apr 24, 2020 at 07:36:23AM -0300, David Bremner wrote:
> Don Zickus <dzickus@redhat.com> writes:
> 
> >
> > The only thing I can think of is my fstrim service runs 1x / week on Monday
> > at midnight and maybe that helped clean things up??  Perhaps I should
> > increase that frequency or run it manually when things go bad.
> >
> 
> Is notmuch new still slow on your mail store, or did that mysteriously
> improve as well?

It improved slightly.  Perhaps my expectations are too high.  After about a
1000, the speed slows down significantly.  Outside of the performance
testing, after about 1000 emails, the processing slows down to about
10/second.

Of course I am using this in a strange context.  I am deleting emails
locally and then running 'notmuch new' to clean up the database.

notmuch search --output=files <filter> |xargs -l rm
notmuch new

and sometimes I delete a couple thousand emails from my mailing list folder
in neomutt and while exiting the folder it takes a long while to sync up
with notmuch.

So maybe my use case is unusual and the slowness is expected.

Cheers,
Don

Re: performance problems with notmuch new

[David Bremner]

Don Zickus <dzickus@redhat.com> writes:

>
> Of course I am using this in a strange context.  I am deleting emails
> locally and then running 'notmuch new' to clean up the database.
>
[snip]
> So maybe my use case is unusual and the slowness is expected.

I'd say it's a known performance bug in notmuch that it doesn't deal
well with large numbers of deletions. At some point it is faster to
reindex the whole database from scratch.

d

Re: performance problems with notmuch new

[David Bremner]

Franz Fellner <alpine.art.de@gmail.com> writes:
>
> On Thu Apr 23 00:21:30 2020, Olly Betts <olly@survex.com> wrote:

>> Then I'd try compacting the database (I think there's a "notmuch
>> compact" subcommand to do this).

> And there we go. Cured the issues.  Dropped the very first indexing
> from several minutes to 1.5 seconds on the desktop.  ?!?!  This is a
> really new setup and I suffered from bad performance from the very
> first notmuch new after the initial indexing.  Is it really needed to
> run notmch compact directly after the initial notmuch new?  Desktop
> currently has 38502 messages indexed, in case that matters.

That's a fairly small number of messages by the usual notmuch
standards. It's possible there is something pathological about the
interaction of notmuch and your particular set of mail. Running the
performance test suite on your machine would be one way of
telling. Also, the size of your notmuch database on disk; really huge
databases are a sign that something is going wrong. I think most people
find the database is 1/3 to 1/2 the size of the mail it is indexing, but
we have had some odd cases where the database is 2 or 3 times the mail,
and that was an indexing bug.

d

Re: performance problems with notmuch new

[Don Zickus]

On Fri, Apr 24, 2020 at 08:07:27PM -0300, David Bremner wrote:
> Don Zickus <dzickus@redhat.com> writes:
> 
> >
> > Of course I am using this in a strange context.  I am deleting emails
> > locally and then running 'notmuch new' to clean up the database.
> >
> [snip]
> > So maybe my use case is unusual and the slowness is expected.
> 
> I'd say it's a known performance bug in notmuch that it doesn't deal
> well with large numbers of deletions. At some point it is faster to
> reindex the whole database from scratch.

Does that mean 'notmuch dump' and 'notmuch restore'?  Willing to try that.

Cheers,
Don

Re: performance problems with notmuch new

[David Bremner]

Don Zickus <dzickus@redhat.com> writes:

> On Fri, Apr 24, 2020 at 08:07:27PM -0300, David Bremner wrote:
>> Don Zickus <dzickus@redhat.com> writes:
>> 
>> >
>> > Of course I am using this in a strange context.  I am deleting emails
>> > locally and then running 'notmuch new' to clean up the database.
>> >
>> [snip]
>> > So maybe my use case is unusual and the slowness is expected.
>> 
>> I'd say it's a known performance bug in notmuch that it doesn't deal
>> well with large numbers of deletions. At some point it is faster to
>> reindex the whole database from scratch.
>
> Does that mean 'notmuch dump' and 'notmuch restore'?  Willing to try that.
>

Exactly.

David

Re: performance problems with notmuch new

[Eric Wong]

David Bremner <david@tethera.net> wrote:
> Franz Fellner <alpine.art.de@gmail.com> writes:
> > mail takes at least 10 seconds, sometimes even more.  It can go into
> > minutes when I get lots of mail (~30...).  When I run it after a
> > reboot I can have breakfast while notmuch starts up...  This is all on
> > spinning rust. I thought of getting an SSD but not in the near future.
> 
> I do have at least one spinning rust configuration with about 300k
> messages, and notmuch new is still fast there.

I've yet to figure out how spinning rust can work well with
giant public-inboxes (git + Xapian + SQLite); but I have
a fair bit of experience with SSDs + Xapian.

But some of my recommendations below come from my experience
with HDDs in the old days, before I used Xapian.

> > What I observe during that time: notmuch doesn't really need much CPU.
> > iotop shows constant read and write with extremely low rates, under
> > 1MB/sec.  So I think it might be an issue in xapian?

Seek times, probably   `iostat -x 1' can give you very useful
information about I/O queue sizes and wait times for reads and
writes (the `-x' is the good stuff :), `1' means it keeps
outputting every second.

> Just in case one of the xapian experts can suggest some kind of test for
> why you might be seeing this behaviour, I've included the xapian list in
> CC.

Newer Xapian has a DB_NO_SYNC which notmuch could set as an
option.  Users of old Xapian (or on Perl XS bindings) also have
libeatmydata LD_PRELOAD which I end up using all the time:

	https://www.flamingspork.com/projects/libeatmydata/

I run `sync' if I have anything important, but I usually
don't ;)   I do set the kernel do flush dirty data in the
background fairly aggressively, though (more below)

For public-inbox v2 hacking in 2018 (indexing LKML archives, ~3M
messages), I found working on a freshly TRIM-ed SSD with plenty
of free space made the SSD firmware happier.  SSDs can get a LOT
slower as they get fuller (so xapian-compact helps, there, too).

SSD quality matters a lot; but even the low-end QLC stuff beats
high-end HDDs in random I/O; but they will slow down more as
they fill up more.

For writes, I set /proc/sys/vm/dirty_background_bytes to 100M or
something reasonably close to what the SSD can write quickly.
Linux tended to hit I/O stalls with lots of dirty data, so
making the kernel flush it sooner tends to help IME.  Maybe newer
kernels do better *shrug*; but it's basically the local storage
version of the network "Bufferbloat" problem.

Flushing dirty data more frequently also frees up more memory
for the kernel to make better caching decisions about
future/current data it needs to read.

notmuch can probably run a background thread (or use liburing)
to do POSIX_FADV_DONTNEED once its done with a message, too (and
POSIX_FADV_WILLNEED for to-be-indexed messages).  Uncompressed
Maildir messages eat cache space real quick, which means less
cache for Xapian.

public-inbox indexes the v2 inbox format in parallel; but
excessive parallelism still causes I/O contention with SSDs (at
least upper-mid-range ones).  So right now the default limit is
3 indexing processes regardless of CPU count.  Reading from git
is still synchronous atm, but will probably be async in a few
months.  git itself tends to generate decent I/O patterns with
its pack format (but makes posix_fadvise hinting impractical).

Anyways, indexing just under 3 million LKML messages took ~4
hours on 4-core system built in 2010 with a SATA SSD from 2014.