compacting the notmuch database through systemd

compacting the notmuch database through systemd

[Antoine Beaupré]

[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]

Hi!

Thanks to Bremner, I just realized that notmuch-compact(1) is a thing,
and that thing allows me to compress my notmuch databases by about 50%.

So I whipped together two systemd units (attached) that will run that
command every month on my notmuch database. Just drop them in
`~/.config/systemd/user/` and run:

    systemctl --user daemon-reload
    systemctl --user enable notmuch-compact.timer
    systemctl --user start notmuch-compact.timer

... and you're done.

Maybe those could be shipped with the Debian package somehow? Not sure
how that works, but I think that's how gpg-agent gets started now, if
you want any inspiration...

It would be great if notmuch-new ran this on its own, when it
thought that this was "important", somehow like git-gc sometimes runs on
its own.

Of course, I suspect the 50% saving will not happen every month, but
hopefully this will keep my database size from exploding in the
future...

Finally, Bremner said that "it could make some operations slower to
compact the database, but I haven't notced that effect". That is strange
because the manpage says that compact "can both reduce the space
required by the database and improve lookup performance". ;) Hopefully,
the documentation is true and the rumors are false and the sky is not
falling and I want my mommy.

A.

PS: any humor or comedy in this email is purely accidental and shouldn't
be construed as a feeble attempt at making you smile.

-- 
I prefer the tumult of liberty to the quiet of servitude.
                        - Thomas Jefferson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: compacting the notmuch database through systemd

[Antoine Beaupré]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

On 2019-12-01 15:52:19, Antoine Beaupré wrote:
> PS: any humor or comedy in this email is purely accidental and shouldn't
> be construed as a feeble attempt at making you smile.

The joke is on me, I guess. I thought I had written some hooks to ever
prevent this from happening again, but it looks like either (a) they
didn't work in this case or (b) they are disabled because I forgot the
attachments...

A.

-- 
There are two ways of constructing a software design: One way is to
make it so simple that there are obviously no deficiencies, and the
other way is to make it so complicated that there are no obvious
deficiencies. The first method is far more difficult.
                        - C.A.R. Hoare

[-- Attachment #2: notmuch-compact.timer --]
[-- Type: text/plain, Size: 110 bytes --]

[Unit]
Description=compact the notmuch database

[Timer]
OnCalendar=monthly

[Install]
WantedBy=timers.target

[-- Attachment #3: notmuch-compact.service --]
[-- Type: text/plain, Size: 114 bytes --]

[Unit]
Description=compact the notmuch database

[Service]
Type=simple
Nice=10
ExecStart=/usr/bin/notmuch compact

Re: compacting the notmuch database through systemd

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 3651 bytes --]

Thanks for raising this, Anarcat!

One more advantage that i think you haven't noted yet about regular
database compaction:

"notmuch compact" tends to get rid of a lot of lingering written data
that is no longer referenced.  While this isn't robust "secure
deletion", it's a lot better than not compacting.  see
https://trac.xapian.org/ticket/742 for more discussion.

Some questions below…

On Sun 2019-12-01 15:52:19 -0500, Antoine Beaupré wrote:

> Thanks to Bremner, I just realized that notmuch-compact(1) is a thing,
> and that thing allows me to compress my notmuch databases by about 50%.

do you know why you get the large size/speed gain?  do you regularly
delete files from your message archive?

> So I whipped together two systemd units (attached) that will run that
> command every month on my notmuch database. Just drop them in
> `~/.config/systemd/user/` and run:
>
>     systemctl --user daemon-reload
>     systemctl --user enable notmuch-compact.timer
>     systemctl --user start notmuch-compact.timer

("systemctl --user enable --now notmuch-compact.timer" will suffice for
the final two commands on any reasonably modern version of systemd)

How long does it take for these the notmuch-compact.service to complete?

What happens if this is happening when, say, you put your machine to
sleep, or you power it down?

While notmuch-compact.service is running, does "notmuch new" or "notmuch
insert" work?  If not, how do they fail (e.g. blocking indefinitely,
returning a comprehensible error message)?

Can you read your mail while notmuch-compact.service is running?

> Maybe those could be shipped with the Debian package somehow? Not sure
> how that works, but I think that's how gpg-agent gets started now, if
> you want any inspiration...

gpg-agent is socket-activated, which is different from the
timer-activation you are proposing here.

We could easily ship these systemd user unit files in the notmuch
package now that #764678 is resolved.  Do you think that the timer
should be enabled by default?

What should happen if the user hasn't set up notmuch?  Maybe we need a
ConditionPathExists= or something like that on either the .timer or the
.service?

Do we expect this to run even when the user isn't logged in at all (a
background compaction?)

it always gets more complex when you think about trying to do it at
scale :)

> It would be great if notmuch-new ran this on its own, when it
> thought that this was "important", somehow like git-gc sometimes runs on
> its own.

I'm not convinced i like this idea without more profiling and an
understanding of what it might cause.  I have grown to *really* dislike
the highly variable latency and warnings caused by GnuPG's
"auto-check-trustdb", for example (especially as the keyring grows
larger).

>  [ notmuch-compact.timer: text/plain ]
>  [Unit]
>  Description=compact the notmuch database

systemd timer unit descriptions typically include some mention of the
duration.  See for example:

/lib/systemd/system/systemd-tmpfiles-clean.timer
"Daily Cleanup of Temporary Directories"

/lib/systemd/system/certbot.timer
"Run certbot twice daily"

/lib/systemd/system/phpsessionclean.timer
"Clean PHP session files every 30 mins"

I recommend:

    Description=Compact the notmuch database every month

> [ notmuch-compact.service: text/plain ]
> [Unit]
> Description=compact the notmuch database

The convention is to lead with an upper-case letter:

    Description=Compact the notmuch database

OK OK enough with the nit-picking!

Happy hacking,

    --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: compacting the notmuch database through systemd

[Antoine Beaupré]

On 2019-12-04 13:09:03, Daniel Kahn Gillmor wrote:
> Thanks for raising this, Anarcat!
>
> One more advantage that i think you haven't noted yet about regular
> database compaction:
>
> "notmuch compact" tends to get rid of a lot of lingering written data
> that is no longer referenced.  While this isn't robust "secure
> deletion", it's a lot better than not compacting.  see
> https://trac.xapian.org/ticket/742 for more discussion.

Cool.

> Some questions below…
>
> On Sun 2019-12-01 15:52:19 -0500, Antoine Beaupré wrote:
>
>> Thanks to Bremner, I just realized that notmuch-compact(1) is a thing,
>> and that thing allows me to compress my notmuch databases by about 50%.
>
> do you know why you get the large size/speed gain?

Not sure, but if I'd venture a guess: I never ran notmuch-compact(1) as
far as I can remember.

> do you regularly delete files from your message archive?

Yes, all the time. I have had `d` mapped to `+deleted` basically
forever, and have a pre-new hook that actually deletes those messages
from this.

Yes, I am an heretic. ;)

>> So I whipped together two systemd units (attached) that will run that
>> command every month on my notmuch database. Just drop them in
>> `~/.config/systemd/user/` and run:
>>
>>     systemctl --user daemon-reload
>>     systemctl --user enable notmuch-compact.timer
>>     systemctl --user start notmuch-compact.timer
>
> ("systemctl --user enable --now notmuch-compact.timer" will suffice for
> the final two commands on any reasonably modern version of systemd)

Whoa. TIL.

> How long does it take for these the notmuch-compact.service to complete?

I don't remember... it took less than a minute at the first run, I
think.

> What happens if this is happening when, say, you put your machine to
> sleep, or you power it down?

No idea. I think it's an atomic process as notmuch-compact(1) says:

       The compacted database is built in a temporary directory and is
       later moved into the place of the origin database. The original
       uncompacted database is discarded, unless the
       --backup=<directory> option is used.

> While notmuch-compact.service is running, does "notmuch new" or "notmuch
> insert" work?  If not, how do they fail (e.g. blocking indefinitely,
> returning a comprehensible error message)?

No idea. Manpage says:

       Note that the database write lock will be held during the
       compaction process (which may be quite long) to protect data
       integrity.

> Can you read your mail while notmuch-compact.service is running?

I don't see why not, but I haven't tried. Considering I run it once a
week, it would seem like a small tradeoff if that would cause problems
anyways.

>> Maybe those could be shipped with the Debian package somehow? Not sure
>> how that works, but I think that's how gpg-agent gets started now, if
>> you want any inspiration...
>
> gpg-agent is socket-activated, which is different from the
> timer-activation you are proposing here.

I thought about socket activation, but I don't think it would work in
this case.

> We could easily ship these systemd user unit files in the notmuch
> package now that #764678 is resolved.  Do you think that the timer
> should be enabled by default?

Sure, I don't see why not, unless we have concerns about
notmuch-compact(1) being unsafe or counter-productive.

> What should happen if the user hasn't set up notmuch?  Maybe we need a
> ConditionPathExists= or something like that on either the .timer or the
> .service?

Maybe:

    ConditionPathExists=$HOME/.notmuch-config

?

> Do we expect this to run even when the user isn't logged in at all (a
> background compaction?)

Maybe not? No idea.

> it always gets more complex when you think about trying to do it at
> scale :)

Yes.

>> It would be great if notmuch-new ran this on its own, when it
>> thought that this was "important", somehow like git-gc sometimes runs on
>> its own.
>
> I'm not convinced i like this idea without more profiling and an
> understanding of what it might cause.  I have grown to *really* dislike
> the highly variable latency and warnings caused by GnuPG's
> "auto-check-trustdb", for example (especially as the keyring grows
> larger).

Again, tradeoffs: I prefer to have my trustdb actually checked once in a
while (right?) and not pay that latency cost at some random gpg
invocation (which seems to happen all the time). So I disable the
built-in, inline checks and queue them in a timer instead.

>>  [ notmuch-compact.timer: text/plain ]
>>  [Unit]
>>  Description=compact the notmuch database
>
> systemd timer unit descriptions typically include some mention of the
> duration.  See for example:
>
> /lib/systemd/system/systemd-tmpfiles-clean.timer
> "Daily Cleanup of Temporary Directories"
>
> /lib/systemd/system/certbot.timer
> "Run certbot twice daily"
>
> /lib/systemd/system/phpsessionclean.timer
> "Clean PHP session files every 30 mins"
>
> I recommend:
>
>     Description=Compact the notmuch database every month

Cool.

>> [ notmuch-compact.service: text/plain ]
>> [Unit]
>> Description=compact the notmuch database
>
> The convention is to lead with an upper-case letter:
>
>     Description=Compact the notmuch database

Yay!

> OK OK enough with the nit-picking!

Thanks for the review!

a.

-- 
L'adversaire d'une vraie liberté est un désir excessif de sécurité.
                        - Jean de la Fontaine

Re: compacting the notmuch database through systemd

[Jorge P. de Morais Neto]

Hello.

Em [2019-12-04 qua 14:51:12-0500], Antoine Beaupré escreveu:

>> What should happen if the user hasn't set up notmuch?  Maybe we need a
>> ConditionPathExists= or something like that on either the .timer or the
>> .service?
>
> Maybe:
>
>     ConditionPathExists=$HOME/.notmuch-config
>
> ?

But the user might have changed ~NOTMUCH_CONFIG~.  On my Debian install
it is "~/.config/notmuch/config" (XDG compliant).

Regards
-- 
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- I am Brazilian.  I hope my English is correct and I welcome feedback.
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: compacting the notmuch database through systemd

[Georg Faerber]

On 19-12-05 08:29:44, Jorge P. de Morais Neto wrote:
> But the user might have changed ~NOTMUCH_CONFIG~. On my Debian
> install it is "~/.config/notmuch/config" (XDG compliant).

It's possible to specify multiple conditions, and join these via OR:

ConditionPathExists=|~/.notmuch-config
ConditionPathExists=|~/.config/notmuch/config

Re: compacting the notmuch database through systemd

[Antoine Beaupré]

On 2019-12-05 08:29:44, Jorge P. de Morais Neto wrote:
> Hello.
>
> Em [2019-12-04 qua 14:51:12-0500], Antoine Beaupré escreveu:
>
>>> What should happen if the user hasn't set up notmuch?  Maybe we need a
>>> ConditionPathExists= or something like that on either the .timer or the
>>> .service?
>>
>> Maybe:
>>
>>     ConditionPathExists=$HOME/.notmuch-config
>>
>> ?
>
> But the user might have changed ~NOTMUCH_CONFIG~.  On my Debian install
> it is "~/.config/notmuch/config" (XDG compliant).

Then you get what you desserve? :p You can still override those in your
local config...

I'm not sure how else to fix this, really...

A.

-- 
Secrecy is the keystone to all tyranny. Not force, but secrecy and
censorship.
                       -  Robert A. Heinlein

Re: compacting the notmuch database through systemd

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1288 bytes --]

On Thu 2019-12-05 11:38:09 +0000, Georg Faerber wrote:
> On 19-12-05 08:29:44, Jorge P. de Morais Neto wrote:
>> But the user might have changed ~NOTMUCH_CONFIG~. On my Debian
>> install it is "~/.config/notmuch/config" (XDG compliant).
>
> It's possible to specify multiple conditions, and join these via OR:
>
> ConditionPathExists=|~/.notmuch-config
> ConditionPathExists=|~/.config/notmuch/config

I think we want to use %h instead of ~, but i haven't done much testing
(see SPECIFIERS in systemd.unit).

As for the variations, i'm kind of of a mind that people who deviate
from the defaults should need to maintain their own deviance explicitly,
rather than trying to accomodate a non-default path in shipped config
files.  For example, "systemctl --user edit notmuch-compact.timer" helps
you drop in the appropriate override/modification.

for that matter, do we want ConditionPathExists to constrain the .timer
or the .service (or both)?

And, what if the config file exists but the database has never been
created?

I've always felt like it was a bit weird that "notmuch setup" would
write the config file without at least initializing the database (though
perhaps this falls back into my general config file vs. database
concerns)

        --dkg


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: compacting the notmuch database through systemd

[Jorge P. de Morais Neto]

Em [2019-12-06 sex 14:41:22-0500], Daniel Kahn Gillmor escreveu:

> As for the variations, i'm kind of of a mind that people who deviate
> from the defaults should need to maintain their own deviance explicitly,
> rather than trying to accomodate a non-default path in shipped config
> files.  For example, "systemctl --user edit notmuch-compact.timer" helps
> you drop in the appropriate override/modification.

But ~/.config is the XDG standard, so maybe it deserves special
consideration.

Regards
-- 
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- I am Brazilian.  I hope my English is correct and I welcome feedback.
- Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

Re: compacting the notmuch database through systemd

[Chris Coutinho]

On Sun, Dec 8, 2019 at 3:38 PM Jorge P. de Morais Neto
<jorge+list@disroot.org> wrote:
>
> Em [2019-12-06 sex 14:41:22-0500], Daniel Kahn Gillmor escreveu:
>
> > As for the variations, i'm kind of of a mind that people who deviate
> > from the defaults should need to maintain their own deviance explicitly,
> > rather than trying to accomodate a non-default path in shipped config
> > files.  For example, "systemctl --user edit notmuch-compact.timer" helps
> > you drop in the appropriate override/modification.
>
> But ~/.config is the XDG standard, so maybe it deserves special
> consideration.
>
> Regards
> --
> - <https://jorgemorais.gitlab.io/justice-for-rms/>
> - I am Brazilian.  I hope my English is correct and I welcome feedback.
> - Please adopt free formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
> - Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
> - [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch

I execute `notmuch new` after an `offlineimap` sync using `systemd`,
and the way I handle non-standard paths is by specifying the
NOTMUCH_CONFIG as an environmental variable like this:

    $ cat ~/.config/systemd/user/offlineimap-oneshot.service.d/override.conf
    [Service]
    Environment="NOTMUCH_CONFIG=/home/chris/.config/notmuch/config"
    ExecStartPost=/usr/bin/notmuch new

In this case I'm using `systemd` override functionality - you might be
able to use it for compacting your `notmuch` database using `systemd`
as well.

Regards,
Chris

Re: compacting the notmuch database through systemd

[David Bremner]

Jorge P. de Morais Neto <jorge+list@disroot.org> writes:

> Em [2019-12-06 sex 14:41:22-0500], Daniel Kahn Gillmor escreveu:
>
>> As for the variations, i'm kind of of a mind that people who deviate
>> from the defaults should need to maintain their own deviance explicitly,
>> rather than trying to accomodate a non-default path in shipped config
>> files.  For example, "systemctl --user edit notmuch-compact.timer" helps
>> you drop in the appropriate override/modification.
>
> But ~/.config is the XDG standard, so maybe it deserves special
> consideration.
>

It would make more sense to have notmuch actually support
XDG_CONFIG_DIR. Alas, config changes seems to be hard / not rewarding.

Re: compacting the notmuch database through systemd

[Tomi Ollila]

On Sun, Dec 08 2019, David Bremner wrote:

> Jorge P. de Morais Neto <jorge+list@disroot.org> writes:
>
>> Em [2019-12-06 sex 14:41:22-0500], Daniel Kahn Gillmor escreveu:
>>
>>> As for the variations, i'm kind of of a mind that people who deviate
>>> from the defaults should need to maintain their own deviance explicitly,
>>> rather than trying to accomodate a non-default path in shipped config
>>> files.  For example, "systemctl --user edit notmuch-compact.timer" helps
>>> you drop in the appropriate override/modification.
>>
>> But ~/.config is the XDG standard, so maybe it deserves special
>> consideration.
>>
>
> It would make more sense to have notmuch actually support
> XDG_CONFIG_DIR. Alas, config changes seems to be hard / not rewarding.

This has been tried quite a few times (also) in notmuch mailing list so
far; search 'config and notmuch and xdg' in my active mailstorage shows:

first mentioned 2011-06-24
first RFC suggestion: 2011-12-21
so far last WIP patch (from bremner): 2017-12-30

"simplest" would be just get rid of the config file and use only database...

Tomi