Re: notmuch killed due to out of memory - how to move forward

[Kushal Kumaran <kushal.kumaran@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]

Jani Nikula <jani@nikula.org> writes:

> On Mon, 03 Feb 2014, Hamish Downer <hamish@foobacca.co.uk> wrote:
>> Hello
>>
>> I recently deleted almost 30000 old messages from my maildirs, and
>> since I did that, notmuch new has not managed to complete.  I have it
>> running on a server with 1 GB of RAM, and the output is telling me
>>
>>     Cleaned up 25515 of 29803 messages (1m 10s remaining).
>>
>> at the point it is killed.  I have followed it up to this point with
>> top and free and I can see memory usage growing to something like
>> 700MB (RES) at which point the process is killed.
>>
>> I am using notmuch 0.17 (from the debian jessie package 0.17-3) on
>> debian wheezy 64 bit.
>>
>> Any pointers as to what to do to get past this point would be greatly
>> appreciated.  If I have just deleted that many messages should I just
>> expect notmuch to use that much memory?  (Unfortunately the company
>> supplying the server does not allow swap to be enabled so I can't fudge
>> it that way)
>>
>> I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't
>> produce a report (though I was unsure if I needed to rebuild notmuch
>> before that would work).
>>
>> Happy to build notmuch from source if this might be an interesting bug
>> worth tracking down.
>
> Please try SIGINT, or ^C, on notmuch new before it gets killed. That
> should be handled gracefully, making progress, and letting you chop up
> and eventually finish the operation. Please let us know if this helps.
>
> Needless to say, we shouldn't use that much memory just to delete files
> from the index.
>

When I initially indexed my email archives into notmuch, I had problems
with thermal cutouts, so I did something like this, which automated the
workaround you mention:

while :; do
    timeout -s INT 2m notmuch new
    sleep 600
done

Of course, with memory issues as the constraint rather than temperature,
the sleep should not be necessary.

-- 
regards,
kushal

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]