notmuch killed due to out of memory - how to move forward

notmuch killed due to out of memory - how to move forward

[Hamish Downer]

Hello

I recently deleted almost 30000 old messages from my maildirs, and
since I did that, notmuch new has not managed to complete.  I have it
running on a server with 1 GB of RAM, and the output is telling me

    Cleaned up 25515 of 29803 messages (1m 10s remaining).

at the point it is killed.  I have followed it up to this point with
top and free and I can see memory usage growing to something like
700MB (RES) at which point the process is killed.

I am using notmuch 0.17 (from the debian jessie package 0.17-3) on
debian wheezy 64 bit.

Any pointers as to what to do to get past this point would be greatly
appreciated.  If I have just deleted that many messages should I just
expect notmuch to use that much memory?  (Unfortunately the company
supplying the server does not allow swap to be enabled so I can't fudge
it that way)

I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't
produce a report (though I was unsure if I needed to rebuild notmuch
before that would work).

Happy to build notmuch from source if this might be an interesting bug
worth tracking down.

Hamish

(PS sorry to moderators - I subscribed and then sent this email from the
wrong address a couple of times)

Re: notmuch killed due to out of memory - how to move forward

[Jani Nikula]

On Mon, 03 Feb 2014, Hamish Downer <hamish@foobacca.co.uk> wrote:
> Hello
>
> I recently deleted almost 30000 old messages from my maildirs, and
> since I did that, notmuch new has not managed to complete.  I have it
> running on a server with 1 GB of RAM, and the output is telling me
>
>     Cleaned up 25515 of 29803 messages (1m 10s remaining).
>
> at the point it is killed.  I have followed it up to this point with
> top and free and I can see memory usage growing to something like
> 700MB (RES) at which point the process is killed.
>
> I am using notmuch 0.17 (from the debian jessie package 0.17-3) on
> debian wheezy 64 bit.
>
> Any pointers as to what to do to get past this point would be greatly
> appreciated.  If I have just deleted that many messages should I just
> expect notmuch to use that much memory?  (Unfortunately the company
> supplying the server does not allow swap to be enabled so I can't fudge
> it that way)
>
> I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't
> produce a report (though I was unsure if I needed to rebuild notmuch
> before that would work).
>
> Happy to build notmuch from source if this might be an interesting bug
> worth tracking down.

Please try SIGINT, or ^C, on notmuch new before it gets killed. That
should be handled gracefully, making progress, and letting you chop up
and eventually finish the operation. Please let us know if this helps.

Needless to say, we shouldn't use that much memory just to delete files
from the index.


BR,
Jani.

Re: notmuch killed due to out of memory - how to move forward

[Kushal Kumaran]

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]

Jani Nikula <jani@nikula.org> writes:

> On Mon, 03 Feb 2014, Hamish Downer <hamish@foobacca.co.uk> wrote:
>> Hello
>>
>> I recently deleted almost 30000 old messages from my maildirs, and
>> since I did that, notmuch new has not managed to complete.  I have it
>> running on a server with 1 GB of RAM, and the output is telling me
>>
>>     Cleaned up 25515 of 29803 messages (1m 10s remaining).
>>
>> at the point it is killed.  I have followed it up to this point with
>> top and free and I can see memory usage growing to something like
>> 700MB (RES) at which point the process is killed.
>>
>> I am using notmuch 0.17 (from the debian jessie package 0.17-3) on
>> debian wheezy 64 bit.
>>
>> Any pointers as to what to do to get past this point would be greatly
>> appreciated.  If I have just deleted that many messages should I just
>> expect notmuch to use that much memory?  (Unfortunately the company
>> supplying the server does not allow swap to be enabled so I can't fudge
>> it that way)
>>
>> I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't
>> produce a report (though I was unsure if I needed to rebuild notmuch
>> before that would work).
>>
>> Happy to build notmuch from source if this might be an interesting bug
>> worth tracking down.
>
> Please try SIGINT, or ^C, on notmuch new before it gets killed. That
> should be handled gracefully, making progress, and letting you chop up
> and eventually finish the operation. Please let us know if this helps.
>
> Needless to say, we shouldn't use that much memory just to delete files
> from the index.
>

When I initially indexed my email archives into notmuch, I had problems
with thermal cutouts, so I did something like this, which automated the
workaround you mention:

while :; do
    timeout -s INT 2m notmuch new
    sleep 600
done

Of course, with memory issues as the constraint rather than temperature,
the sleep should not be necessary.

-- 
regards,
kushal

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: notmuch killed due to out of memory - how to move forward

[Hamish Downer]

On 4 February 2014 08:25, Jani Nikula <jani@nikula.org> wrote:
> On Mon, 03 Feb 2014, Hamish Downer <hamish@foobacca.co.uk> wrote:
>> I recently deleted almost 30000 old messages from my maildirs, and
>> since I did that, notmuch new has not managed to complete.  I have it
>> running on a server with 1 GB of RAM, and the output is telling me
>>
>>     Cleaned up 25515 of 29803 messages (1m 10s remaining).
>>
>> at the point it is killed.  I have followed it up to this point with
>> top and free and I can see memory usage growing to something like
>> 700MB (RES) at which point the process is killed.
>
> Please try SIGINT, or ^C, on notmuch new before it gets killed. That
> should be handled gracefully, making progress, and letting you chop up
> and eventually finish the operation. Please let us know if this helps.

That did it, thank you. Good to know Ctrl-C is gracefully handled.

> Needless to say, we shouldn't use that much memory just to delete files
> from the index.

I have saved a copy of the index from before I fixed the problem.  I am
happy to re-run it with a rebuilt notmuch if that would be useful from the
point of view of uncovering bugs.

Otherwise I'll carry on as a happy notmuch user
Hamish