From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id B53DA431FBC for ; Tue, 4 Feb 2014 01:01:22 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.799 X-Spam-Level: X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4TtVlK3eIWbN for ; Tue, 4 Feb 2014 01:01:15 -0800 (PST) Received: from mail-yk0-f172.google.com (mail-yk0-f172.google.com [209.85.160.172]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 0C8C6431FAF for ; Tue, 4 Feb 2014 01:01:14 -0800 (PST) Received: by mail-yk0-f172.google.com with SMTP id 200so20714687ykr.3 for ; Tue, 04 Feb 2014 01:01:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:subject:in-reply-to:references:user-agent:date:message-id :mime-version:content-type; bh=AJUu7+gxvVgzYpgIaFMXEFkHVTA8W++gTYZCywtnGRA=; b=E2Unmpa7yZqWNrp9QN8HKPamgZydFt3zph+/lrCAadzTx1CJ1NqkfJskwiJbNIiOrs OEqgAI8dCMRFAM0oKBzFHGYrBQggs1UmGPSF4vjtTch2GfdrTRpID+8F3fGbo1XutHir 0NCXqD6JRlzPahy/mmThia8BCa9e90kWT4n4dQgfJZIC9/99BzsM87CnWKI46azndvfv 08VrWXG+UQmRyMdBbpvPLDY9ZczUIhXCWybRbYRD7js5bL/+QNbq5w1GSgQTL1tA3PRR NsCpQmZXoqkI6Hn95Y4fxgUk2wqd4VCBLxRXZHS5CjWpa4dC2OOx2tWqowpZaM6AkBPH lbMA== X-Received: by 10.236.151.198 with SMTP id b46mr38141076yhk.3.1391504473164; Tue, 04 Feb 2014 01:01:13 -0800 (PST) Received: from magnesium ([198.199.83.138]) by mx.google.com with ESMTPSA id v41sm20730223yhi.19.2014.02.04.01.01.11 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Feb 2014 01:01:12 -0800 (PST) Received: by magnesium (Postfix, from userid 1000) id 330404503E; Tue, 4 Feb 2014 14:31:11 +0530 (IST) From: Kushal Kumaran To: notmuch@notmuchmail.org Subject: Re: notmuch killed due to out of memory - how to move forward In-Reply-To: <87r47jcmj6.fsf@nikula.org> References: <87r47jcmj6.fsf@nikula.org> User-Agent: Notmuch/0.17+35~g3b36898 (http://notmuchmail.org) Emacs/24.3.1 (x86_64-pc-linux-gnu) Date: Tue, 04 Feb 2014 14:31:10 +0530 Message-ID: <52f0ac58.4188ec0a.7ff8.ffff9762@mx.google.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha1; protocol="application/pgp-signature" X-Mailman-Approved-At: Tue, 04 Feb 2014 09:26:02 -0800 X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Feb 2014 09:01:22 -0000 --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Jani Nikula writes: > On Mon, 03 Feb 2014, Hamish Downer wrote: >> Hello >> >> I recently deleted almost 30000 old messages from my maildirs, and >> since I did that, notmuch new has not managed to complete. I have it >> running on a server with 1 GB of RAM, and the output is telling me >> >> Cleaned up 25515 of 29803 messages (1m 10s remaining). >> >> at the point it is killed. I have followed it up to this point with >> top and free and I can see memory usage growing to something like >> 700MB (RES) at which point the process is killed. >> >> I am using notmuch 0.17 (from the debian jessie package 0.17-3) on >> debian wheezy 64 bit. >> >> Any pointers as to what to do to get past this point would be greatly >> appreciated. If I have just deleted that many messages should I just >> expect notmuch to use that much memory? (Unfortunately the company >> supplying the server does not allow swap to be enabled so I can't fudge >> it that way) >> >> I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't >> produce a report (though I was unsure if I needed to rebuild notmuch >> before that would work). >> >> Happy to build notmuch from source if this might be an interesting bug >> worth tracking down. > > Please try SIGINT, or ^C, on notmuch new before it gets killed. That > should be handled gracefully, making progress, and letting you chop up > and eventually finish the operation. Please let us know if this helps. > > Needless to say, we shouldn't use that much memory just to delete files > from the index. > When I initially indexed my email archives into notmuch, I had problems with thermal cutouts, so I did something like this, which automated the workaround you mention: while :; do timeout -s INT 2m notmuch new sleep 600 done Of course, with memory issues as the constraint rather than temperature, the sleep should not be necessary. =2D-=20 regards, kushal --=-=-= Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) iQEcBAEBAgAGBQJS8KxWAAoJEKolkdITDqLt8bEIALOGQuoW+tiREnDcM4EFJ7FS OWnEOi6aqxHTQoIY6ADQ3dm1a6p2albi8QY4jMlNVC0ODJiRFd8hExcIkUXlU2hU cNA8KOKcM4ILEVrzjoies7yJLiibCx/mxRtSyGe8HC78z0j58C/HT9W4cl3fxcQA k1ImqsVeY56VkOAClPUeUT8jwseZIMIbZTlq735HfkLk+gHwBkrCHeIMdjFf7eWg PfLQXaEnkq5jNFvG8jVf4JI+BADYr+IOi507yiynMhdtxf5zHTumn2m+qLxw6+tW +Nnv2NXwSwdTW0Z7HHQYRcTfHZql5x//4tNWzgJyMp/w3a2tljfdcf4JEo6p6CE= =oqgO -----END PGP SIGNATURE----- --=-=-=--