From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 25E7E431FBD for ; Tue, 11 Dec 2012 10:26:47 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.799 X-Spam-Level: X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BD7mH5h8VfnV for ; Tue, 11 Dec 2012 10:26:46 -0800 (PST) Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com [209.85.212.175]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 51B15431FBC for ; Tue, 11 Dec 2012 10:26:46 -0800 (PST) Received: by mail-wi0-f175.google.com with SMTP id hm11so2094754wib.2 for ; Tue, 11 Dec 2012 10:26:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:content-disposition:to:from:in-reply-to :cc:references:message-id:user-agent:subject:date; bh=R9pZyJkU0CFmldU2joZA0jvKor0C37/xDY78a1vH49Q=; b=gIxScFvlbW9Wuti5xZpTjEk8Q3kLDdTCGfqZ17qMGcgKVa70ElhlJLq0hex0jFChZF oqyIN5Klmv8vvmt0Sg8rs2uMDcGZhQiWM8fdtynmC3UWfs9pAXnrt9pThsGsAlmHjnUV wUcfEy4kSLpTx+76F5lk/LiQwDgGRIcsUqZbA89bK+POA9SG6QJxxh7SbLQZNWvcvRAm TVpzfr2XRB8e8XKEOjzq+U0dKuCpLaSJjxy6Rd7OsrUHVmxKp7HXKt7ff1+w8MwU7I1j GfSv9cdv5iRz1c+EWExRkhiRONks6hX7dHAel2ym1w+1JqwS0VBpBl2YrLwF3pT03lH2 CB9Q== Received: by 10.194.82.2 with SMTP id e2mr2122285wjy.39.1355250402531; Tue, 11 Dec 2012 10:26:42 -0800 (PST) Received: from localhost (cpc14-sgyl28-2-0-cust332.18-2.cable.virginmedia.com. [82.39.85.77]) by mx.google.com with ESMTPS id bd7sm17437419wib.8.2012.12.11.10.26.39 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 11 Dec 2012 10:26:40 -0800 (PST) Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="pgp-sha1"; boundary="===============7418330742985369319==" MIME-Version: 1.0 Content-Disposition: inline To: "Jason A. Donenfeld" , From: Patrick Totzke In-Reply-To: References: Message-ID: <20121211182638.27237.98903@brick.lan> User-Agent: alot/0.3.3+ Subject: Re: notmuch python bindings corrupt db index (was: gmail importer script) Date: Tue, 11 Dec 2012 18:26:38 +0000 Cc: notmuch@notmuchmail.org X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Dec 2012 18:26:47 -0000 --===============7418330742985369319== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Quoting Jason A. Donenfeld (2012-12-11 13:47:32) > On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke > wrote: > > I ended up killing the process after i saw that ETA was >3h. > = > Gmail's bottleneck. Very unfortunate. Leave it going over night. OK, i assumed i'd not have to download all messages because i already have = them on disk -- synced via offlineimap. but then i remembered that OI stores uses its own naming scheme for the files in a maildir. so thats probably why you= r script did not realize theyre already there. (maybe it should? it could look up th= e MIDs from notmuch) > > This broke my index! > > > > afterwards, a `notmuch new` printed > > > > A Xapian exception occurred creating a directory: Expected block > > 485 to be level 1, not 0. > > > > I wasnt able to properly read the index anymore and had to > > restore an old index dump. not cool! > = > = > Can you provide any more information about this? The script properly > uses atomic transactions. Any corruption here is the result of a > notmuch library bug, which should be investigated. i am kind of reluctant to reproduce this again (on my work-desktop). All i did was start your script, writing to the index in another terminal i= n between (using alot) possibly my cronjob fired a "notmuch new; afew.." in between also. when I killed your script the index war broken. Can i turn on some global logging for notmuch to debug this? Maybe libnotmuch had trouble with so many db-writes at a time? /p --===============7418330742985369319== MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Description: signature Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEABECAAYFAlDHet4ACgkQlDQDZ9fWxap4BgCdFxBrgs+jIadzi/xvsAVB9mnY s2UAoJZXSab5ftLEa1+d7ORHFiHsJy62 =WT0i -----END PGP SIGNATURE----- --===============7418330742985369319==--