From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 575DC431FBC for ; Wed, 12 Dec 2012 10:36:00 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.1 X-Spam-Level: X-Spam-Status: No, score=-0.1 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4NrkWu09u8Fz for ; Wed, 12 Dec 2012 10:35:59 -0800 (PST) Received: from homiemail-a38.g.dreamhost.com (caiajhbdcbhh.dreamhost.com [208.97.132.177]) by olra.theworths.org (Postfix) with ESMTP id C3173431FB6 for ; Wed, 12 Dec 2012 10:35:59 -0800 (PST) Received: from homiemail-a38.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a38.g.dreamhost.com (Postfix) with ESMTP id 2124110AFA5 for ; Wed, 12 Dec 2012 10:35:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=zx2c4.com; h=mime-version :in-reply-to:references:date:message-id:subject:from:to:cc: content-type; s=zx2c4.com; bh=cQNUIkTx7EONuKqRMhpFA9wooYE=; b=eN EJS88dfsp1ZObKCu/lYnwALQeczHFL1E1E2KFiAfVlXA4lXhlD+4/jOfk2XBrtCq 0C6j+Mr1mWYyewxYB77Yoeac4R4RkLMbOXp2Ckuvgj5xZpk9A8wLigRg4m82CMSp Mp65SdD939PN9+sdLcF4lMTrI5+O/iZ9KZcpEf7AM= Received: from mail-oa0-f53.google.com (mail-oa0-f53.google.com [209.85.219.53]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: jason@zx2c4.com) by homiemail-a38.g.dreamhost.com (Postfix) with ESMTPSA id E9E8810AFAD for ; Wed, 12 Dec 2012 10:35:58 -0800 (PST) Received: by mail-oa0-f53.google.com with SMTP id j6so1097080oag.26 for ; Wed, 12 Dec 2012 10:35:58 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.32.67 with SMTP id g3mr939326oei.77.1355337357241; Wed, 12 Dec 2012 10:35:57 -0800 (PST) Received: by 10.76.33.70 with HTTP; Wed, 12 Dec 2012 10:35:57 -0800 (PST) In-Reply-To: <20121211182638.27237.98903@brick.lan> References: <20121211182638.27237.98903@brick.lan> Date: Wed, 12 Dec 2012 19:35:57 +0100 Message-ID: Subject: Re: notmuch python bindings corrupt db index (was: gmail importer script) From: "Jason A. Donenfeld" To: Patrick Totzke Content-Type: text/plain; charset=ISO-8859-1 Cc: notmuch@notmuchmail.org X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Dec 2012 18:36:00 -0000 On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke wrote: > OK, i assumed i'd not have to download all messages because i already have them > on disk -- synced via offlineimap. but then i remembered that OI stores uses > its own naming scheme for the files in a maildir. so thats probably why your script > did not realize theyre already there. (maybe it should? it could look up the MIDs from notmuch) Gmail bandwidth restrictions make this unfeasible. Requesting mail header IDs via IMAP will rate limit the listing to 2 mails per second. The best gmail will allow us to do at quasi rapid speed (~180kB/s) is ask for the IMAP sequence number and the unique gmail 64bit number. So, the maildir scheme winds up being "${gmail-unique-number}.gmail", and the script doesn't redownload based on the preexistence of a file with the corresponding name. > i am kind of reluctant to reproduce this again (on my work-desktop). It'd really be a huge help if you could backup and try... > All i did was start your script, writing to the index in another terminal in between (using alot) > possibly my cronjob fired a "notmuch new; afew.." in between also. > when I killed your script the index war broken. > > Can i turn on some global logging for notmuch to debug this? > Maybe libnotmuch had trouble with so many db-writes at a time? That is very wrong and bizarre. I open the database in read/write mode, which is supposed to lock other programs from accessing it in this way... Very strange.