From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id D4EA4431FB6 for ; Thu, 21 Aug 2014 01:58:59 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.7 X-Spam-Level: X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u74j+oLJijAa for ; Thu, 21 Aug 2014 01:58:50 -0700 (PDT) Received: from mail-la0-f42.google.com (mail-la0-f42.google.com [209.85.215.42]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 4CE73431FAF for ; Thu, 21 Aug 2014 01:58:50 -0700 (PDT) Received: by mail-la0-f42.google.com with SMTP id pv20so8377135lab.29 for ; Thu, 21 Aug 2014 01:58:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:subject:to:references:in-reply-to :user-agent:message-id:mime-version:content-type :content-transfer-encoding; bh=FxjRwAOO9beMJyXC9fBjjP4FY/NPHbGPl/peq+7npwE=; b=S3X240f/Yq3GDLrLFQyT+RHQ4FJsImxGKnIFgio9v0Fd9AQOWeQiDIiJ5Kox5hRDj/ tB8eprKSYm4Vh1SMr47e7VoKXTXrMqIp5UUDR+bkxUEGmzTKPkKrixjSM8jbIn0esB8u HhJ9A2rNtIdPDZNixYpAGXOnzxI9C+thqUdyW71pBxDMrEmY6L6qXt+jDtfMS1l3Iyfw DxFFybvl+dXEhhcXUan1JpmQrqUsyEkHYkHxIKVtaTMAKdZ45KfqZQF3uv0vpDH8dK9E +9vbnvRSkPWWo8DqbHncei2Ntp5bgoaZZEerxX7sjwppFOwjGAmcNHHuQetqJKScjdf+ GEvA== X-Gm-Message-State: ALoCoQn6Ew1NUatZ9EOFm1SXqYGsSs/5sutLGt/7T1qRGTTTLn/VuLhJ5FMsQWvs5+Me6xX8CpjE X-Received: by 10.112.138.102 with SMTP id qp6mr44401999lbb.60.1408611527400; Thu, 21 Aug 2014 01:58:47 -0700 (PDT) Received: from localhost (cD572BF51.dhcp.as2116.net. [81.191.114.213]) by mx.google.com with ESMTPSA id y8sm666676lad.30.2014.08.21.01.58.45 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Aug 2014 01:58:46 -0700 (PDT) Date: Thu, 21 Aug 2014 10:59:11 +0200 From: Gaute Hope Subject: Re: DatabaseModifiedErrors causing troubles To: notmuch@notmuchmail.org References: In-Reply-To: User-Agent: astroid/v3f318711 (https://github.com/gauteh/astroid) Message-Id: <1408610770-astroid-0-4on2jlyj8n-28086@strange> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Aug 2014 08:59:00 -0000 Gaute Hope wrote on Mon, 11 Aug 2014 14:17:54 +0200: > Hi, > > I've been working on an application that keeps a read-only handle on > the notmuch database open for a long time. In some cases when a new > message is added along with some renames of other messages using > 'notmuch new' while the application is running I get an Xapian > exception: DatabaseModifiedError: > > A Xapian exception occurred performing query: The revision being > read has been discarded - you > should call Xapian::Database::reopen() and retry the operation. > > Which seems to be printed from: notmuch_query_search_threads -> > notmuch_query_search_messages:294. > > I have not been able to make a smaller test case at the moment (this > happens with offlineimap updating an maildir and notmuch new run > afterwards + some tagging). > > I can work around this by checking for a NULL pointer returned from > notmuch_query_search_threads () and re-open the database > (notmuch_database_close () -> notmuch_database_open ()). But I have no > way of knowing programatically if this really is the error that has > happened. There should be some way of propagating the error > information or (even better for my case; for notmuch to reopen the > database), one option is the Gmime way of passing an pointer to an > error structure that is filled up by the notmuch interface function. Hi again, I have not yet been able to create a test for this with a fresh db (I must be missing out on some of the parts triggering this). I need to keep a query open for a while (because running it takes a while), and this keeps coming back and biting me. I currently do a test every time I need to access the query further (load more threads..). The test is as follows, which is the easiest and quickest test I've found to work reliably: /* tries to provoke an Xapian::DatabaseModifiedError * * returns true if db is invalid. queries will be invalid * afterwards, but might still respond true to notmuch_valid * calls. * */ time_t start =3D clock (); bool invalid =3D false; /* testing */ notmuch_query_t * q =3D notmuch_query_create (nm_db, "thread:fail"); /* this will produce an error, but no way to ensure failure */ notmuch_query_count_threads (q); /* this returns NULL when error */ notmuch_threads_t * threads =3D notmuch_query_search_threads (q); if (threads =3D=3D NULL) invalid =3D true; notmuch_query_destroy (q); float diff =3D (clock () - start) * 1000.0 / CLOCKS_PER_SEC; cout << "db: test query time: " << diff << " ms." << endl; if (invalid) { cout << "db: no longer valid, reopen required." << endl; } else { cout << "db: valid." << endl; } As mentioned before, I am basically guessing that the reason I get threads =3D=3D NULL is the DatabaseModifiedError. There is no way to know for sure. For portability I would suggest starting to move towards the GError scheme provided by glib (also used by gmime). This is a somewhat major effort though since ensuring that error propagation is done right [0] is somewhat tricky. It provides more or less the same functionality as exceptions do in C++. There is also the problem of having to change the API - one way to avoid that is to create wrappers that behave like the old API, but don't handle errors that good. This requires the addition of addiontal _error variations of the current set of functions. It will be a mess. Cheers, Gaute [0] https://developer.gnome.org/glib/stable/glib-Error-Reporting.html =