From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id CA912431FD0 for ; Sun, 17 Jul 2011 12:35:48 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.799 X-Spam-Level: X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cqH2DmLFtvSw for ; Sun, 17 Jul 2011 12:35:48 -0700 (PDT) Received: from mail-ww0-f45.google.com (mail-ww0-f45.google.com [74.125.82.45]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id D5EC9431FB6 for ; Sun, 17 Jul 2011 12:35:47 -0700 (PDT) Received: by wwj40 with SMTP id 40so2111183wwj.2 for ; Sun, 17 Jul 2011 12:35:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=date:from:to:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:x-offlineimap :user-agent; bh=AedytK2wwXJOO8BwhqxK5gW8LshabjUwTv5Vjh7zTFg=; b=mCxKAyD/jeiiBw+j99NrCm1I8PXHanpIjz8dtjvTV0Na3PUh2WhvetJBYD6P9ECqF9 CZfJfTmgZHd0dWCrPFELyTW4aASuYVkyLU0fEPB7fWXU8gLaPTcFCDt+7i9sl71K37qt onDJ5rmdqBPWo/i8h2EnuS0Z04pAzfmAFYVEY= Received: by 10.227.28.212 with SMTP id n20mr4800710wbc.82.1310931346454; Sun, 17 Jul 2011 12:35:46 -0700 (PDT) Received: from brick.lan (cpc1-sgyl2-0-0-cust47.sgyl.cable.virginmedia.com [80.192.18.48]) by mx.google.com with ESMTPS id fe4sm2837975wbb.62.2011.07.17.12.35.42 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 17 Jul 2011 12:35:44 -0700 (PDT) Date: Sun, 17 Jul 2011 20:35:38 +0100 From: Patrick Totzke To: notmuch@notmuchmail.org Subject: Re: xapian exceptions not caught in python bindings? Message-ID: <20110717193538.GA16615@brick.lan> References: <20110626202733.GA26837@brick> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="r5Pyd7+fXNt84Ff3" Content-Disposition: inline In-Reply-To: <20110626202733.GA26837@brick> X-OfflineIMAP: 2521481603-8359016323 User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jul 2011 19:35:48 -0000 --r5Pyd7+fXNt84Ff3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi all, I know this issue is not easy to fix and that it's partly a problem with xapians bad error handling. I just want to state another problem we run into with xapians exceptions in combination with the python bindings. If you run the following snippet, you notice that not only do we get xapian-garbage on stderr but we don't really get any exceptions at the position where it would make sense: -------------------------------- =66rom notmuch import Database bad_querystring =3D "test AND" # This should raise a NotmuchError and query should be NoneType query =3D Database().create_query(bad_querystring) # this prints to stderr but returns a Threads object. threads =3D query.search_threads() # this finally raises a NotmuchError(STATUS.NOT_INITIALIZED) for t in threads: print t -------------------------------- Although the querystring is invalid syntactically, the query object gets=20 instantiated and can of course not behave as it should later on. =46rom a user point of view it is really quite strange that only later (if one iterates over threads) an exception is raised. One solution that comes to mind here is simply bending sys.sdterr to someth= ing internal, interpreting its output and raising the correct exceptions accordingly. The downside is, this would have to happen every time the library gets call= ed. Also, a fix like this would be python specific, so everybody else would sti= ll get garbage printed to stderr occasionally. Thoughts anyone? /p On Sun, Jun 26, 2011 at 09:27:33PM +0100, Patrick Totzke wrote: > Hi all! > It's me again, with some strange python behaviour :/ >=20 > When I iterate over threads or messages only partially, > then make changes to the index and continue the iteration, > then I don't necessarily get NotmuchError's but the underlying > libnotmuch seems to call terminate. I get the following two lines on > stderr: > terminate called after throwing an instance of 'Xapian::DatabaseModifie= dError' > Aborted > then python aborts without an exception. > This happens on current master with current python bindings. To reproduce > use this to iterate over messages: >=20 > ------------------%<------------------- > #it_read.py > from notmuch import Query,Database > msgs =3D Database().create_query('*').search_messages() >=20 > i=3D0 > for m in msgs: > i=3Di+1 > if i%50=3D=3D0: > raw_input() > readit =3D str(m) > ------------------>%------------------- >=20 > start it, then use this to change the index: >=20 > ------------------%<------------------- > #it_write.py=20 > from notmuch import Query,Database > db =3D Database(mode =3D Database.MODE.READ_WRITE) > msgs =3D Query(db,'*').search_messages() >=20 > i=3D0 > for m in msgs: > i=3Di+1 > if i%20=3D=3D0: > m.add_tag('REMOVEME') > ------------------>%------------------- >=20 > you have to alternate a bit between hitting enter in the > read code and rerunning the write code. > Read will fail at some point. >=20 > The strange thing is that its not consistent: > I stumbled upon this while reading a list of threads from notmuch.threads= on > demand. If reading from the iterator catches a NotmuchError, i create a n= ew one > and update my list threads read so far. This does work, but: on every > NotmuchError, I get this on stderr: > A Xapian exception occurred performing query: The revision being read ha= s been discarded - you should call Xapian::Database::reopen() and retry the= operation > Query string was: thread:0000000000001876 >=20 > After a couple of resets this does not happen again but the next time i r= ead > from an outdated index, the whole UI terminates with the "Aborted" message > stated above. >=20 > To sum up, I think there are two bad things happening here: > 1) should libnotmuch really print to stderr? > 2) the python bindings are either not relaying all exceptions or libnotmu= ch is not relaying=20 > all xapian exceptions in order for the bindings to intercept them >=20 > cheers, > /p --r5Pyd7+fXNt84Ff3 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk4jOYoACgkQlDQDZ9fWxaqAqACg0YgDBWosSX5bPFp0uRBJ17nY EBYAnjfDYSXsGqpDpTi5xghHi4REcyKz =0rOH -----END PGP SIGNATURE----- --r5Pyd7+fXNt84Ff3--