From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 6D4B5429E25 for ; Thu, 26 May 2011 15:22:19 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: 0.514 X-Spam-Level: X-Spam-Status: No, score=0.514 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, URI_HEX=1.313] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Cpl6dBGNfXgP for ; Thu, 26 May 2011 15:22:18 -0700 (PDT) Received: from mail-ww0-f45.google.com (mail-ww0-f45.google.com [74.125.82.45]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 7B1BB431FB6 for ; Thu, 26 May 2011 15:22:18 -0700 (PDT) Received: by wwi36 with SMTP id 36so1035544wwi.2 for ; Thu, 26 May 2011 15:22:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:cc:subject:from:to:in-reply-to:references:date :message-id:user-agent:content-transfer-encoding:mime-version :content-type; bh=pPYGloO5O3CBU5k0tDAhcl2/TZNywJVT2JZ99IHRfR4=; b=kW0xlqn7jM9dxRh015NG/YXRKihaRjgWbG8nEdP39EELGf+8Fbr5+GO101SY2FYS2X c1z4hjd+gP6rqSogzKCDF9RKxzQT8+zPEA2y9hUkZ6WLhbAROM4wuoEUK/QDCMLyxv3r lRKNFZtmM+lhQhaq3mj9FSlVo8AZZqSVrtTUo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=cc:subject:from:to:in-reply-to:references:date:message-id :user-agent:content-transfer-encoding:mime-version:content-type; b=gaY30vbWZzhcQyPrWfk4e0YeiCbdUoIUALBk0LGRGyMQfcw+TSu0NBm3Yxhh5gddLv SsEW0bLpC/sPUaPECheXRG082EEyuGMqyWoLgOW4ibwP8I95RLAC7WbjFZdDoVzDlI3V o0XcRMPlFw7SzQwl84Tudhn/Bew6xQxXzNYfk= Received: by 10.227.165.10 with SMTP id g10mr1396157wby.91.1306448536167; Thu, 26 May 2011 15:22:16 -0700 (PDT) Received: from localhost (cpc1-sgyl2-0-0-cust47.sgyl.cable.virginmedia.com [80.192.18.48]) by mx.google.com with ESMTPS id en1sm775917wbb.1.2011.05.26.15.22.13 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 26 May 2011 15:22:14 -0700 (PDT) Subject: Re: one-time-iterators From: Patrick Totzke To: Austin Clements In-reply-to: <20110526214302.GR29861@mit.edu> References: <1306397849-sup-3304@brick> <877h9d9y5m.fsf@yoom.home.cworth.org> <1306442683-sup-9315@brick> <20110526214302.GR29861@mit.edu> Date: Thu, 26 May 2011 23:22:11 +0100 Message-Id: <1306446621-sup-3184@brick> User-Agent: Sup/git Content-Transfer-Encoding: 8bit MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-1306448531-827817-32122-6706-2-="; protocol="application/pgp-signature" Cc: notmuch@notmuchmail.org X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 May 2011 22:22:19 -0000 --=-1306448531-827817-32122-6706-2-= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Excerpts from Austin Clements's message of Thu May 26 22:43:02 +0100 2011= : > http://notmuch.198994.n3.nabble.com/notmuch-s-idea-of-concurrency-faili= ng-an-invocation-tp2373468p2565731.html ah, good old peterson :P thanks. > > > Though, Patrick, that solution doesn't address your problem.=C2=A0 = On the > > > other hand, it's not clear to me what concurrent access semantics > > > you're actually expecting.=C2=A0 I suspect you don't want the remai= ning > > > iteration to reflect the changes, since your changes could equally > > > well have affected earlier iteration results.=C2=A0 > > That's right. = > > > But if you want a > > > consistent view of your query results, something's going to have to= > > > materialize that iterator, and it might as well be you (or Xapian > > > would need more sophisticated concurrency control than it has).=C2=A0= But > > > this shouldn't be expensive because all you need to materialize are= > > > the document ids; you shouldn't need to eagerly fetch the per-threa= d > > > information. = > > I thought so, but it seems that Query.search_threads() already > > caches more than the id of each item. Which is as expected > > because it is designed to return thread objects, not their ids. > > As you can see above, this _is_ too expensive for me. > = > I'd forgotten that constructing threads on the C side was eager about > the thread tags, author list and subject (which, without Istvan's > proposed patch, even requires opening and parsing the message file). > This is probably what's killing you. > = > Out of curiosity, what is your situation that you won't wind up paying > the cost of this iteration one way or the other and that the latency > of doing these tag changes matters? I'm trying to implement a terminal interface for notmuch in python that resembles sup. For the search results view, i read an initial portion from a Threads ite= rator = to fill my teminal window with threadline-widgets. Obviously, for a large number of results I don't want to go through all of them. The problem arises if you toggle a tag on the selected threadline and aft= erwards continue to scroll down. > > > Have you tried simply calling list() on your thread > > > iterator to see how expensive it is? My bet is that it's quite che= ap, > > > both memory-wise and CPU-wise. > > Funny thing: > > q=3DDatabase().create_query('*') > > time tlist =3D list(q.search_threads()) > > raises a NotmuchError(STATUS.NOT_INITIALIZED) exception. For some rea= son > > the list constructor must read mere than once from the iterator. > > So this is not an option, but even if it worked, it would show > > the same behaviour as my above test.. > = > Interesting. Looks like the Threads class implements __len__ and that > its implementation exhausts the iterator. Which isn't a great idea in > itself, but it turns out that Python's implementation of list() calls > __len__ if it's available (presumably to pre-size the list) before > iterating over the object, so it exhausts the iterator before even > using it. > = > That said, if list(q.search_threads()) did work, it wouldn't give you > better performance than your experiment above. > = > > would it be very hard to implement a Query.search_thread_ids() ? > > This name is a bit off because it had to be done on a lower level. > = > Lazily fetching the thread metadata on the C side would probably > address your problem automatically. But what are you doing that > doesn't require any information about the threads you're manipulating? Agreed. Unfortunately, there seems to be no way to get a list of thread ids or a reliable iterator thereof by using the current python bindings. It would be enough for me to have the ids because then I could search for the few threads I actually need individually on demand. Here is the branch in which I'm trying out these things. Sorry for the messy code, its late :P https://github.com/pazz/notmuch-gui/tree/toggletags /p --=-1306448531-827817-32122-6706-2-= Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3e0pMACgkQlDQDZ9fWxaqKuQCfTm9nYgX4HJb2+tZlZUuGnXfp mBcAn1pyXaVzNQR9dJfIgHCyGSv/G6hl =X10m -----END PGP SIGNATURE----- --=-1306448531-827817-32122-6706-2-=--