From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 6888F431FD0 for ; Fri, 27 May 2011 11:04:34 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.799 X-Spam-Level: X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hLG693irdLfW for ; Fri, 27 May 2011 11:04:33 -0700 (PDT) Received: from mail-wy0-f181.google.com (mail-wy0-f181.google.com [74.125.82.181]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 84D57431FB6 for ; Fri, 27 May 2011 11:04:33 -0700 (PDT) Received: by wyi11 with SMTP id 11so1649289wyi.26 for ; Fri, 27 May 2011 11:04:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:cc:subject:from:to:in-reply-to:references:date :message-id:user-agent:content-transfer-encoding:mime-version :content-type; bh=QTTqPztYvAINquWVhJ4Fkm3VHDde+swnQ/Ivdk+2ZWU=; b=vuQf9FnFXaOovjPaLhhgrhnuhqHtIcV/O91bnPGZoXlF1JeAvOCSNpjnb+PZ5ksYw7 xgRn7Gvfu9uRPfoooMf1ewICv7uA4f3AbEbTf5cKXA5yD+AtAwfRlEKFEts02C41EJxd ndPN6T+hfDBFpjNJbEhsgVMbkeUZUu1GOemC4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=cc:subject:from:to:in-reply-to:references:date:message-id :user-agent:content-transfer-encoding:mime-version:content-type; b=YFD7gqVUyVdVr/z9fxWXhqKiX0VExjnew7/MVXEb9kl6TPGyYbaAQSRNt7KLBv+lYY gnro4nTO+OBnxAjJYlttdZNhtZ3HaJ3uw/WRAMOZa/X9Q8kalgQjqIEBr7/pSmuVSU63 83ys/LCr57pZ/zwYw8sDjE6Pr0pJXpmwST1Ts= Received: by 10.216.136.207 with SMTP id w57mr7141300wei.63.1306519471843; Fri, 27 May 2011 11:04:31 -0700 (PDT) Received: from localhost (cpc1-sgyl2-0-0-cust47.sgyl.cable.virginmedia.com [80.192.18.48]) by mx.google.com with ESMTPS id fw15sm1355413wbb.44.2011.05.27.11.04.28 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 27 May 2011 11:04:30 -0700 (PDT) Subject: Re: one-time-iterators From: Patrick Totzke To: Austin Clements In-reply-to: References: <1306397849-sup-3304@brick> <877h9d9y5m.fsf@yoom.home.cworth.org> <1306442683-sup-9315@brick> <20110526214302.GR29861@mit.edu> <1306446621-sup-3184@brick> Date: Fri, 27 May 2011 19:04:27 +0100 Message-Id: <1306518628-sup-5396@brick> User-Agent: Sup/git Content-Transfer-Encoding: 8bit MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-1306519467-975049-22881-1746-2-="; protocol="application/pgp-signature" Cc: Patrick Totzke , notmuch X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 May 2011 18:04:34 -0000 --=-1306519467-975049-22881-1746-2-= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Excerpts from Austin Clements's message of Fri May 27 03:41:44 +0100 2011= : > >> > > Have you tried simply calling list() on your thread > >> > > iterator to see how expensive it is? =C2=A0My bet is that it's q= uite cheap, > >> > > both memory-wise and CPU-wise. > >> > Funny thing: > >> > =C2=A0q=3DDatabase().create_query('*') > >> > =C2=A0time tlist =3D list(q.search_threads()) > >> > raises a NotmuchError(STATUS.NOT_INITIALIZED) exception. For some = reason > >> > the list constructor must read mere than once from the iterator. > >> > So this is not an option, but even if it worked, it would show > >> > the same behaviour as my above test.. > >> > >> Interesting. =C2=A0Looks like the Threads class implements __len__ a= nd that > >> its implementation exhausts the iterator. =C2=A0Which isn't a great = idea in > >> itself, but it turns out that Python's implementation of list() call= s > >> __len__ if it's available (presumably to pre-size the list) before > >> iterating over the object, so it exhausts the iterator before even > >> using it. > >> > >> That said, if list(q.search_threads()) did work, it wouldn't give yo= u > >> better performance than your experiment above. true. Nevertheless I think that list(q.search_threads()) should be equivalent to [t for t in q.search_threads()], which is something to be fixed in the bindings. Should I file an issue somehow? Or is enough to state this as a TODO here on the list? > >> > would it be very hard to implement a Query.search_thread_ids() ? > >> > This name is a bit off because it had to be done on a lower level.= > >> > >> Lazily fetching the thread metadata on the C side would probably > >> address your problem automatically. =C2=A0But what are you doing tha= t > >> doesn't require any information about the threads you're manipulatin= g? > > Agreed. Unfortunately, there seems to be no way to get a list of thre= ad > > ids or a reliable iterator thereof by using the current python bindin= gs. > > It would be enough for me to have the ids because then I could > > search for the few threads I actually need individually on demand. > = > There's no way to do that from the C API either, so don't feel left > out. ]:--8) It seems to me that the right solution to your problem > is to make thread information lazy (effectively, everything gathered > in lib/thread.cc:_thread_add_message). Then you could probably > materialize that iterator cheaply. = Alright. I'll put this on my mental notmuch wish list and = hope that someone will have addressed this before I run out of ideas how to improve my UI and have time to look at this myself. For now, I go with the [t.get_thread_id for t in q.search_threads()] approach to cache the thread ids myself and live with the fact that this takes time for large result sets. > In fact, it's probably worth > trying a hack where you put dummy information in the thread object > from _thread_add_message and see how long it takes just to walk the > iterator (unfortunately I don't think profiling will help much here > because much of your time is probably spent waiting for I/O). I don't think I understand what you mean by dummy info in a thread object. > I don't think there would be any downside to doing this for eager > consumers like the CLI. one should think so, yes. /p --=-1306519467-975049-22881-1746-2-= Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3f56sACgkQlDQDZ9fWxaqA9QCgpvXeNv8MJkKU+4knpY7VJC24 VUwAoK5dFwwmSkEhtD8dLt2EYc9PqQTb =t6vZ -----END PGP SIGNATURE----- --=-1306519467-975049-22881-1746-2-=--