From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id D29A4429E54 for ; Mon, 23 Jan 2012 17:05:02 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -1.098 X-Spam-Level: X-Spam-Status: No, score=-1.098 tagged_above=-999 required=5 tests=[DKIM_ADSP_CUSTOM_MED=0.001, FREEMAIL_FROM=0.001, NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QL31TjYWcdYU for ; Mon, 23 Jan 2012 17:05:02 -0800 (PST) Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by olra.theworths.org (Postfix) with ESMTPS id 9C57F429E21 for ; Mon, 23 Jan 2012 17:05:01 -0800 (PST) Received: from smtp.qmul.ac.uk ([138.37.6.40]) by mail2.qmul.ac.uk with esmtp (Exim 4.71) (envelope-from ) id 1RpUoi-0002XO-1p; Tue, 24 Jan 2012 01:04:56 +0000 Received: from 94-192-233-223.zone6.bethere.co.uk ([94.192.233.223] helo=localhost) by smtp.qmul.ac.uk with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) (envelope-from ) id 1RpUoh-0001V2-GT; Tue, 24 Jan 2012 01:04:55 +0000 From: Mark Walters To: Austin Clements Subject: Re: [PATCH] Automatically exclude tags in notmuch-show In-Reply-To: <20120123015222.GB7600@mit.edu> References: <874nvric7c.fsf@qmul.ac.uk> <1327010583-23954-1-git-send-email-markwalters1009@gmail.com> <20120119225910.GT16740@mit.edu> <871uqvgrnm.fsf@qmul.ac.uk> <20120120171801.GA16740@mit.edu> <20120122181609.GQ16740@mit.edu> <87zkdfgr0m.fsf@qmul.ac.uk> <20120123015222.GB7600@mit.edu> User-Agent: Notmuch/0.11+99~g7f60f7e (http://notmuchmail.org) Emacs/23.2.1 (i486-pc-linux-gnu) Date: Tue, 24 Jan 2012 01:05:52 +0000 Message-ID: <87r4yphpu7.fsf@qmul.ac.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Sender-Host-Address: 94.192.233.223 X-QM-SPAM-Info: Sender has good ham record. :) X-QM-Body-MD5: 4b67a66e247fb602c3fb83c452a12b5f (of first 20000 bytes) X-SpamAssassin-Score: -1.8 X-SpamAssassin-SpamBar: - X-SpamAssassin-Report: The QM spam filters have analysed this message to determine if it is spam. We require at least 5.0 points to mark a message as spam. This message scored -1.8 points. Summary of the scoring: * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, * medium trust * [138.37.6.40 listed in list.dnswl.org] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (markwalters1009[at]gmail.com) * -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay * domain * 0.5 AWL AWL: From: address is in the auto white-list X-QM-Scan-Virus: ClamAV says the message is clean Cc: notmuch@notmuchmail.org X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jan 2012 01:05:03 -0000 On Sun, 22 Jan 2012 20:52:22 -0500, Austin Clements wrote: > Quoth Mark Walters on Jan 23 at 1:13 am: > > On Sun, 22 Jan 2012 13:16:09 -0500, Austin Clements wrote: > > > Quoth myself on Jan 20 at 12:18 pm: > > > > Quoth Mark Walters on Jan 20 at 12:10 am: > > > > > > > > > > Ok Having said this is trivial I have found a problem. What should > > > > > notmuch do if you do something like > > > > > > > > > > notmuch show id: > > > > > and that message is marked with a deleted tag? To be consistent with the > > > > > other cases (where a deleted message is in a matched thread) we might > > > > > want to return the message with the not-matched flag set (eg in > > > > > JSON). But my patch doesn't, as it never even sees the thread since it > > > > > doesn't match. > > > > > > > > > > Looking at notmuch-show.c I think we should not apply the exclude tags > > > > > to do_show_single, but usually should apply it to do_show. One solution > > > > > which is simple and is at least close to right would be to get do_show > > > > > to return the number of threads found. If this is zero then retry the > > > > > query without the excludes (possible setting the match_flag to zero on > > > > > each message since we know it does not match) > > > > > > > > > > This is not a completely correct solution as if you ask notmuch-show to > > > > > show more than one thread it might threads which only contain deleted > > > > > messages. > > > > > > > > > > I can't see other good possibilities without slowing down the normal > > > > > path a lot (eg find all threads that match the original query and then > > > > > apply the argument above). > > > > > > > > > > Any thoughts? > > > > > > > > Oh dear. > > > > > > > > Well, here's one idea. Instead of doing a single thread query in > > > > show, do a thread query without the exclusions and then a message > > > > query with the exclusions. Output all of the messages from the first > > > > query, but use the results of the second query to determine which > > > > messages are "matched". The same could be accomplished in the library > > > > somewhat more efficiently, but it's not obvious to me what the API > > > > would be. > > > > > > Here's a slightly crazier idea that's more library-invasive than the > > > original approach, but probably better in the long run. > > > > > > Have notmuch_query_search_* return everything and make exclusion a > > > message flag like NOTMUCH_MESSAGE_FLAG_MATCH. Tweak the definition of > > > "matched" to mean "matched and not excluded" (specifically, a message > > > would have the match flag or the excluded flag or neither, but not > > > both). Search would skip threads with zero matched messages and I > > > think show would Just Work. > > > > > > I can think of two ways to implement this. notmuch_query_search_* > > > could perform both the original query and the query with exclusions > > > and use the docid set from the second to compute the "excluded" > > > message flag. Alternatively, it could examine the tags of each > > > message directly to compute the flag. The latter is probably easier > > > to implement, but probably slower. > > > > > > Thoughts? > > > > I have now thought about this some more and think I understand your idea > > (and how it would work) rather better now. > > > > I would suggest one small change: the flags for the messages returned > > should be "independent": so a message can match the query or not, and it > > can be excluded or not, with all 4 combinations being possible. (The > > consumer of notmuch_query_search_* would extract the information it > > wanted.) > > I'd initially approached it this way, but went with redefining a > "matched" messages because it had much less impact on the API. For > example, with the redefined "match", > notmuch_thread_get_matched_messages still does the right thing for > search and things like the thread subject can still be based on > "matched" messages. If we orthongonalize these flags, then we at > least need to count matched non-excluded messages and provide an API > to access this (while I don't have a solid argument against such an > API it just seems weirdly specific to me). Ok I have an initial implementation of this which I will post as a reply to this thread: it does make the flags orthogonal but that would be easy to change. If we do want to keep match to mean match and not excluded then I would argue for a third flag so that the emacs frontend could see all 4 possibilities. Note that in your suggestion we still need to do something in notmuch_thread_get_matched_messages to set the subject etc in threads with no matching non-excluded messages. > My other concern is performance. In thread queries, marking > non-matched messages as excluded would require either an extra query > per thread or a single query to match all excluded messages (not > filtered by the primary query). The former is prohibitive, though the > latter might be acceptable (that might depend on how many things > people mark as spam or deleted). If the cost is too high, this > suggests that we shouldn't mark non-matched messages as excluded, but > then we're back to effectively having three levels of matching: not > matched, matched but not excluded, and matched but excluded. The implementation follows the second suggestion. I'll discuss performance below. > > I have thought about some implementation ideas but I think sorting is > > going to be the deciding factor: what order should > > notmuch_query_search_* return messages/threads? > > Yes. This is exactly what I've been puzzling over, too. > > > For notmuch_query_search_messages either it returns them all together > > with the excluded messages marked, or returns all included ones, and > > then all excluded one. > > I would prefer them intermingled. I feel like returning one and then > the other is just exposing implementation details. Plus, it's unclear > if the order of the two groups should depend on the sort order, be > configurable, or what. Intermingling them seems like the obvious > answer. > > > For notmuch_query_search_threads it is less clear. Currently it returns > > threads in order of first matching message. It is not clear what > > matching means now: is matching and included, or just matching? If the > > former then we will be returning some threads with no matching and > > included messages so we need to decide where to put them in the order. > > I would argue that, if the caller cares about the sort order of the > results, it only makes sense for it to skip over threads consisting > only of excluded messages, and if the caller doesn't care about the > sort order, we can choose whatever's most convenient. I have gone with this intermingled suggestion. One other implementation detail: the code uses the exclude query generated by your _notmuch_exclude_tags function; thus if the user over-rides the exclude with tag:deleted then these messages will not be marked excluded. I can't work out whether that is the right behaviour. The code as it stands should finish where we started, but have the infrastructure to show excluded messages. I think a nice interface would be for the emacs notmuch-search frontend to take all messages and hide all threads with 0 matching messages, but make them unhideable. (Possibly if there are no non-hidden threads it could offer to show them.) The notmuch-show frontend could take all messages but show only a header line for excluded messages (perhaps in a different colour). It would jump to the first matching non excluded message unless there aren't any, in which case it would go to the first matching (necessarily excluded) message instead. Performance: the code/notmuch is generally fast. On my computer/email archive of circa 70000 the time to constuct the list of excluded messages is of the order of 20 milliseconds. (Inherently it should not be much slower since the thread return already has to construct a complete list of all messages matching the query before it can return any threads.) However, in some case it will appear slower: if there are lots of threads that only match in excluded messages then this code will waste time constructing these "hidden threads". In other words the code will be comparable to the pre-exclude notmuch: since it is constructing the same threads. Finally, the code is obviously an early draft to see what people think. I know there are things that need freeing, and probably some error returns that should be checked. Best wishes Mark