Re: [Patch v7 00/13] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters <markwalters1009@gmail.com>]

On Sat, 03 Mar 2012 03:42:44 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> On Thu,  1 Mar 2012 22:30:32 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> > This is essentially the same as
> > id:"1330157204-26094-1-git-send-email-markwalters1009@gmail.com" but
> > has been rebased against master. The changes are to patch 12/13 for
> > notmuch-show.el (which was posted as a followup to the previous series)
> > and to the tests (patch 9/13) which changed in Austin's JSON show
> > rewrite.
> 
> There are some problems with this series which were discussed on irc
> today. In particular the cli notmuch search output is too cluttered for
> human readability (with lots of excluded threads showing [0/n]) and in
> some cases it is too slow (it is the same speed as if there were no
> excludes but looks slower as it outputs less). The following is a
> proposal to address this.

I have thought some more about this and this does seem the correct way
to go. I have implemented this and will post it as a patch series
shortly.

The original series essentially distinguished between show
everything and mark excluded (default) and show everything not marking
excluded (with the --no-exclude option). These two options are very
similar and do not include the option of just throwing away the excluded
messages. 

This proposal makes the two options roughly "return only non-excluded
messages" (the default) and return all messages marking the excluded
ones (with a new option --with-excluded which replace the --no-exclude
option).

The exact behaviour is a little more subtle but that is essentially it
(as detailed in the parent message).

> PROPOSAL
> 
> lib: Change the notmuch_query_set_omit_excluded_messages to a
> notmuch_query_set_with_excluded_messages option (essentially the
> negation) This just affects the `seeding' of the thread search: whole
> threads are returned regardless but unless this is set only threads that
> match in a non-excluded message are returned. In either case excluded
> messages have the exclude flag set.
> 
> cli: replace the --no-excludes option with a --with-excluded option
> which is roughly the *same*: it outputs the excluded messages (but marks
> them excluded in so far as the output allows it). I deal with each of
> count/search and show in turn.
> 
> count: apply excludes unless --with-excluded. This includes the excluded
> messages/threads in the count. This could be done with the lib call
> above but it is easier not to set the excludes in the first place.
> 
> search: for output=threads|messages|tags we either set or not the
> exclude terms depending on the --with-excluded option. 
> 
> For output=summary (default) we normally return just those threads which
> match in a non-excluded message. With the --with-excluded option return
> all threads that match (using the with_excluded lib call). In both cases
> the count is [x/y] where x is the number of matching non-excluded
> messages and y is the total number of messages.  It will not be possible
> to get the pre-exclude output; with the --with-excluded option you get
> the same threads but the count (x in the above) will show matched and
> non-excluded instead of matched.
> 
> show: raw/part are irrelevant as they do not (and should not) apply
> excludes. 
> 
> When given --with-excluded it returns all messages (in all threads if
> given --entire-thread) and marks excluded if possible (i.e., unless
> format=mbox).
> 
> The remaining cases are all when --with-excluded is not passed. We can
> have format=json/text/mbox and entire-thread or not. We do not pass
> with_excluded to the lib so we only get threads matching in a
> non-excluded message. The question is which of the messages in these
> threads to output. Note emacs will not care as it will pass in the
> --with-excluded option.
> 
> Perhaps --entire-thread should return the whole thread including
> excluded messages but without --entire-thread just return the
> matching-non-excluded messages. We have flexibility here as show did not
> do anything with exclude_tags until this series so not even git users
> will have got used to any particular behaviour.

I have followed this behaviour as it does seem the logical one. If the
user asks for entire-thread it would seem odd to omit some messages from
it (and if passing it to some other program the missing messages could
inhibit reconstruction of the thread). When it is not entire-thread it
would be weird to return all matched messages (including the excluded
ones) in threads with a matched non-excluded message but not
matched-excluded messages in threads which only contain matched-excluded
messages.

As an example of a use for the former is an emacs frontend showing
threaded messages (eg notmuch-pick): it would want all the messages in
the threads it gets (or it will mess up the threading) but probably does
not want threads that only match in excluded messages.

Best wishes

Mark