notmuch.el: controlling what does and doesn't get expanded in searches

notmuch.el: controlling what does and doesn't get expanded in searches

[Matt Armstrong]

This question pertains to notmuch built from recent git HEAD, using
notmuch.el in show mode (i.e. not tree mode).

I sometimes read a thread with a bunch of messages and notmuch.el
collapses a bunch of them (even if unread and the search matches tags in
every message).  I can't figure out the heuristic notmuch is applying
here.  In particular, pressing SPC does not seem to navigate to the
collapsed messages (again, even if they are unread).

Worst case: only the first messages is initially expanded and all
subsequent are collapsed.  I press SPC and the cursor goes to the end of
search results.  SPC again all the entire thread is archived.

This behavior has caused me to accidentally skip messages.  First step
for me is understanding what is going on so I can fix it.

Re: notmuch.el: controlling what does and doesn't get expanded in searches

[Jani Nikula]

On Thu, 04 Aug 2016, Matt Armstrong <marmstrong@google.com> wrote:
> This question pertains to notmuch built from recent git HEAD, using
> notmuch.el in show mode (i.e. not tree mode).
>
> I sometimes read a thread with a bunch of messages and notmuch.el
> collapses a bunch of them (even if unread and the search matches tags
> in every message).  I can't figure out the heuristic notmuch is
> applying here.

The idea is that all the messages matching the query are expanded, the
others collapsed. Each expanded message must fully match the query. The
unread or inbox tags are not special in this regard.

I am not saying this is ideal, but this is how it's supposed to
work. (Indeed I'd personally like to define e.g. saved search specific
tags or queries to use for deciding which messages to expand.)

> In particular, pressing SPC does not seem to navigate to the collapsed
> messages (again, even if they are unread).

SPC and n and p are supposed to navigate expanded messages only. N and P
navigate all messages (but do not expand by default). Again, the tags
the messages have do not matter. You can manually expand/collapse
messages, and that'll affect the navigation.

> Worst case: only the first messages is initially expanded and all
> subsequent are collapsed.  I press SPC and the cursor goes to the end of
> search results.  SPC again all the entire thread is archived.
>
> This behavior has caused me to accidentally skip messages.  First step
> for me is understanding what is going on so I can fix it.

Yes, let's first check that notmuch behaves as it is expected, and then
figure out how to improve it.

BR,
Jani.

Re: notmuch.el: controlling what does and doesn't get expanded in searches

[Matt Armstrong]

Jani Nikula <jani@nikula.org> writes:

> On Thu, 04 Aug 2016, Matt Armstrong <marmstrong@google.com> wrote:
>> This question pertains to notmuch built from recent git HEAD, using
>> notmuch.el in show mode (i.e. not tree mode).
>>
>> I sometimes read a thread with a bunch of messages and notmuch.el
>> collapses a bunch of them (even if unread and the search matches tags
>> in every message).  I can't figure out the heuristic notmuch is
>> applying here.
>
> The idea is that all the messages matching the query are expanded, the
> others collapsed. Each expanded message must fully match the query. The
> unread or inbox tags are not special in this regard.
>
> I am not saying this is ideal, but this is how it's supposed to
> work. (Indeed I'd personally like to define e.g. saved search specific
> tags or queries to use for deciding which messages to expand.)

Thank you.

Yes, I find the query semantics with respect to tags and threads a bit
confusing at times.  This is not a problem specific to notmuch, as I
find the same kinds of issues in GMail.  Usually the problem occurs at
the semantic border between per-message tags and thread-based
operations.

I can now explain what I am seeing.  In my notmuch-post-hook I tag
messages according to various categories (mailing lists, etc.).  Every
time I do this I also tag them with 'filtered'.  My
`notmuch-saved-searches` has about 15 sub-queries of the form "tag:inbox
AND tag:<something>".

I also have a saved search to catch the "unfiltered" stuff:

  tag:inbox AND tag:unfiltered

So this occurred:

1) One message was sent to a foo-announce mailing list.  This was not
   caught by my filters, so it was simply tagged 'inbox'.

2) All replies were sent to the main "foo" mailing list, which *was*
   caught by my filters and tagged 'inbox' and 'foo' and 'filtered'.

This is bad for two reasons:

a) If I observed this thread by searching for 'tag:inbox AND tag:foo',
   the initial foo-announce message would not be expanded.  But, as the
   first message in the thread, it is the most important!

b) If I observed this thread by searching for 'tag:inbox AND not
   tag:filtered' (as I do to find all the "uncategorized" stuff in my
   inbox), then the foo-announce mail is expanded but none of the
   others.

This problem isn't specific to my 'filtered' tag approach.  I think it
generalizes to any approach that attempts to split incoming mail.

I've been seeing this problem quite frequently because I'm in an
environment where messages are cc:'d to different mailing lists all the
time.  It is common for threads to be cc'd to new mailing lists
mid-thread, or for people to pull lists off the cc: list mid-thread
(sometimes into private per-person distribution).

In this environment, different messages in those threads area expanded
depending on which query I use to find them.  This is undesirable
because, generally, I want to read every message I have not yet read in
these threads.


>> In particular, pressing SPC does not seem to navigate to the collapsed
>> messages (again, even if they are unread).
>
> SPC and n and p are supposed to navigate expanded messages only. N and P
> navigate all messages (but do not expand by default). Again, the tags
> the messages have do not matter. You can manually expand/collapse
> messages, and that'll affect the navigation.

Note that SPC also archives, and when it does, it archives the entire
thread, not just the expanded messages (i.e. those that match the
current search).  So, this gave rise to the above situation, where I
pressed SPC twice and archived a 40 message thread, with most messages
still unread.


>> Worst case: only the first messages is initially expanded and all
>> subsequent are collapsed.  I press SPC and the cursor goes to the end
>> of search results.  SPC again all the entire thread is archived.
>>
>> This behavior has caused me to accidentally skip messages.  First
>> step for me is understanding what is going on so I can fix it.
>
> Yes, let's first check that notmuch behaves as it is expected, and
> then figure out how to improve it.

I think the semantics make coherent sense for ad-hoc searches.  Things
break down for me when considering an inbox processing workflow heavily
based on archiving.

If I return to a thread after reading the first N messages I need not
see those messages expanded.  I have already read them, so I'd prefer
they be collapsed.  Not much usually does this for me because I archive
aggressively, but I don't always archive.  In these cases I think I'd
prefer expansion instead be based on whether I've read the message
(tag:unread).

Also, I do want threads consisting entirely of read messages to appear
in my inbox searches, so I do not want to simply add "AND tag:unread" to
all of my `notmuch-saved-searches`.

Additionally, if messages that don't match the query are pulled into
threads that don't match the query, and are "unread", I'd like to see
them.  Such messages are quite likely to provide important context.

I can think of two ideas:

 i) notmuch could have an "also expand tags" feature, where thread based
    results would auto expand matching tags.  I would set this to
    "unread".

 ii) notmuch could have an "expand query" feature, where thread based
     results would use an entirely different query to decide, within a
     thread, which messages to expand.  I would set this query to
     "tag:unread".

This would be most natural with the current archive semantics in
notmuch.el, which apply tags to entire threads on the assumption that
SPC advances over them in meaningful ways.

Re: notmuch.el: controlling what does and doesn't get expanded in searches

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]

On Thu, Aug 04 2016, Matt Armstrong wrote:
> Yes, I find the query semantics with respect to tags and threads a bit
> confusing at times.  This is not a problem specific to notmuch, as I
> find the same kinds of issues in GMail.  Usually the problem occurs at
> the semantic border between per-message tags and thread-based
> operations.
>
> I can now explain what I am seeing.

Hi Matt,

I haven't been very active on the notmuch mailing list in quite some
time, but I wanted to poke my head in quickly to welcome you and to
thank you for your contribution.

I really like detailed discussions like this about coming up with good
workflows, and trying to figure out how notmuch could better accommodate
those workflows. I think this is one of the most valuable aspects of
notmuch, (that it lets us ask these kinds of questions).

So, please, keep it up!

>  i) notmuch could have an "also expand tags" feature, where thread based
>     results would auto expand matching tags.  I would set this to
>     "unread".

This approach makes a lot of sense to me based on how notmuch.el works.

>  ii) notmuch could have an "expand query" feature, where thread based
>      results would use an entirely different query to decide, within a
>      thread, which messages to expand.  I would set this query to
>      "tag:unread".

This approach would necessarily be quite a bit more complex in the
implementation without much difference in the user-visible behavior. So
I don't think we would want to pursue this.

-Carl

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: notmuch.el: controlling what does and doesn't get expanded in searches

[Jani Nikula]

On Thu, 04 Aug 2016, Carl Worth <cworth@cworth.org> wrote:
>>  i) notmuch could have an "also expand tags" feature, where thread based
>>     results would auto expand matching tags.  I would set this to
>>     "unread".
>
> This approach makes a lot of sense to me based on how notmuch.el works.

My idea on how to do this: I'd like to have a key binding in the show
view to go through a customizable list of rules on how to
collapse/expand the messages. The rules could be:

* [ ] expand all matching messages
  [ ] expand messages having any of the specified tags
  [ ] expand messages having all of the specified tags
* expand all messages
* collapse all messages

(* are mutually exclusive, [ ] are not)

The first rule would define what is displayed by default. So you could
have, for example, "expand all matching messages and any messages that
have both inbox and unread tags", followed by "expand all matching
messages", followed by "expand messages that have inbox tag", followed
by "expand all messages", etc. any way you wish.

It would be a nice bonus if you could specify at which rule to start per
each saved search, instead of the first in the list.

I think this could replace the current M-RET and C-u M-RET
expand/collapse all bindings. Maybe M-RET could be reused for this.

This would obviously not require any changes to the SPC, n, p or other
navigation bindings, which I think are currently just fine.


BR,
Jani.

Re: notmuch.el: controlling what does and doesn't get expanded in searches

[Matt Armstrong]

Jani Nikula <jani@nikula.org> writes:

> On Thu, 04 Aug 2016, Carl Worth <cworth@cworth.org> wrote:
>>>  i) notmuch could have an "also expand tags" feature, where thread based
>>>     results would auto expand matching tags.  I would set this to
>>>     "unread".
>>
>> This approach makes a lot of sense to me based on how notmuch.el works.
>
> My idea on how to do this: I'd like to have a key binding in the show
> view to go through a customizable list of rules on how to
> collapse/expand the messages. The rules could be:
>
> * [ ] expand all matching messages
>   [ ] expand messages having any of the specified tags
>   [ ] expand messages having all of the specified tags
> * expand all messages
> * collapse all messages
>
> (* are mutually exclusive, [ ] are not)
>
> The first rule would define what is displayed by default. So you could
> have, for example, "expand all matching messages and any messages that
> have both inbox and unread tags", followed by "expand all matching
> messages", followed by "expand messages that have inbox tag", followed
> by "expand all messages", etc. any way you wish.
>
> It would be a nice bonus if you could specify at which rule to start per
> each saved search, instead of the first in the list.
>
> I think this could replace the current M-RET and C-u M-RET
> expand/collapse all bindings. Maybe M-RET could be reused for this.
>
> This would obviously not require any changes to the SPC, n, p or other
> navigation bindings, which I think are currently just fine.

I've begun a WIP patch series for this at
id:1474003701-19831-1-git-send-email-marmstrong@google.com under subject
"emacs: show: expand unread tags"

Jani, compared to your idea what I have is very simple.  notmuch-show
retains just the two modes when initially displaying a thread,
controlled by notmuch-show-elide-non-matching-messages.  This is usually
toggled with a prefix arg to RET (e.g. the command that shows a thread).

It employs a search you didn't include explicitly in your list:

* [ ] expand all matching messages and all messages having
      any of the specified flags.

I find it quite useful.

As for your proposal, I'll respond to each:

> * [ ] expand all matching messages

Agreed.  This is the default today.

>   [ ] expand messages having any of the specified tags

See above: I think it makes sense to expand the both the matching
messages and those matching the additional expansion tags
(e.g. tag:unread).

>   [ ] expand messages having all of the specified tags

In the context of a configuration for notmuch-search-messages, this
feels overly flexible.  You could just as well include the required tags
in the original query.  It may make sense as a global default?

I call this out because I think as soon as the feature allows expressing
both a tag conjunction (sequence of AND) and tag disjunction (OR), I
begin to wonder why we don't just ask the user to supply a query.

Perhaps you had a use case in mind?

> * expand all messages
> * collapse all messages

Agreed. These are available today.

Re: notmuch.el: controlling what does and doesn't get expanded in searches

[Jan Malakhovski]

Hi.

I struggled with the same problem and wrote the following bit of code to
solve it. I've been using it for a couple of years now and I'm so happy
with it, that I forgot I wanted to get around to fix those TODOs. Feel
free do whatever with it. Hope it helps.

Cheers,
  Jan

;; A bit of notmuch-show-hook magic
;;
;; It makes notmuch-show
;; * show messages matching the specific query and hide the rest, or, if the query matches the whole thread,
;; * show unread or flagged messages of the thread and hide the rest, or, if there're no such messages,
;; * show all the messages
;; and jump to the very first one of those.
;;
;; This saves a lot of key presses.
;;
;; TODO: more generic one: by search query
;; TODO: rewrite using notmuch-show-mapc
(defun oxij/notmuch-show-only-tags (tags)
  "In notmuch-show-mode show messages matching tags, hide the rest"
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (loop do
      (let ((props (notmuch-show-get-message-properties)))
           (if (remove-if-not (lambda (e) (member e tags))
                              (plist-get props :tags))
               (notmuch-show-message-visible props t)
               (notmuch-show-message-visible props nil)))
    until (not (notmuch-show-goto-message-next)))))

;; TODO: how do I break out of mapc?
(defun oxij/notmuch-show-interesting ()
  "Hide boring messages when all messages in the buffer match the specified query"
  (let ((all-matched t))
    (notmuch-show-mapc (lambda ()
      (let ((props (notmuch-show-get-message-properties)))
        (unless (plist-get props :match)
          (setq all-matched nil)))))
    (when all-matched
      (oxij/notmuch-show-only-tags '("unread" "flagged")))))

;; notmuch-show-goto-first-wanted-message will be called just after
;; this hook and will undo the result of this when
;; oxij/notmuch-show-interesting hides everything. see the source of
;; notmuch-show-goto-first-wanted-message for details
(add-hook 'notmuch-show-hook 'oxij/notmuch-show-interesting)