notmuch 'index' mode.

notmuch 'index' mode.

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 3330 bytes --]


I posted a patch adding an 'index' mode to notmuch and though I'd
explain my idea. Most mail systems provide a 'folder view' mode which
displays the set of folders and a count of messages in each folder. I
used this myself as the first sort of which messages I want to read.

Notmuch doesn't have folders, but it does have tags, so I thought I'd
take advantage of that, along with the quick searching ability in Xapian
to construct something much like the folder view.

In 'index' mode, notmuch shows a list of 'folders', each of which is
defined by a Xapian query. The new 'notmuch count' command is used to
figure out how many messages matching the query are in the
database. When you hit 'enter' (or click the mouse) on any line, it
executes the search for real using the notmuch-search command.

Here's the set of indexes I'm using right now (oddly, it looks a lot
like the set of tags I auto-configure in my notmuch-poll script).

(setq notmuch-indexes '(("inbox" . "tag:inbox")
			("me" . "tag:inbox AND tag:me")
			("announce" . "tag:inbox AND tag:announce")
			("bugs" . "tag:inbox AND tag:bugs")
			("cairo" . "tag:inbox AND tag:cairo")
			("debian" . "tag:inbox AND tag:debian");
			("hackers" . "tag:inbox AND tag:hackers")
			("intel" . "tag:inbox AND tag:intel")
			("mesa" . "tag:inbox AND tag:mesa")
			("notmuch" . "tag:inbox AND tag:notmuch")
			("oes" . "tag:inbox AND tag:oes")
			("rockets" . "tag:inbox AND tag:rockets")
			("x" . "tag:inbox AND tag:x")
			("xorg-board" . "tag:inbox AND tag:xorg-board")
			("xpatches" . "tag:inbox AND tag:x AND tag:patch")))

And here's what I see in the *notmuch-index* buffer:

inbox		3
me		3
announce	0
bugs		0
cairo		0
debian		0
hackers		0
intel		0
mesa		0
notmuch		0
oes		0
rockets		0
x		0
xorg-board	0
xpatches	0

This is all pretty simple, and it does much of what I want. However, one
could always get more :-)

One missing piece here is that the searches don't have a
total/unread count. Because the 'notmuch count' command is so fast
(notmuch count the returns 532891 on my machine, and takes 0.240
seconds), we really could count the number of messages matching a single
tag and then *also* count the number of messages matching that tag and
the inbox tag.

Another missing piece is that these should be shared with the
notmuch-poll script, probably via the .notmuch-config file. I do like
that I can have some entries here that aren't in my notmuch-poll script.

It might be nice to make this hierarchical as I used to do with
evolution's vfolder stuff -- an 'intel' folder with an 'intel/unread'
sub-folder. That would be done with simple conjunctions of tags, but
providing some structure on the screen might be nice.

I'm also wondering if we should elide, or at least change the appearance,
of lines which match zero messages.

I'd also like to be able to get the system to display my mail in three
panes -- one with the index, one with the current search and the last
with the current thread. Getting those laid out on the screen correctly
and letting you get them back to that layout easily would make notmuch
look a lot more like other email clients, a form I've found fairly
functional as I do not generally read mail in a linear fashion.

-- 
keith.packard@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: notmuch 'index' mode.

[Carl Worth]

On Fri, 20 Nov 2009 23:35:29 -0800, Keith Packard <keithp@keithp.com> wrote:
> I posted a patch adding an 'index' mode to notmuch and though I'd
> explain my idea. Most mail systems provide a 'folder view' mode which
> displays the set of folders and a count of messages in each folder. I
> used this myself as the first sort of which messages I want to read.

I've just pushed this set of patches out now.

I like it quite a bit. Here are some thoughts:

  * The mode documentation really needs to walk the user through how to
    setup a custom set of folders.

  * It's not opening my "to-me" folder for some reason. (I thought it
    was the '-' at first, but "xorg-board" is working fine). I can debug
    this later.

  * The presentation is impressively spartan[*]. :-)

If we spruce this up a bit, I think we'll want to make this the default
view of "M-x notmuch".

Oh, and instead of just documenting how to set a variable in .emacs we
should just provide commands to add/remove folders.

> One missing piece here is that the searches don't have a
> total/unread count. Because the 'notmuch count' command is so fast
> (notmuch count the returns 532891 on my machine, and takes 0.240
> seconds), we really could count the number of messages matching a single
> tag and then *also* count the number of messages matching that tag and
> the inbox tag.

I think the number of threads matching the search is the only
interesting number actually. Otherwise, you just end up printing a bunch
of big numbers that the user doesn't need for anything.

> Another missing piece is that these should be shared with the
> notmuch-poll script, probably via the .notmuch-config file. I do like
> that I can have some entries here that aren't in my notmuch-poll script.

Yes, once we add configuration for automated tagging, we'll want to
figure out how to include those here. I'm not sure if they should just
be an initial default that the user tweaks, or if any configured rules
from that file should always be presented until the user explicitly asks
for one to be ignored (think about a "spam" tag). So that might take
some experimentation.

You're definitely right that we want to support searches other than
simply the tag names in the configuration file. For example, here's the
search I'm currently using for the "notmuch" folder (so I can process
outstanding patches in order):

	tag:notmuch and (tag:inbox or tag:todo)

Though, really, I probably wouldn't mind using a search string like that
for any of my automated tags. It's equivalent to what you were doing if
there are no "todo" tags on messages with the tag of interest, and then
as soon as there are any "todo" tags on any of those messages, then I
probably do want them in that folder.

So maybe the user configures a search template to use for each automatic
tag or so?

> It might be nice to make this hierarchical as I used to do with
> evolution's vfolder stuff -- an 'intel' folder with an 'intel/unread'
> sub-folder. That would be done with simple conjunctions of tags, but
> providing some structure on the screen might be nice.
> 
> I'm also wondering if we should elide, or at least change the appearance,
> of lines which match zero messages.

Two nice ideas.

> I'd also like to be able to get the system to display my mail in three
> panes -- one with the index, one with the current search and the last
> with the current thread. Getting those laid out on the screen correctly
> and letting you get them back to that layout easily would make notmuch
> look a lot more like other email clients, a form I've found fairly
> functional as I do not generally read mail in a linear fashion.

Yes. Emacs has pretty good support for saving and restoring an entire
window configuration, so we should be able to make this work.

And by the way, lest you be misled by the "magic space bar", I don't
read my mail very linearly either. I always filter by at least one tag
first and I want to prioritize which one. The folder mode is great for
this. It's only when I get my search results down below a particular
threshold that I decide I want to read everything linearly.

(And if I can't get a particular folder down that small, then I'll do
selective reading (with 'x', yes, I think we're agreeing on that now),
and then archive away the lot.)

Anyway, thanks for the great addition to notmuch.

-Carl

[*] A nice thing about this is it give us plenty of room to have a
paragraph or two explaining how notmuch works, what the user can expect,
what keys to use, etc.

Re: notmuch 'index' mode.

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 2992 bytes --]

On Mon, 23 Nov 2009 07:06:06 +0100, Carl Worth <cworth@cworth.org> wrote:
> On Fri, 20 Nov 2009 23:35:29 -0800, Keith Packard <keithp@keithp.com> wrote:
> > I posted a patch adding an 'index' mode to notmuch and though I'd
> > explain my idea. Most mail systems provide a 'folder view' mode which
> > displays the set of folders and a count of messages in each folder. I
> > used this myself as the first sort of which messages I want to read.
> 
> I've just pushed this set of patches out now.
> 
> I like it quite a bit. Here are some thoughts:
> 
>   * The mode documentation really needs to walk the user through how to
>     setup a custom set of folders.

You can use the standard emacs customization interface, although it's
not exactly well documented there. Additional docs are clearly
indicated.

>   * It's not opening my "to-me" folder for some reason. (I thought it
>     was the '-' at first, but "xorg-board" is working fine). I can debug
>     this later.

It shouldn't work on xorg-board either -- it just skips forward one
'word' and assumes it's gotten the folder name. Lame programming, I
know. Should be easy to fix.

>   * The presentation is impressively spartan[*]. :-)

What more do you need? Pretty fonts? Icons? A pony?

> If we spruce this up a bit, I think we'll want to make this the default
> view of "M-x notmuch".

Let's figure out what we want it to look like then. I suggest that we
use some font tricks (bold/regular) to highlight folders with
un-archived mail. I'd also like to be able to create a hierarchy.

> Oh, and instead of just documenting how to set a variable in .emacs we
> should just provide commands to add/remove folders.

That seems doable; it really only takes two text fields after all.

> I think the number of threads matching the search is the only
> interesting number actually. Otherwise, you just end up printing a bunch
> of big numbers that the user doesn't need for anything.

Threads or messages? Threads are more expensive to compute, and when the
number is zero, it means the same thing. But, it is surprising to see a
huge number in the folder view and then get only a handful of threads in
the search view. Perhaps if the count is small enough we can go through
and actually figure out how many threads are involved.

> So maybe the user configures a search template to use for each automatic
> tag or so?

Ack! I suggest that the simpler approach of just duplicating the search
field will work fine in practice. Otherwise, you're talking miles of UI
goo, and constantly confused users. Really, you can use emacs to edit
these things if you get too many.

> It's only when I get my search results down below a particular
> threshold that I decide I want to read everything linearly.

I have started doing this sometimes too. But, I still skim things like
the mesa list and only read a few by clicking through and 'x'ing back.

-- 
keith.packard@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: notmuch 'index' mode.

[Carl Worth]

On Sun, 22 Nov 2009 23:18:33 -0800, Keith Packard <keithp@keithp.com> wrote:
> On Mon, 23 Nov 2009 07:06:06 +0100, Carl Worth <cworth@cworth.org> wrote:
> > 
> >   * The mode documentation really needs to walk the user through how to
> >     setup a custom set of folders.
> 
> You can use the standard emacs customization interface, although it's
> not exactly well documented there. Additional docs are clearly
> indicated.

Yes. I think that's a bit too hidden.

I'm imagining the case of a new user running notmuch. First, if we're
going to allow them to start off from inside emacs rather than the
command-line (and why not?) then I think we can pretty much just run
"notmuch setup" and "notmuch new" for the user (and those walk the user
through things well already.

Then, I'd like the user to be seeing the folder view right away (since
that's the primary view). We've got support for searching while still
indexing, so if we refresh occasionally, the user will be able to see
the inbox count just count up while indexing occurs.

So what next for the user? We've got lots of empty space in the folder
view, so I think what I'd like is to provide some suggestions on search
strings to try, and then to make it obvious how to associate a name with
a search string. And at that point, I'd like that name to be added to
the folder view.

And notice, I don't think we'll need to do the "virtual tag" thing of
associating a tag name with a search and doing the work of making
notmuch maintain the consistency of that tag and search string. It will
be much more clean (and shouldn't be any less fast) to just do the
searches for the original search terms, (rather than searching for tag
terms that were previously applied by searching for the search terms).

Then tags become something that are just for manual manipulation. What
do you think?

> >   * It's not opening my "to-me" folder for some reason. (I thought it
> >     was the '-' at first, but "xorg-board" is working fine). I can debug
> >     this later.
> 
> It shouldn't work on xorg-board either -- it just skips forward one
> 'word' and assumes it's gotten the folder name. Lame programming, I
> know. Should be easy to fix.

Ah, that's it. So "xorg-board" only looked like it worked because I had
a "xorg" folder as well. :-)

> What more do you need? Pretty fonts? Icons? A pony?

Heh. Just some guidance on what the user can do. It's nice that we have
room to do this, (since we don't on the search results view).

> Threads or messages? Threads are more expensive to compute, and when the
> number is zero, it means the same thing. But, it is surprising to see a
> huge number in the folder view and then get only a handful of threads in
> the search view. Perhaps if the count is small enough we can go through
> and actually figure out how many threads are involved.

Maybe something like that.

-Carl

Re: notmuch 'index' mode.

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 1379 bytes --]

On Mon, 23 Nov 2009 19:16:54 -0800, Carl Worth <cworth@cworth.org> wrote:

> So what next for the user? We've got lots of empty space in the folder
> view, so I think what I'd like is to provide some suggestions on search
> strings to try, and then to make it obvious how to associate a name with
> a search string. And at that point, I'd like that name to be added to
> the folder view.

Right, associate a name for the search from the *search* view instead of
the folder view. Then it's just a matter of using that command and
typing a name. At that point, you get the new folder (or named search)
added to the main window. That's a great plan.

> Then tags become something that are just for manual manipulation. What
> do you think?

So you wouldn't generally end up using tags? And each time you wanted to
view a folder you'd really get a new search result? I can see some
benefits if you're using time-range based searches (show me all the mail
From Carl that was received in the last week). And disadvantages as
searching might actually be slow at some point?

> Ah, that's it. So "xorg-board" only looked like it worked because I had
> a "xorg" folder as well. :-)

Yea, sorry 'bout that.

> Maybe something like that.

Let's figure it out later; it's just a number and we can change it
without any penalty.

-- 
keith.packard@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: notmuch 'index' mode.

[Jed Brown]

On Mon, 23 Nov 2009 19:43:26 -0800, Keith Packard <keithp@keithp.com> wrote:
> On Mon, 23 Nov 2009 19:16:54 -0800, Carl Worth <cworth@cworth.org> wrote:
> > Then tags become something that are just for manual manipulation. What
> > do you think?

I really think this is the way to go.

> And disadvantages as searching might actually be slow at some point?

If this happens, there is nothing to prevent notmuch from caching the
search by actually writing a corresponding tag.  This could be made
automatic by logging the cost of each named search (and perhaps the
frequency of making that search), and using a tradeoff function to
decide which searches to optimize.  Once a search was selected for
optimization, there are at least two alternatives, depending on which
queries xapian can answer quickly.

  1. Log the time and spawn an asynchronous notmuch tag process.  Searches
  for
  
    vtag:named-search
  
  (vtag: doesn't need to be a keyword, but I'm only distinguishing here
  for clarity) which was normally translated into
  
     long search expression
  
  becomes
  
    tag:named-search OR newer:timestamp AND (long search expression)
  
  This option guarantees that notmuch new remains fast and simple because
  it does no special tagging, but this query might not be any better.
  
  2. Inform notmuch new that it should apply the chosen tag to messages
  matching the query and spawn the asynchronous notmuch tag process.  Once
  the tag process has finished, searches for vtag:named-search are
  translated to tag:named-search.  This implies concurrent modification of
  the database, otherwise it would cause a stall in incoming mail (not
  important if mass tagging somehow became faster).
  

Admittedly my archive is not huge (100k messages in 3.5 GB plus 1.1 GB
for .notmuch) but queries returning a reasonable number of messages are
still quite fast.  Additionally, searches for tags do not seem to be
greatly faster than queries for complex queries returning a similar
number of results.

Jed

Re: notmuch 'index' mode.

[Jan Janak]

On Tue, Nov 24, 2009 at 4:16 AM, Carl Worth <cworth@cworth.org> wrote:
> And notice, I don't think we'll need to do the "virtual tag" thing of
> associating a tag name with a search and doing the work of making
> notmuch maintain the consistency of that tag and search string. It will
> be much more clean (and shouldn't be any less fast) to just do the
> searches for the original search terms, (rather than searching for tag
> terms that were previously applied by searching for the search terms).
>
> Then tags become something that are just for manual manipulation. What
> do you think?

I like this. I think this is much cleaner than the "virtual tag" approach.

  -- Jan

Re: notmuch 'index' mode.

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

On Tue, 24 Nov 2009 13:14:07 +0100, Jan Janak <jan@ryngle.com> wrote:

> I like this. I think this is much cleaner than the "virtual tag"
> approach.

A disadvantage I see is that you would not see this 'virtual tags' in
the list of tags on each message. And, we'd have to put the virtual tags
in the .config file or the command line would become a lot less
useful. Speaking of which, we should put the folders in the .config file
in any case, and provide command line syntax to use them.

-- 
keith.packard@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: notmuch 'index' mode.

[Jed Brown]

On Tue, 24 Nov 2009 08:38:20 -0800, Keith Packard <keithp@keithp.com> wrote:
> A disadvantage I see is that you would not see this 'virtual tags' in
> the list of tags on each message.

Is it not cheap to check the current search results against the vtags
(named searches) that you have defined.  Then you can show them right up
there with the real tags.

Jed

Re: notmuch 'index' mode.

[Jan Janak]

On 24-11 08:38, Keith Packard wrote:
> On Tue, 24 Nov 2009 13:14:07 +0100, Jan Janak <jan@ryngle.com> wrote:
> 
> > I like this. I think this is much cleaner than the "virtual tag"
> > approach.
> 
> A disadvantage I see is that you would not see this 'virtual tags' in
> the list of tags on each message. 

Maybe I don't understand how virtual tags are really supposed to work. I
thought that the term "virtual tags" is just another name for "named" search
terms and it is named "virtual tags" because they can actually be attached to
messages stored in the database. And if I remember it correctly, Carl said
somewhere that virtual tags will be added/removed automatically to messages if
the corresponding search term gets changed.

If my understanding is correct, then we won't be able to show virtual tags
along with "normal" tags anyway. We would probably need a separate list just
for virtual tags. One of the reasons is that you can add and remove normal
tags to messages manually, but you cannot easily do the same with virtual
tags.

If you remove a virtual tag from a message and then update the search term
bound to that virtual tag, as soon as you change the search term, the virtual
tag might be added to the message again if the new search expression still
matches the message.

Unlike real tags, the user won't be able to add and remove virtual tags to
messages arbitrarily without the risk that notmuch changes the selection
behind his back on the next update of the configuration file (assuming that
the search expression is stored in the configuration file).

We would probably need to make sure that the user knows which tags are virtual
and which are real and that means notmuch would have to present them
differently, possibly in a separate list.

Thinking more about this, we could probably make it possible to add and remove
virtual tags to messages manually by storing the information about what the
user did with the virtual tag along with the message (i.e. this virtual tag is
pinned to the message so don't remove it when the search term is changed).

Maybe I just don't understand how virtual tags are really supposed to work,
but it seems to me as a can of worms just waiting to be opened :-).

  -- Jan

Re: notmuch 'index' mode.

[Bart Trojanowski]

* Keith Packard <keithp@keithp.com> [091124 11:38]:
> A disadvantage I see is that you would not see this 'virtual tags' in
> the list of tags on each message. And, we'd have to put the virtual tags
> in the .config file or the command line would become a lot less
> useful. Speaking of which, we should put the folders in the .config file
> in any case, and provide command line syntax to use them.

I too would vote for this approach.  I'd like to keep as much smarts in
the notmuch executable as it would mean less to port to other (non
emacs) interfaces.

-Bart

-- 
				WebSig: http://www.jukie.net/~bart/sig/