Initial attempt at a "merge window" for notmuch

Initial attempt at a "merge window" for notmuch

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 4315 bytes --]

I want to figure out how to make a release process that will work for
notmuch. An idea I have now is to use a date-based "merge window" during
which features are nominated for the release.

So I took a window starting at my original request for 0.2 features
until now. In our current (lame) syntax for specifying date ranges,
that gives me a search string of:

tag:notmuch and 1270678364..1270828505 and not (tag:merged or tag:postponed)

What I want to do now is to work through this list and tag things as
either "merged" or "postponed" until my list is empty. That should give
a good indication that we've actually got all the features we
want. (We'll still need something more for tracking bugs, of course.)

Here's the current list:

[22/22] Carl Worth, James... Plans for the 0.2 release (this week)
[2/6]   Carl Worth, Micha... Notmuch release 0.1 now available
[2/4]   Dirk Hohndel, Car... [PATCH] Fix code extracting the MTA from Received: headers
[1/2]   Mike Kelly, Carl ... [PATCH] Fix the default value for --includedir.
[2/5]   David Edmondson, ... [notmuch] [PATCH] notmuch.el: 'F' in search mode takes us to a list of folders.
[2/2]   Sebastian Spaeth,... RFC: User-Agent header
[1/30]  Aneesh Kumar K.V,... [notmuch] [PATCH] notmuch: Add Maildir directory name as tag name for messages
[3/10]  Sebastian Spaeth,... [notmuch] [Sebastian Spaeth] Pull requests
[5/5]   Michal Sojka         [PATCH 0/4] Mailstore abstraction v4
[2/2]   Michal Sojka         [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)
[2/2]   Mike Kelly, Sebas... [PATCH] Have notmuch count default to showing the total.
[3/3]   Mike Kelly           [PATCH 1/3] Initial support for maildir flags.
[1/2]   Dirk Hohndel, Mic... [RFC] reordering and cleanup of thread authors
[1/10]  Jesse Rosenthal, ... [notmuch] [PATCH] notmuch.el: Make notmuch-show buffer name first subject, instead of thread-id
[1/11]  Jesse Rosenthal, ... [notmuch] [PATCH] notmuch.el: add functionality to add or remove tags by region.

Of course, just looking at this list makes me want many more features
(but they are too late for this merge window):

  1. Real support for date-range-based searches

  2. The ability to split a thread.

     I know I argued against this previously, but I know that "Plans for
     the 0.2 release" contains multiple independent topics, and I'd
     really like to be able to track those separately.

  3. Ability to easily post search results to a web page.

     I'd like people to be able to easily view a web page with three
     lists on it for tracking the upcoming release: "proposed features",
     "merged features", and "postponed features".

  4. Fancy support for thread- vs. message-based search matches.

     We've talked about supporting a "muted" tag to make obnoxious
     threads disappear entirely. The idea we have there is a tag on a
     message, and then support for searches which compute set-theoretic
     operations based on sets of threads. So I want the set of threads
     which have a message of "tag:inbox" and subtract from that the set
     of threads which have a message of "tag:muted".

     Note that this result is different than what we can currently
     compute which is a set of threads containing messages that match
     "tag:inbox and not tag:muted". This will return threads that I have
     muted when new messages arrive to the thread after I added the
     muted tag to the older messages. So we do need some new support
     here.

     For my merge window, I also want something that can't be obtained
     today. I want to see all threads that contain at least one message
     that matches my date range and at least one message that doesn't
     have the "merged" or "postponed" tag. That is, I can merge a
     feature and mark it "merged", but if someone replies later noticing
     a defect in the patch I merged, then I want that thread to reappear
     in my search results. But my current date restriction prevents
     that.

     I'm not sure how to best provide the ability to express the result
     I want here. But clearly, we want to come up with a syntax that's
     powerful enough to capture these kinds of things. (And I think this
     will go beyond the search capabilities of most email systems that
     I'm aware of.)

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Initial attempt at a "merge window" for notmuch

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

On Fri, 09 Apr 2010 09:23:27 -0700, Carl Worth <cworth@cworth.org> wrote:
>      For my merge window, I also want something that can't be obtained
>      today. I want to see all threads that contain at least one message
>      that matches my date range and at least one message that doesn't
>      have the "merged" or "postponed" tag.
...
>      I'm not sure how to best provide the ability to express the result
>      I want here.

Of course, it's the same set-theoretic operation I described earlier. I
want the set of threads having messages matching:

	tag:notmuch and <date-range>

intersected with the set of threads having messages matching:

	tag:notmuch and not ("merged" or "postponed")

So I've got uses cases for set-difference and intersection already. Now
we just need some search syntax to express that.

-Carl


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Initial attempt at a "merge window" for notmuch

[Anthony Towns]

On Sat, Apr 10, 2010 at 02:23, Carl Worth <cworth@cworth.org> wrote:
>  3. Ability to easily post search results to a web page.

Isn't that a job for "noneatall" [0] -- maybe it just needs the
ability to export to static pages, that can be rsync'ed somewhere?

[0] http://github.com/dme/noneatall

>  4. Fancy support for thread- vs. message-based search matches.
>     We've talked about supporting a "muted" tag to make obnoxious
>     threads disappear entirely. The idea we have there is a tag on a
>     message, and then support for searches which compute set-theoretic
>     operations based on sets of threads.

Is that an argument for tags on a thread rather than a message? With a
message mute tag, if you happen to delete the single message you've
tagged muted (maybe you killfile the sender of that message), suddenly
the whole thread reappears, which doesn't sound entirely desirable.

>  2. The ability to split a thread.
>     I know I argued against this previously, but I know that "Plans for
>     the 0.2 release" contains multiple independent topics, and I'd
>     really like to be able to track those separately.

An MUA that did that well (or just effectively) would be massively fantastic.

Perhaps you could do it by associating threads with any message, so if
you've got an announcement A, with three followups, B, C and D, of
which C and D are interesting and novel subthreads, you could have
thread:00001 start at A and include everything, thread:00002 manually
pointed at C and include both its parents (A) and any children, but
not any siblings/cousins (B/D), and thread:00003 manually pointed at D
and behave likewise (including A, any responses to D, but not B, C or
any replies to B or C).

I don't know how you'd handle a mail, E, that came in that following
up to C though -- do you just report thread:00001 as having new mail,
or just thread:00002, or both of them? What if F comes in that
simultaneously replies to E and D? (Personally, I could probably be
comfortable with any of those behaviours)

>     For my merge window, I also want something that can't be obtained
>     today. I want to see all threads that contain at least one message
>     that matches my date range and at least one message that doesn't
>     have the "merged" or "postponed" tag. That is, I can merge a
>     feature and mark it "merged", but if someone replies later noticing
>     a defect in the patch I merged, then I want that thread to reappear
>     in my search results. But my current date restriction prevents
>     that.

The above hypothetical features could let you do:

    # create new threads at messages that are marked as postponed or merged

    notmuch newthread -- not is:threadroot \( tag:merged or tag:postponed \)

    # assume threads get the same tags as their "root" message, so that
    # any messages in the above new threads will automagically match on
    # threadtag:merged or threadtag:postponed. Thus:

    notmuch search threadtag:merged 123456..123789

That's abusing subthreads as a poor man's set. Not really convinced
that's a good idea, but what the hey... Something to think about
maybe.

Cheers,
aj

-- 
Anthony Towns <aj@erisian.com.au>

Re: Initial attempt at a "merge window" for notmuch

[Mark Anderson]

On Fri, 9 Apr 2010 11:29:20 -0500, Carl Worth <cworth@cworth.org> wrote:
> On Fri, 09 Apr 2010 09:23:27 -0700, Carl Worth <cworth@cworth.org> wrote:
> >      For my merge window, I also want something that can't be obtained
> >      today. I want to see all threads that contain at least one message
> >      that matches my date range and at least one message that doesn't
> >      have the "merged" or "postponed" tag.
> ...
> >      I'm not sure how to best provide the ability to express the result
> >      I want here.
> 
> Of course, it's the same set-theoretic operation I described earlier. I
> want the set of threads having messages matching:
> 
> 	tag:notmuch and <date-range>
> 
> intersected with the set of threads having messages matching:
> 
> 	tag:notmuch and not ("merged" or "postponed")
> 
> So I've got uses cases for set-difference and intersection already. Now
> we just need some search syntax to express that.
> 

Can we just start grouping with a set:( ) or { } on the command line?
I'm guessing the parentheses are slightly easier.

   set:( tag:notmuch and <date-range> ) 
     isect 
   set:( tag:notmuch and not (tag:merged or tag:postponed) )

Isn't that close to what you're asking for?

It seems easy enough, and making the set:( explicit keeps you from
inventing a new quoting syntax if you tried to say 

  set:"tag:notmuch and <date-range>" 

then users get to play shell-quote-mambo to actually use this outside of
a client if they want (or think they want) quotes used in their search.

You can reuse the "and" and "or" term for set math, unless you are
averse to overloading, then isect, union are explicit enough about the
terms that you could infer the "set:" term for searches:

    tag:notmuch and <date-range> isect not ( tag:postponed or tag:merged ) 

But at the cost of introducing a new order of ops hierarchy.  I'm not
sure if you want to go there either.

Just thinking about completeness, I wonder if there are some searches
for threads that aren't currently available.

You could introduce a search modifier that would allow you to treat a
tag on a message in a thread as a tag on the entire thread, so that if
you have tag:mute on on message and tag:merged on another message in the
same thread, currently that does match:

  tag:merged and not tag:mute

But maybe you wanted only the THREADS instead of THREAD CONTAINS
searching.

I guess we're hampered by the fact that a thread object isn't a separate
object from the messages which comprise it's conversation, so we can't
look at the tags on a thread separately from the tags on messages in the
thread.  And maybe we don't want to.

Actually, this is where we differ slightly from gmail.  They have tags
on threads, and unread tags on messages.

I don't know that there's value in distinguishing between tags on a
thread and tags on a message in a thread when there isn't an object in
the mailstore that actually corresponds to a thread.

Hmmm, this isn't clear enough, so I'll just stop the rambling, but let
it stand.

-Mark

P.S. Sometimes I write a long note and don't get to the last sentence
for an hour or two.  I can be funny that way.

Re: Initial attempt at a "merge window" for notmuch

[Sebastian Spaeth]

On 2010-04-09, Carl Worth wrote:
> 
> Here's the current list:

Having such a list is great. So now we need to get you a key-binding
that says, "take query xy results and stuff them into the wiki" (or some
pastebin like service with a fixed URL). Looking forward to 0.2.

Sebastian

(THanks to the debian pkgs I can now use notmuch on my netbook which has
not compiler installed. Very useful, thanks. Now the issue of syncing
tags is becoming more pressing for me :-) )

Re: Initial attempt at a "merge window" for notmuch

[Michal Sojka]

On Fri, 09 Apr 2010, Mark Anderson wrote:
> On Fri, 9 Apr 2010 11:29:20 -0500, Carl Worth <cworth@cworth.org> wrote:
> > Of course, it's the same set-theoretic operation I described earlier. I
> > want the set of threads having messages matching:
> > 
> > 	tag:notmuch and <date-range>
> > 
> > intersected with the set of threads having messages matching:
> > 
> > 	tag:notmuch and not ("merged" or "postponed")
> > 
> > So I've got uses cases for set-difference and intersection already. Now
> > we just need some search syntax to express that.
> > 
> 
> Can we just start grouping with a set:( ) or { } on the command line?
> I'm guessing the parentheses are slightly easier.
> 
>    set:( tag:notmuch and <date-range> ) 
>      isect 
>    set:( tag:notmuch and not (tag:merged or tag:postponed) )

If we go in this direction, I think that the syntax should explicitely
state the it is the set of threads and not the set of messages. So maybe
something like

  threads:( ... ) isect threads:( ... )

> Just thinking about completeness, I wonder if there are some searches
> for threads that aren't currently available.

I think that having a way for searching all threads started by a certain
person (e.g. project maintainer) would be very useful. For this we may
need some search operator for specifying the position of the message in
the thread. For example: notmuch search from:cworth and position:first.

In id:4b9d4e24.0f67f10a.31e3.ffffadf7@mx.google.com, Sandra asked for
something like: notmuch search not threads:( from:me and position:last )

-Michal

Re: Initial attempt at a "merge window" for notmuch

[Sandra Snan]

On Sat, 10 Apr 2010 22:24:01 +0200, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> In id:4b9d4e24.0f67f10a.31e3.ffffadf7@mx.google.com, Sandra asked for
> something like: notmuch search not threads:( from:me and position:last )

Naw, actually I forgot to follow that up. I was looking for it to work
it already worked, I just had a little PEBKAC going on.

Basically, what I wanted was to tag specific *messages* based on
search criteria and I was under the impression that the system tagged
*threads*, but after using it for a while, it seems that it does tag
messages.