folder and path completely broken in HEAD?

folder and path completely broken in HEAD?

[dm-list-email-notmuch]

Hey, I'm playing around with the head of the git repository
(bc64cdce289d84be2550c4fccb1f008d15eaeb0e) to try to figure out how the
new folder: prefixes work, as folders are a critical part of how I
organize my mail.  (Since tags are not hierarchical, folders are the
best way for me to group mail to a bunch of related addresses, while
retaining the ability to separate out any mailboxes that become high
traffic.)

I'm using a pretty standard maildir++ layout.  For example, underneath
my database.path I have a bunch of mail in directories such as:

    .INBOX.Main/{new,cur}
    .mail.class/{new,cur}
    .mail.voicemail/{new,cur}

It used to be the case that if I wanted to read all of my "mail" mail, I
could search for folder:mail, while to look at just voicemail, I could
say folder:mail.voicemail, etc.  Now, I can't get anything to match a
folder predicate period.  For example, using notmuch as notmuch-0.17 and
./notmuch as notmuch-0.18-rc2+2~gbc64cdc, here's what I get:

        linux2$ notmuch count folder:mail
        16257
        linux3$ notmuch count folder:mail.class
        1896
        linux4$ notmuch count folder:mail.voicemail
        34
        linux5$ notmuch count folder:mail.voicemail/cur
        34
        linux6$ notmuch count folder:.mail.voicemail/cur
        34
        linux7$ ./notmuch count folder:mail
        0
        linux8$ ./notmuch count folder:.mail
        0
        linux9$ ./notmuch count folder:.mail.voicemail
        0
        linux10$ ./notmuch count folder:.mail.voicemail/cur
        0
        linux 11$ ./notmuch count path:.mail.voicemail
        0
        linux 12$ ./notmuch count path:.mail.voicemail/'**'
        0
        linux 13$ ./notmuch count path:.mail.voicemail/cur 
        0
        linux 14$ ./notmuch count folder:mail.voicemail
        0

What gives?  Are the path and folder predicates completely broken, or is
something very important missing from the new notmuch-search-terms
manual page?  How can I make this work?

Thanks,
David

Re: folder and path completely broken in HEAD?

[Mark Walters]

Hi

Before checking other things: have you run notmuch new? That's needed to
update the database. It is an irreversible update so notmuch-0.17 will
not work with the updated database.

The update should save a dump of the tags before updating but you might
want to do that manually too (and maybe even keep a complete copy of the
database). The update might take quite a long time (but it can be
interrupted and restarted)

Best wishes

Mark


On Fri, 02 May 2014, dm-list-email-notmuch@scs.stanford.edu wrote:
> Hey, I'm playing around with the head of the git repository
> (bc64cdce289d84be2550c4fccb1f008d15eaeb0e) to try to figure out how the
> new folder: prefixes work, as folders are a critical part of how I
> organize my mail.  (Since tags are not hierarchical, folders are the
> best way for me to group mail to a bunch of related addresses, while
> retaining the ability to separate out any mailboxes that become high
> traffic.)
>
> I'm using a pretty standard maildir++ layout.  For example, underneath
> my database.path I have a bunch of mail in directories such as:
>
>     .INBOX.Main/{new,cur}
>     .mail.class/{new,cur}
>     .mail.voicemail/{new,cur}
>
> It used to be the case that if I wanted to read all of my "mail" mail, I
> could search for folder:mail, while to look at just voicemail, I could
> say folder:mail.voicemail, etc.  Now, I can't get anything to match a
> folder predicate period.  For example, using notmuch as notmuch-0.17 and
> ./notmuch as notmuch-0.18-rc2+2~gbc64cdc, here's what I get:
>
>         linux2$ notmuch count folder:mail
>         16257
>         linux3$ notmuch count folder:mail.class
>         1896
>         linux4$ notmuch count folder:mail.voicemail
>         34
>         linux5$ notmuch count folder:mail.voicemail/cur
>         34
>         linux6$ notmuch count folder:.mail.voicemail/cur
>         34
>         linux7$ ./notmuch count folder:mail
>         0
>         linux8$ ./notmuch count folder:.mail
>         0
>         linux9$ ./notmuch count folder:.mail.voicemail
>         0
>         linux10$ ./notmuch count folder:.mail.voicemail/cur
>         0
>         linux 11$ ./notmuch count path:.mail.voicemail
>         0
>         linux 12$ ./notmuch count path:.mail.voicemail/'**'
>         0
>         linux 13$ ./notmuch count path:.mail.voicemail/cur 
>         0
>         linux 14$ ./notmuch count folder:mail.voicemail
>         0
>
> What gives?  Are the path and folder predicates completely broken, or is
> something very important missing from the new notmuch-search-terms
> manual page?  How can I make this work?
>
> Thanks,
> David
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[David Mazieres expires 2014-07-31 PDT]

Mark Walters <markwalters1009@gmail.com> writes:

> Hi
>
> Before checking other things: have you run notmuch new? That's needed to
> update the database. It is an irreversible update so notmuch-0.17 will
> not work with the updated database.

No, I haven't.  Okay, so that must be it.  Sorry for bothering people.
The discussion of the NEWS was a bit confusing, so I wanted to check it
out.  I knew something had to be very wrong.

Weill the new primitives still allow me to group my mail hierarchically
in searches?  The new man page is not totally clear on what is being
matched.

Thanks,
David

Re: folder and path completely broken in HEAD?

[Jani Nikula]

On Fri, 02 May 2014, dm-list-email-notmuch@scs.stanford.edu wrote:
> Hey, I'm playing around with the head of the git repository
> (bc64cdce289d84be2550c4fccb1f008d15eaeb0e) to try to figure out how the
> new folder: prefixes work, as folders are a critical part of how I
> organize my mail.  (Since tags are not hierarchical, folders are the
> best way for me to group mail to a bunch of related addresses, while
> retaining the ability to separate out any mailboxes that become high
> traffic.)
>
> I'm using a pretty standard maildir++ layout.  For example, underneath
> my database.path I have a bunch of mail in directories such as:
>
>     .INBOX.Main/{new,cur}
>     .mail.class/{new,cur}
>     .mail.voicemail/{new,cur}
>
> It used to be the case that if I wanted to read all of my "mail" mail, I
> could search for folder:mail, while to look at just voicemail, I could
> say folder:mail.voicemail, etc.  Now, I can't get anything to match a
> folder predicate period.  For example, using notmuch as notmuch-0.17 and
> ./notmuch as notmuch-0.18-rc2+2~gbc64cdc, here's what I get:

First, read Mark's reply about database upgrade, and heed the notice
about upgrade being automatic but downgrade requiring a reindex of all
your mails.

Here's additional commentary on the specific queries.

>         linux7$ ./notmuch count folder:mail
>         0
>         linux8$ ./notmuch count folder:.mail
>         0

These two will not match anything.

>         linux9$ ./notmuch count folder:.mail.voicemail
>         0

After the upgrade, this should work for .mail.voicemail/{new,cur}.

>         linux10$ ./notmuch count folder:.mail.voicemail/cur
>         0

This won't match anything. To specifically search for messages in cur,
you can use the new path prefix, path:.mail.voicemail/cur

>         linux 11$ ./notmuch count path:.mail.voicemail
>         0

This will match messages in .mail.voicemail only, and nothing in the cur
or new subdirectories.

>         linux 12$ ./notmuch count path:.mail.voicemail/'**'
>         0

I'm not sure your quoting will work, but if you don't have any other
subfolders than cur and new, this should be equal to
folder:.mail.voicemail

>         linux 13$ ./notmuch count path:.mail.voicemail/cur 
>         0

This should match messages in the cur subdirectory.

>         linux 14$ ./notmuch count folder:mail.voicemail
>         0

This won't match anything.


BR,
Jani.

Re: folder and path completely broken in HEAD?

[dm-list-email-notmuch]

Jani Nikula <jani@nikula.org> writes:

> On Fri, 02 May 2014, dm-list-email-notmuch@scs.stanford.edu wrote:
>>
>> I'm using a pretty standard maildir++ layout.  For example, underneath
>> my database.path I have a bunch of mail in directories such as:
>>
>>     .INBOX.Main/{new,cur}
>>     .mail.class/{new,cur}
>>     .mail.voicemail/{new,cur}
>> ...
> Here's additional commentary on the specific queries.
>
>>         linux7$ ./notmuch count folder:mail
>>         0
>>         linux8$ ./notmuch count folder:.mail
>>         0

Oh, man.  That's a serious bummer.

Is there any mechanism left that would let me hierarchically group
messages?  I've got a ton of mail.* folders, and create new ones
dynamically.  I really want a mechanism to group them hierarchically, so
I can have a search that matches all current and future mail
directories.  I organized my whole mail setup around folders because a)
tags do not provide this kind of hierarchical control, and b) there
doesn't seem to be a convenient way to apply tags 100% reliably on
message delivery, whereas I *can* control the folder 100% reliably.

Worse, because of my poor performance, I was hoping to segregate
messages by year.  So it would be:

  2013/.mail.class
  2013/.mail.voicemail
  2014/.mail.class
  2014/.mail.voicemail

All the way back.  Now you are saying there will be no convenient way to
match just the "mail.class" part without the year?  How very
distressing.  Ugh.

David

Re: folder and path completely broken in HEAD?

[Suvayu Ali]

Hello David,

On Fri, May 02, 2014 at 02:16:07PM -0700, David Mazieres expires 2014-07-31 PDT wrote:
> Mark Walters <markwalters1009@gmail.com> writes:
> >
> > Before checking other things: have you run notmuch new? That's needed to
> > update the database. It is an irreversible update so notmuch-0.17 will
> > not work with the updated database.
> 
> No, I haven't.  Okay, so that must be it.  Sorry for bothering people.
> The discussion of the NEWS was a bit confusing, so I wanted to check it
> out.  I knew something had to be very wrong.
> 
> Weill the new primitives still allow me to group my mail hierarchically
> in searches?  The new man page is not totally clear on what is being
> matched.

Here are some examples:

  $ notmuch config get database.path
  /home/user/Mail
  $ notmuch count -- folder:Gmail/Tux
  5113
  $ notmuch count -- 'path:Gmail/**'
  87807

Hope this helps,

-- 
Suvayu

Open source is the future. It sets us free.

Re: folder and path completely broken in HEAD?

[Suvayu Ali]

On Fri, May 02, 2014 at 09:57:49PM -0700, dm-list-email-notmuch@scs.stanford.edu wrote:
> 
> Worse, because of my poor performance, I was hoping to segregate
> messages by year.  So it would be:
> 
>   2013/.mail.class
>   2013/.mail.voicemail
>   2014/.mail.class
>   2014/.mail.voicemail
> 
> All the way back.  Now you are saying there will be no convenient way to
> match just the "mail.class" part without the year?  How very
> distressing.  Ugh.

If you are using shell commands, you can always use:

  $ notmuch count -- folder:201{3,4}/.mail.class

Not sure how to achieve the same from the Emacs UI (or other UIs).

Hope this helps,

-- 
Suvayu

Open source is the future. It sets us free.

Re: folder and path completely broken in HEAD?

[Mark Walters]

On Sat, 03 May 2014, dm-list-email-notmuch@scs.stanford.edu wrote:
> Jani Nikula <jani@nikula.org> writes:
>
>> On Fri, 02 May 2014, dm-list-email-notmuch@scs.stanford.edu wrote:
>>>
>>> I'm using a pretty standard maildir++ layout.  For example, underneath
>>> my database.path I have a bunch of mail in directories such as:
>>>
>>>     .INBOX.Main/{new,cur}
>>>     .mail.class/{new,cur}
>>>     .mail.voicemail/{new,cur}
>>> ...
>> Here's additional commentary on the specific queries.
>>
>>>         linux7$ ./notmuch count folder:mail
>>>         0
>>>         linux8$ ./notmuch count folder:.mail
>>>         0
>
> Oh, man.  That's a serious bummer.
>
> Is there any mechanism left that would let me hierarchically group
> messages?  I've got a ton of mail.* folders, and create new ones
> dynamically.  I really want a mechanism to group them hierarchically, so
> I can have a search that matches all current and future mail
> directories.  I organized my whole mail setup around folders because a)
> tags do not provide this kind of hierarchical control, and b) there
> doesn't seem to be a convenient way to apply tags 100% reliably on
> message delivery, whereas I *can* control the folder 100% reliably.
>
> Worse, because of my poor performance, I was hoping to segregate
> messages by year.  So it would be:
>
>   2013/.mail.class
>   2013/.mail.voicemail
>   2014/.mail.class
>   2014/.mail.voicemail
>
> All the way back.  Now you are saying there will be no convenient way to
> match just the "mail.class" part without the year?  How very
> distressing.  Ugh.

Hi

I am not quite sure what you are meaning by hierarchically group
messages. Searching for path:dir/foo/bar/** should give all messages in
all directories beneath dir/foo/bar. 

So for the year example you give you could have directories
mail/class/2013 and mail/class/2014/ etc. 

Anyway if you can describe your use more fully we may be able to
help. And if it can't do what you want it would also give us some idea
of what would need to be changed.

Finally, I am not quite sure what this means

> there doesn't seem to be a convenient way to apply tags 100% reliably
> on message delivery, whereas I *can* control the folder 100% reliably.

Are you wanting to tag messages based on some properties when doing
notmuch new while avoiding races etc? Or are you worried about duplicate
message-ids? Or something else?

Best wishes

Mark




>
> David
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: folder and path completely broken in HEAD?

[Tomi Ollila]

On Sat, May 03 2014, dm-list-email-notmuch@scs.stanford.edu wrote:

> Jani Nikula <jani@nikula.org> writes:
>
>> On Fri, 02 May 2014, dm-list-email-notmuch@scs.stanford.edu wrote:
>>>
>>> I'm using a pretty standard maildir++ layout.  For example, underneath
>>> my database.path I have a bunch of mail in directories such as:
>>>
>>>     .INBOX.Main/{new,cur}
>>>     .mail.class/{new,cur}
>>>     .mail.voicemail/{new,cur}
>>> ...
>> Here's additional commentary on the specific queries.
>>
>>>         linux7$ ./notmuch count folder:mail
>>>         0
>>>         linux8$ ./notmuch count folder:.mail
>>>         0
>
> Oh, man.  That's a serious bummer.
>
> Is there any mechanism left that would let me hierarchically group
> messages?  I've got a ton of mail.* folders, and create new ones
> dynamically.  I really want a mechanism to group them hierarchically, so
> I can have a search that matches all current and future mail
> directories.  I organized my whole mail setup around folders because a)
> tags do not provide this kind of hierarchical control, and b) there
> doesn't seem to be a convenient way to apply tags 100% reliably on
> message delivery, whereas I *can* control the folder 100% reliably.
>
> Worse, because of my poor performance, I was hoping to segregate
> messages by year.  So it would be:
>
>   2013/.mail.class
>   2013/.mail.voicemail
>   2014/.mail.class
>   2014/.mail.voicemail
>
> All the way back.  Now you are saying there will be no convenient way to
> match just the "mail.class" part without the year?  How very
> distressing.  Ugh.

I use git-style segregation where mails are scattered into 256
subdirectories under a directory (based on md5hash of the mail file
contents). By default mails go under received/??/. results:

$  notmuch count folder:received
3
(I seem to have 3 test emails under the root of  ~/.mail/received)

$  notmuch count folder:received/**
0
(Apparently unsupported search ;D -- I have setopt no_nomatch in my
zsh so it doesn't barf when 'folder:received/**' doesn't match anything
and that saves me quoting effort :D)

$ notmuch count path:received/
0
$ notmuch count path:received
3
(again)

$ notmuch count path:received/**
19439

That's probably right. Inconvenient syntax though (No, I am not complaining)

>
> David

Tomi

Lets briefly verify this...


$ find  ~/mail/received -type f | wc
  21151   21151 1353561

Hmm

$ ~/vc/projects/toomuch/check-mid-duplicates.pl mail/received | wc
mail/received/00/9feb4f33523a003768a22b2f9df8f9: no Message-ID
mail/received/10/10ab4e570fdd8bf744dad93d2f1c7a: no Message-ID
mail/received/32/60aef86b3bf8837c1c6a8a20d5d519: no Message-ID
mail/received/zap: 'from' or 'date' header missing, not a mail file ?
mail/received/e1/f1be5aa84c8b28d1056aa9a8bc5b4f: no Message-ID
mail/received/19/4c0e3e812721da11ad71dfe26f5b05: no Message-ID
mail/received/41/d7083964e92c70ed57a3c8afd4d1d6: no Message-ID
mail/received/96/22de40b7c48d3ff32101777eade693: no Message-ID
mail/received/e7/8f5b883692e79627d53cf607228b02: no Message-ID
mail/received/d1/248c296b7ce635848c442b4c06a9f2: no Message-ID
   1711   17110  331583

Uh huh, have to check what is going on there....

Re: folder and path completely broken in HEAD?

[dm-list-email-notmuch]

Mark Walters <markwalters1009@gmail.com> writes:

>> All the way back.  Now you are saying there will be no convenient way to
>> match just the "mail.class" part without the year?  How very
>> distressing.  Ugh.
>
> Hi
>
> I am not quite sure what you are meaning by hierarchically group
> messages. Searching for path:dir/foo/bar/** should give all messages in
> all directories beneath dir/foo/bar. 

The problem is that the maildir++ spec disallows such pathnames.  If I
need compatibility with maildir++ (for instance for an imap server), at
least on a per-year basis, then my maildirs have to have names like:

   2013/.foo.foo
   2013/.foo.bar
   2013/.foo.baz
   2014/.foo.foo
   2014/.foo.bar
   2014/.foo.baz

So if I want a way to aggregate all my foo mail in a single search,
right now I just ask for folder:foo.  Will there be any equivalent in
the new notmuch?

Thanks,
David

Re: folder and path completely broken in HEAD?

[Jani Nikula]

On Sat, 03 May 2014, dm-list-email-notmuch@scs.stanford.edu wrote:
> Mark Walters <markwalters1009@gmail.com> writes:
>
>>> All the way back.  Now you are saying there will be no convenient way to
>>> match just the "mail.class" part without the year?  How very
>>> distressing.  Ugh.
>>
>> Hi
>>
>> I am not quite sure what you are meaning by hierarchically group
>> messages. Searching for path:dir/foo/bar/** should give all messages in
>> all directories beneath dir/foo/bar. 
>
> The problem is that the maildir++ spec disallows such pathnames.  If I
> need compatibility with maildir++ (for instance for an imap server), at
> least on a per-year basis, then my maildirs have to have names like:
>
>    2013/.foo.foo
>    2013/.foo.bar
>    2013/.foo.baz
>    2014/.foo.foo
>    2014/.foo.bar
>    2014/.foo.baz
>
> So if I want a way to aggregate all my foo mail in a single search,
> right now I just ask for folder:foo.  Will there be any equivalent in
> the new notmuch?

Given that we're in feature freeze and ready for an imminent release of
0.18, I think it's pretty safe to say, no.

---

It's not going to help you, but I'll mention a few of the issues the old
folder: search had, which we also had complaints about, and which would
have been quite hard to fix while preserving the behaviour you want. In
short, we considered the old folder: search broken.

Given layout:

	Foo/{cur,new}
	foo/{cur,new}
	fooing/{cur,new}
	bar/foo/{cur,new}
	cur
	new

It was impossible to refer to the top level folder.

It was impossible to refer to foo without also referring to Foo, fooing,
and bar/foo.

In your layout, if you also had 2013/.bar.foo, folder:foo would match
that as well. To not match that, you would have to include each
folder:.foo.xxx in the search.


BR,
Jani.

Re: folder and path completely broken in HEAD?

[Mark Walters]

Hi

On Sat, 03 May 2014, dm-list-email-notmuch@scs.stanford.edu wrote:
> Mark Walters <markwalters1009@gmail.com> writes:
>
>>> All the way back.  Now you are saying there will be no convenient way to
>>> match just the "mail.class" part without the year?  How very
>>> distressing.  Ugh.
>>
>> Hi
>>
>> I am not quite sure what you are meaning by hierarchically group
>> messages. Searching for path:dir/foo/bar/** should give all messages in
>> all directories beneath dir/foo/bar. 
>
> The problem is that the maildir++ spec disallows such pathnames.  If I
> need compatibility with maildir++ (for instance for an imap server), at
> least on a per-year basis, then my maildirs have to have names like:
>
>    2013/.foo.foo
>    2013/.foo.bar
>    2013/.foo.baz
>    2014/.foo.foo
>    2014/.foo.bar
>    2014/.foo.baz
>
> So if I want a way to aggregate all my foo mail in a single search,
> right now I just ask for folder:foo.  Will there be any equivalent in
> the new notmuch?

I agree with Jani that for 0.18 this won't be possible. However, there
is probably no urgent reason to upgrade (given you know it will cause
you some problems) so the question is can we add some functionality to
cover this use case. Given the way we do the folder matching I think it
unlikely we would want to add globbing/wildcards to the start of a path
(*).

However, we might be able to support maildir++ as its own prefix. If you
could so searches of the form maildir++:.foo.bar.** which would match
the maildir .foo.bar and all submaildirs++ (eg .foo.bar.a.b etc) would
that be sufficient?

This wouldn't deal with 2013/.foo.bar but would allow .foo.bar.2013

Would this be enough help to make it worth considering?

Best wishes

Mark

(*) if we get a custom parser then it may all be manageable.

Re: folder and path completely broken in HEAD?

[dm-list-email-notmuch]

Jani Nikula <jani@nikula.org> writes:

> It's not going to help you, but I'll mention a few of the issues the old
> folder: search had, which we also had complaints about, and which would
> have been quite hard to fix while preserving the behaviour you want. In
> short, we considered the old folder: search broken.
>
> Given layout:
>
> 	Foo/{cur,new}
> 	foo/{cur,new}
> 	fooing/{cur,new}
> 	bar/foo/{cur,new}
> 	cur
> 	new
>
> It was impossible to refer to the top level folder.
>
> It was impossible to refer to foo without also referring to Foo, fooing,
> and bar/foo.
>
> In your layout, if you also had 2013/.bar.foo, folder:foo would match
> that as well. To not match that, you would have to include each
> folder:.foo.xxx in the search.

First, thanks for the response.  The responsiveness and friendliness of
the notmuch mailing list goes a long way towards compensating for any
missing features / customizability one might want.

I was already aware of the issues you raise, and had worked around them
by just renaming all my mail folders.  I agree that searching for a
particular folder is crucial functionality, and found it weird that I
had to abandon my main top-level mailbox (which I just renamed
.INBOX.Main).

However, currently it seems strange that there are *two* different
search terms (folder and path), and that neither one lets you search for
a portion of your folder name.  Admittedly the old folder code was one
of the parts of the notmuch source that didn't make sense to me (and now
I'm starting to understand why--e.g., the fact that it used stemming,
for instance, was just weird and maybe accidental).

I may be able to hack around this problem in the emacs lisp part or with
a wrapper script.  I'm already having to defadvice around
notmuch-call-notmuch-sexp to implement features I'm missing from
wanderlust and gnus (e.g., the inability to specify regexps matching all
my email addresses).

But to help me understand the current design, can you answer a couple of
questions?


First, are there people out there who do not use a collection of maildir
directories, with all mail in cur and new?  If not, why does notmuch try
to find mail in non-mail-directories, and why do you need search terms
differentiating new and cur?  Conversely, I find it particularly weird
that there's no convenient way to say "stop trying to index stuff that
isn't in a maildir (cur or new)."  You can do the inverse and blacklist
files, but then I end up with stuff like this in my .notmuch-config:

        ignore=dovecot-keywords;dovecot-uidlist;dovecot-uidvalidity;dovecot.index;dovecot.index.cache;dovecot.index.log;dovecot.index.log.2;dovecot.index.search;dovecot.index.search.uids;maildirfolder;

(And still I think there are other junk files that get left around by my
imap server or other software, so notmuch new still repeatedly tries to
index junk whenever I run it.)


Second, does anyone out there have a collection with more than a few
thousand maildirs?  I understand some people index millions of messages,
but even with tiny maildirs of a few hundred messages each, that's a
mere 10,000 maildirs.  So for that, who needs any kind of Xapian
indexing fanciness or dedicated XFOLDER prefix?  Scanning the entire set
of XDIRECTORY terms to apply an arbitrary predicate should take
negligible time.  So all you really need is a boolean search term
containing the directory docid (or with the existing schema the ability
to do a prefix search on XFDIRENTRYn:*).

Thanks,
David

Re: folder and path completely broken in HEAD?

[Jani Nikula]

Hi Carl -

On Tue, 06 May 2014, Carl Worth <cworth@cworth.org> wrote:
> dm-list-email-notmuch@scs.stanford.edu writes:
>> However, currently it seems strange that there are *two* different
>> search terms (folder and path), and that neither one lets you search for
>> a portion of your folder name.
>
> For what it's worth, I totally agree. I'm guilty as far as sitting out
> of the detailed design discussions, (I don't use any sort of
> folder-based searching, so I don't care too much). I was aware of the
> problems of the original "folder:" code I wrote, and I was happy to hear
> that people were addressing those problems.
>
> But it's terribly strange to me that notmuch now has two different
> search terms that overlap so much in functionality. I know that I will
> never trust myself to be able to distinguish/describe the folder:
> vs. path: semantics without consulting the documentation. And that's
> discouraging.

The discussions about this were lengthy and tedious, and I was glad we
eventually reached some consensus about what we wanted. It's always
disappointing to find out one hasn't found the solution to satisfy
everyone, but in the end I think I'm happier we were able to reach a
decision, do something about the real issues people were facing with
folder: and move on, rather than just grind to a halt. I think we were
pretty close to everyone just dropping the ball and letting the folder:
prefix be, warts and all.

The idea of path: is that it's the exact filesystem directory, relative
from the maildir root, with an rsync like ** syntax for recursive
matching. Turns out people want folder: to hide maildir implementation
details like cur and new. These are not compatible, or you need to add a
syntax that's not easy or discoverable.

path: is now pretty much complete, and allows one to do robust scripting
that won't break in surprising ways. folder: is something we can still
add new functionality into, for example fancier interpretations of
maildir++, or anchoring if we ever get the custom query parser.

> I think the original "folder:" shortcomings could have been addressed
> without adding two terms, and also without losing some functionality,
> (as shown in David's use case).
>
> I would have liked to have seen some explicit syntax for anchoring the
> beginning and end of the directory name, (which could have then been
> re-used for anchoring subject: or even some future header: prefix).

As I understood it, that would have required writing a custom query
parser, a significant effort. At least nobody came up with a scheme to
do the anchoring without the parser while addressing the other issues
with the old folder: prefix.

> I've always thought that the "cur" and "new" directories were somewhere
> on the spectrum between pointless and annoying. The idea with the
> original "folder:" indexing was to implicitly ignore these, (when both
> existed).
>
> It seems that the new "folder:" continues this idea, while the new
> "path:" does not.

Correct.

> Meanwhile, the new "folder:" anchors the search to the beginning of
> the directory, while "path:" does not.

Incorrect (or I don't understand you).

> And finally, "path:" adds a magic syntax to do hierarchical matching
> while "folder:" does not.

Correct.

> That's an odd hodgepodge of non-orthogonal distinction in
> functionality.

I'm sorry to hear you think that way. One is verbatim filesystem path,
the other hides mail store folder implementation details as a
convenience.


BR,
Jani.

Re: folder and path completely broken in HEAD?

[Mark Walters]

On Mon, 05 May 2014, Jani Nikula <jani@nikula.org> wrote:
> Hi Carl -
>
> On Tue, 06 May 2014, Carl Worth <cworth@cworth.org> wrote:
>> dm-list-email-notmuch@scs.stanford.edu writes:
>>> However, currently it seems strange that there are *two* different
>>> search terms (folder and path), and that neither one lets you search for
>>> a portion of your folder name.
>>
>> For what it's worth, I totally agree. I'm guilty as far as sitting out
>> of the detailed design discussions, (I don't use any sort of
>> folder-based searching, so I don't care too much). I was aware of the
>> problems of the original "folder:" code I wrote, and I was happy to hear
>> that people were addressing those problems.
>>
>> But it's terribly strange to me that notmuch now has two different
>> search terms that overlap so much in functionality. I know that I will
>> never trust myself to be able to distinguish/describe the folder:
>> vs. path: semantics without consulting the documentation. And that's
>> discouraging.
>
> The discussions about this were lengthy and tedious, and I was glad we
> eventually reached some consensus about what we wanted. It's always
> disappointing to find out one hasn't found the solution to satisfy
> everyone, but in the end I think I'm happier we were able to reach a
> decision, do something about the real issues people were facing with
> folder: and move on, rather than just grind to a halt. I think we were
> pretty close to everyone just dropping the ball and letting the folder:
> prefix be, warts and all.

I would just like to second what Jani said. There was a lot of
discussion and, at the time, the outcome covered all use cases that
anyone showed any sign of wanting. And these included things like
distinguishing between messages in cur or new or the toplevel of a maildir,
wanting to search all subdirectories of a particular path (if eg the
main mail folder is split into 256 subsirectories 00..ff).

> The idea of path: is that it's the exact filesystem directory, relative
> from the maildir root, with an rsync like ** syntax for recursive
> matching. Turns out people want folder: to hide maildir implementation
> details like cur and new. These are not compatible, or you need to add a
> syntax that's not easy or discoverable.
>
> path: is now pretty much complete, and allows one to do robust scripting
> that won't break in surprising ways. folder: is something we can still
> add new functionality into, for example fancier interpretations of
> maildir++, or anchoring if we ever get the custom query parser.
>
>> I think the original "folder:" shortcomings could have been addressed
>> without adding two terms, and also without losing some functionality,
>> (as shown in David's use case).
>>
>> I would have liked to have seen some explicit syntax for anchoring the
>> beginning and end of the directory name, (which could have then been
>> re-used for anchoring subject: or even some future header: prefix).
>
> As I understood it, that would have required writing a custom query
> parser, a significant effort. At least nobody came up with a scheme to
> do the anchoring without the parser while addressing the other issues
> with the old folder: prefix.
>
>> I've always thought that the "cur" and "new" directories were somewhere
>> on the spectrum between pointless and annoying. The idea with the
>> original "folder:" indexing was to implicitly ignore these, (when both
>> existed).

I think many of us would agree, but there were users who wanted to
distinguish new and cur, and in at least one case, the toplevel as well.


>>
>> It seems that the new "folder:" continues this idea, while the new
>> "path:" does not.
>
> Correct.
>
>> Meanwhile, the new "folder:" anchors the search to the beginning of
>> the directory, while "path:" does not.
>
> Incorrect (or I don't understand you).
>
>> And finally, "path:" adds a magic syntax to do hierarchical matching
>> while "folder:" does not.
>
> Correct.
>
>> That's an odd hodgepodge of non-orthogonal distinction in
>> functionality.

>
> I'm sorry to hear you think that way. One is verbatim filesystem path,
> the other hides mail store folder implementation details as a
> convenience.

I think it is unfortunate that we need two variants, but I think they do
do different things. Also, I think any single user will find one matches
their setup and use that one essentially exclusively: if everything is
in maildirs then you probably want folder:, if you want exact control
then use path:.

Indeed, it may be that a third option of roughly a maildir++: search term
might solve David's use case.

Best wishes

Mark

Re: folder and path completely broken in HEAD?

[Mark Walters]

> The trick here is that it's easy to miss people who are happy with
> current functionality. Adding functionality to address newly-identified
> use cases makes a lot of sense. But removing functionality runs the risk
> of only discovering that people were relying on it after the fact,
> (Which seems to have happened here).

Yes indeed. I think one thing that I, at least, hadn't realised is that
people have lots of strange mail layouts often to work around problems
in mail agents (notmuch or others) or filesystem limitations etc.

>>> The idea of path: is that it's the exact filesystem directory, relative
>>> from the maildir root, with an rsync like ** syntax for recursive
>>> matching.
>
> This definition of "path:" seems good. It covers a lot of cases that the
> original "folder:" did not and allows precise, predictable control.
>
>>> Turns out people want folder: to hide maildir implementation
>>> details like cur and new.
>
> Which is something that "folder:" always did.
>
>>> These are not compatible, or you need to add a syntax that's not easy
>>> or discoverable.
>
> OK. So I won't argue that we don't need two different syntaxes here. But
> I will continue to discuss a use case that was addressed before and is
> no longer.
>
>> I think many of us would agree, but there were users who wanted to
>> distinguish new and cur, and in at least one case, the toplevel as
>> well.
>
> So now, "path:" allows for that, right?
>
> My concern is not so much that "path:" was added to address new things,
> but more than "folder:" was modified in a way that dropped useful
> functionality, (beyond just fixing bugs in "folder:" such as the
> accidental support of stemming).

One possibly perverse remark: it wouldn't surprise me if someone
actually used the stemming. I have used folders called old older and
oldest and that probably can be excluded by stemming.....

All I am really saying is that any change we made was going to break
some people's setups.

>> I think it is unfortunate that we need two variants, but I think they do
>> do different things.
>
> I'll accept that.
>
> But, while I have heard a good definition of "path:", (see above), I
> haven't heard a good one for "folder:" yet. Can you provide that?

I think folder:foo/bar means "In the maildir exactly foo/bar"

>> Also, I think any single user will find one matches their setup and
>> use that one essentially exclusively:
>
> The current discussion is evidence against that. We have a user of
> folder: that can no longer get at the desired functionality, (that used
> to exist).

Sorry I wasn't trying to suggest that the current setup does everything
people want (clearly not!): just answering your question about the
differences being confusing. since each user will probably only use one
of them they probably become accustomed to its use.

Indeed, I think of the choice as being analogous to allowing the user to
choose which path scheme they like (so sort of like a customisation):
maildir folder based or filesystem path based.

>> Indeed, it may be that a third option of roughly a maildir++: search term
>> might solve David's use case.
>
> Or just making "folder:" index each term of a filesytem path like it
> used to do. And if that doesn't give sufficient control to some users,
> then "path:" is available.

We could do this. It might break things for user who are using the new
syntax....  Maybe we could make an initial / match the root of the
maildir store. But I think some people will dislike any of the options.

>
> I've already lost what I would have preferred, (a single syntax for all
> use cases---which was lost not to a design problem, but simply the
> implementation difficulty of requiring a custom query parser). I really
> would not like to see things continue down to have a *third* syntax.

So this third syntax would fit with my view of it being a customisation
like thing.

Best wishes

Mark

Re: folder and path completely broken in HEAD?

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

On Sat, May 03 2014, dm-list-email-notmuch@scs.stanford.edu wrote:
> First, are there people out there who do not use a collection of maildir
> directories, with all mail in cur and new?

o/ I completely abandoned the usage of separate mail "folders" since I
started using notmuch.  All my mail now just goes into a single maildir.
I consider this one of notmuch's awesome features.  It's honestly been a
bit perplexing to me that folks have put so much work into
parsing/indexing paths and folders given that it seems to me that
notmuch has utterly obviated the need to worry about such things (thank
god!).

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 818 bytes --]