Re: unread message appear `folded'

[David Bremner <david@tethera.net>]

Austin Clements <amdragon@MIT.EDU> writes:

> Quoth Ramakrishnan Muthukrishnan on Aug 05 at 11:35 pm:
>> Ramakrishnan Muthukrishnan <ram@rkrishnan.org> writes:
>> 
>> > Hi,
>> >
>> > Pardon me for using non-standard terms here because of lack of knowledge
>> > of the language used in the email world.
>> >
>> > I have this thread in which some messages are tagged unread. When the
>> > thread is opened, I see that those unread messages are not shown in full
>> > in the threaded view. Instead they appear `folded' with only the summary
>> > appearing in the thread, as if they are read. 
>> >
>> > Here is a screenshot: http://i.imgur.com/Eh8SKe6.png
>> >
>> > I am attaching the messages in the thread shown in the screenshot as a
>> > tar.bz2 file.
>> 
>> I just forgot to mention that I see this only when I do a
>> notmuch-search with `*' and not when I search with a tag. 
>
> '*' turns out to be the key.  When showing a thread,
> notmuch-show-build-buffer constructs a new query from the thread ID
> and the original search query of the form
>
>   thread:X and (original query)
>
> If the original query is '*', this is
>
>   thread:X and (*)
>
> But * isn't part of the Xapian query syntax.  Notmuch specially
> handles queries that exactly match "*" before passing the query to
> Xapian.  When * is embedded in a larger query, this special casing
> doesn't apply.  In fact, Xapian parses this query as
>
>   (Tmail AND (and:(pos=1) FILTER GX))
>
> The "and" in the query turned into a plain search term (the "AND" is a
> proper boolean operator, but is unrelated to the provided query).
>
> This is a symptom of a general problem where we assume queries are
> textually composable, when they are not.  We have the same problem at
> least in notmuch-search-filter and in notmuch tag query optimization.
> In this particular case, the * causes Xapian's query grammar to fail
> to parse the query, which Xapian handles by re-parsing the entire
> query will all query features disabled (which includes disabling
> support for boolean operators).  Unfortunately, just handling * better
> isn't really a solution because it's just one of many things that
> violates query composability.
>
> Some solutions I can see are:
>
> 1) Switch to a composable query syntax (which would include *).  This,
>    obviously, requires a custom query parser, but is the most
>    localized change I can think of and keeps queries as strings.
>
> 2) Never construct queries by pasting strings together.  This would
>    require changes to both the libnotmuch and CLI interfaces and
>    queries could no longer be strings, but in the words of Alan
>    Perlis, the string is a stark data structure.  (In the case of
>    show, I would actually love it if we could specify separate search
>    and match queries because that would eliminate --entire-thread as
>    well as the fallback in notmuch-show-build-buffer when search
>    returns nothing, but I digress.)
>
> 3) Keep queries as plain strings, but switch to some hybrid syntax
>    that lets us combine Xapian queries with composable operators
>    parsed by notmuch.  When we need to combine queries, do it using
>    the composable operators.  This actually may not be a bad way to
>    transition to a full custom query parser; I think it would be
>    relatively easy to take over parsing Xapian's boolean syntax, but
>    leave the "prob" parsing to Xapian.
>
> I lean towards 3 because it seems like the least disruptive and offers
> a smooth transition to 1.

Update on this very old bug. As of notmuch 0.34 the notmuch CLI supports
both 1 and 3 via --query=sexp, if compiled with the (currently optional)
sfsexp library. My plan is that once sfsexp is sufficiently widely
available (whatever that turns out to mean) to make it a mandatory
dependency. At that point we can transition the emacs front end to using
notmuch-sexp-queries(7) to compose queries.

In principle we could start that work now by checking "notmuch config
get built_with.sexp_queries". I'm not sure if it's worth the extra
complexity, but it would make the transition smoother, and probably help
test the s-expression query parser.

d