From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 6A9BE431FB6 for ; Mon, 5 Aug 2013 12:00:03 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.7 X-Spam-Level: X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zt7tlyITOaMD for ; Mon, 5 Aug 2013 11:59:57 -0700 (PDT) Received: from dmz-mailsec-scanner-5.mit.edu (dmz-mailsec-scanner-5.mit.edu [18.7.68.34]) by olra.theworths.org (Postfix) with ESMTP id EF31C431FAE for ; Mon, 5 Aug 2013 11:59:56 -0700 (PDT) X-AuditID: 12074422-b7ef78e000000935-5a-51fff62a96fd Received: from mailhub-auth-4.mit.edu ( [18.7.62.39]) by dmz-mailsec-scanner-5.mit.edu (Symantec Messaging Gateway) with SMTP id 16.8C.02357.A26FFF15; Mon, 5 Aug 2013 14:59:54 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id r75IxrMW006141; Mon, 5 Aug 2013 14:59:54 -0400 Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) (authenticated bits=0) (User authenticated as amdragon@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id r75Ixosb022754 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Mon, 5 Aug 2013 14:59:52 -0400 Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80) (envelope-from ) id 1V6Q0S-0003q6-Sa; Mon, 05 Aug 2013 14:59:49 -0400 Date: Mon, 5 Aug 2013 14:59:47 -0400 From: Austin Clements To: Ramakrishnan Muthukrishnan Subject: Re: unread message appear `folded' Message-ID: <20130805185947.GD7794@mit.edu> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpkleLIzCtJLcpLzFFi42IRYrdT19X69j/QYHqrhcX1mzOZLdZ+TXJg 8ni26hazx6I96QFMUVw2Kak5mWWpRfp2CVwZTU1TmQoeSlW8nLWDqYGxXbSLkZNDQsBE4tmE G4wQtpjEhXvr2UBsIYF9jBJfzvt2MXIB2RsYJV7f388G4Zxikrja8JYFwlnCKPF/8z0gh4OD RUBF4m+XMEg3m4CGxLb9y8GmiggYSVxrucIKYjMLSEt8+93MBGILC2hLXGpeAFbDC2SfW3Gc CWLmckaJeRtWMEMkBCVOznzCAtGsJXHj30smkF0gg5b/4wAJcwpESkzf9x2sXBTohCknt7FN YBSahaR7FpLuWQjdCxiZVzHKpuRW6eYmZuYUpybrFicn5uWlFuma6uVmluilppRuYgSHtIvS DsafB5UOMQpwMCrx8CZc/R8oxJpYVlyZe4hRkoNJSZR3/xegEF9SfkplRmJxRnxRaU5q8SFG CQ5mJRHe+VuAcrwpiZVVqUX5MClpDhYlcd5nT88GCgmkJ5akZqemFqQWwWRlODiUJHh5vwI1 ChalpqdWpGXmlCCkmTg4QYbzAA1nBanhLS5IzC3OTIfIn2JUlBLn/QtykQBIIqM0D64XlnJe MYoDvSLM+xmkigeYruC6XwENZgIabPLzL8jgkkSElFQDo++CUMGY029+bWZPYb44M+XaqxRb 9V0FzXu+n8p0/pTtxd5RujtmabnyhX8HWe5u/z4t80DGv48VczT4le/1cnTs0/5+ninAhOVv ePWP8P3KfX8vqBmU7/qx8pxqt/lExdwcnSB17jt3Nv12jfdPDtIomrVw6nk982Vn5h+fub9O 7tUGvouqS5VYijMSDbWYi4oTAbViH3IUAwAA Cc: notmuch@notmuchmail.org X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Aug 2013 19:00:03 -0000 Quoth Ramakrishnan Muthukrishnan on Aug 05 at 11:35 pm: > Ramakrishnan Muthukrishnan writes: > > > Hi, > > > > Pardon me for using non-standard terms here because of lack of knowledge > > of the language used in the email world. > > > > I have this thread in which some messages are tagged unread. When the > > thread is opened, I see that those unread messages are not shown in full > > in the threaded view. Instead they appear `folded' with only the summary > > appearing in the thread, as if they are read. > > > > Here is a screenshot: http://i.imgur.com/Eh8SKe6.png > > > > I am attaching the messages in the thread shown in the screenshot as a > > tar.bz2 file. > > I just forgot to mention that I see this only when I do a > notmuch-search with `*' and not when I search with a tag. '*' turns out to be the key. When showing a thread, notmuch-show-build-buffer constructs a new query from the thread ID and the original search query of the form thread:X and (original query) If the original query is '*', this is thread:X and (*) But * isn't part of the Xapian query syntax. Notmuch specially handles queries that exactly match "*" before passing the query to Xapian. When * is embedded in a larger query, this special casing doesn't apply. In fact, Xapian parses this query as (Tmail AND (and:(pos=1) FILTER GX)) The "and" in the query turned into a plain search term (the "AND" is a proper boolean operator, but is unrelated to the provided query). This is a symptom of a general problem where we assume queries are textually composable, when they are not. We have the same problem at least in notmuch-search-filter and in notmuch tag query optimization. In this particular case, the * causes Xapian's query grammar to fail to parse the query, which Xapian handles by re-parsing the entire query will all query features disabled (which includes disabling support for boolean operators). Unfortunately, just handling * better isn't really a solution because it's just one of many things that violates query composability. Some solutions I can see are: 1) Switch to a composable query syntax (which would include *). This, obviously, requires a custom query parser, but is the most localized change I can think of and keeps queries as strings. 2) Never construct queries by pasting strings together. This would require changes to both the libnotmuch and CLI interfaces and queries could no longer be strings, but in the words of Alan Perlis, the string is a stark data structure. (In the case of show, I would actually love it if we could specify separate search and match queries because that would eliminate --entire-thread as well as the fallback in notmuch-show-build-buffer when search returns nothing, but I digress.) 3) Keep queries as plain strings, but switch to some hybrid syntax that lets us combine Xapian queries with composable operators parsed by notmuch. When we need to combine queries, do it using the composable operators. This actually may not be a bad way to transition to a full custom query parser; I think it would be relatively easy to take over parsing Xapian's boolean syntax, but leave the "prob" parsing to Xapian. I lean towards 3 because it seems like the least disruptive and offers a smooth transition to 1.