bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[jao <jao@gnu.org>]

On Wed, Nov 11 2020, Eric Abrahamsen wrote:

[...]

> This code that comes straight from the old nnir.el, and has always
> looked a bit fragile to me. The problem is that it really just expects
> each message to be in a regular file, with groups as folders.
>
> So you're using notmuch as a search engine for a local leafnode nntp
> server, and indexing its message store directly?

Yes.

> Is there any leafnode setting that could influence how it stores its
> messages? Can it be convinced to store them in hierarchical folders?

It is exactly what it is doing.  Leafnode messages in the nntp group
gmane.emacs.bugs are stored in /prefix/gmane/emac/bugs/<article no.>
exactly as nnmail would do.  And the result of the search by notmuch is
displaying those paths correctly.  The problem is that, in the code i
mentioned, when that output is processed, the code is expecting to see
in the results list gmane.emacs.bugs instead of pathnames.  So i don't
understand how that code has ever worked, even on the old nnir.

> I suppose I could change the group-regexp to munge periods, but that
> could cause breakage in other cases, and I would be hesitant to do that.

In may case, the problem is the other way around: the regexp should
accept /, but is only accepting periods (because it's constructed
directly from the group name, which, for nntp at least, contains
periods, not slashes).

> Otherwise, all the indexed search engines have a
> `gnus-search-indexed-extract' method that's used to actually return the
> file name from the results buffer.

See above.  It's really very easy for me to fix this with an around
method like this one:

     (cl-defmethod gnus-search-indexed-parse-output :around ((e gnus-search-notmuch) s q groups)
        (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g))
                          groups)))
          (cl-call-next-method e s q gs)))

because the problem in my case is the group name, not the search
results. Note that, for cases (if any) where the group names already
look like "gmane/emacs/bug" (which i suspect is what nnml and nnmaildir
are doing and that's why it works), that's a no-op.

So yes, for me it's a solved problem, even without defining a new
engine.

>> On other news, i was trying to find a way in Gnus to go from Message-ID
>> to article no. for IMAP groups or nnmaildirs (which would make using
>> notmuch with dovecot really trivial), but without luck: anyone knows of
>> an easy way?
>
> Come to think of it, nnimap can already accept article numbers
> as message-ids. So if notmuch returns its results as message-ids, it
> should work transparently.

Yes, i thought that at first.  i got stuck when i discovered that
nnselect apparently only accepts article numbers to build its groups.  i
guess one could extend nnselect to also accept message ids, and that
that would be useful also in other contexts.

> The problem is that while the mechanism is there, it works by searching
> each message-id and getting the proper article number that way. My guess
> is that you'd be negating most of the speed advantage of using notmuch
> like this, and you'd still be better off using dovecot's own full-text
> indexing. You don't have to use xapian!
>
> https://doc.dovecot.org/configuration_manual/fts/

Ah, thanks for that, i'll definitely take a look.  The nice thing about
using notmuch would be that is one less moving piece to maintain, and
that then one could move very, very transparently between notmuch's
emacs interface and gnus, using the same message files and indexes.

> BUT, if you really wanted to do this, it would be relatively easy to
> check for "--output=messages" in 'switches, and use that instead of
> "--output=files".

Yes, it's trivial to get a vector of results of the form 
[group message-id score] ... a full engine like that is just a few
lines:

    (require 'notmuch-query)

    (defclass jao-gnus-notmuch (gnus-search-notmuch) ())

    (cl-defmethod gnus-search-run-search ((engine jao-gnus-notmuch) srv query groups)
      (let ((qstring (gnus-search-make-query-string engine query))
            (res))
        (dolist (group groups)
          (let* ((folder (format "folder:%s"
                                 (replace-regexp-in-string "\\." "/" group)))
                 (ids (notmuch-query-get-message-ids qstring folder)))
            (dolist (id ids)
              (push (vector (gnus-group-full-name group srv) id 100) res))))
        res))

That's all (well, obviating thread seaches, but those are not hard
either).  Except that it doesn't work because nnselect wants 
[group article-no score].  If you tell me how to obtain article-no from
message-id, i'm done :)

Thanks,
jao
-- 
I don't necessarily agree with everything I say.
 -Marshall McLuhan (1911-1980)