bug#56442: gnus-search-run-search: Hits notmuch command line length limits

bug#56442: gnus-search-run-search: Hits notmuch command line length limits

[Sean Whitton]

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

Hello,

I'm running a pretty innocent notmuch query over a fairly small Maildir:

"((List:debian-devel.lists.debian.org) or ... or
(List:debian-haskell.lists.debian.org) or
(List:debconf-discuss.lists.debian.org)) and (not path:annex/**)"

but gnus-search-run-search fails to return any results.  The reason is
that Gnus first runs the query with --output=threads to obtain a list of
thread ids, and then runs another query with --output=files and a query
constructed from the output of the first query: "thread:000000000000d9d0
or thread:000000000000d9e0 or thread:000000000000d268 or ..."

The resulting command fails completely:

    emacs: /usr/bin/notmuch: Argument list too long

Instead of running two searches like this, we can just surround the
whole query like this: "thread:{QUERY}".  The manual says it's exactly
equivalent:

    ... the user should think of the query thread:{<something>} as
    expanding to all of the thread IDs which match <something>; not‐
    much then performs a second search using the expanded query.

This should be faster, too, with only running a single external command.
Here is the patch I'm thinking I'll apply, if anyone has comments.

-- 
Sean Whitton

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Use-notmuch-thread-operator-instead-of-performing-tw.patch --]
[-- Type: text/x-diff, Size: 3370 bytes --]

From d62398cc27abd317d31d8f07e151a82081444217 Mon Sep 17 00:00:00 2001
From: Sean Whitton <spwhitton@spwhitton.name>
Date: Thu, 7 Jul 2022 09:34:04 -0700
Subject: [PATCH] Use notmuch thread:{} operator instead of performing two
 searches

This also avoids some command line length limit problems.

* gnus-search.el
(gnus-search-run-search (engine gnus-search-notmuch)): Delete method.
(gnus-search-indexed-search-command (engine gnus-search-notmuch)):
When searching for threads, wrap whole query in thread:{} operator.
Always use --output=files, never --output=threads.
---
 lisp/gnus/gnus-search.el | 46 +++++-----------------------------------
 1 file changed, 5 insertions(+), 41 deletions(-)

diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el
index 369df81d9b..53b6d1b4c6 100644
--- a/lisp/gnus/gnus-search.el
+++ b/lisp/gnus/gnus-search.el
@@ -1672,43 +1672,6 @@ gnus-search-transform-expression
       (format "date:%s.." (notmuch-date (cdr expr))))
      (t (ignore-errors (cl-call-next-method))))))
 
-(cl-defmethod gnus-search-run-search :around ((engine gnus-search-notmuch)
-					      server query groups)
-  "Handle notmuch's thread-search routine."
-  ;; Notmuch allows for searching threads, but only using its own
-  ;; thread ids.  That means a thread search is a \"double-bounce\":
-  ;; once to find the relevant thread ids, and again to find the
-  ;; actual messages.  This method performs the first \"bounce\".
-  (if (alist-get 'thread query)
-      (with-slots (program proc-buffer) engine
-	(let* ((qstring
-		(gnus-search-make-query-string engine query))
-	       (cp-list (gnus-search-indexed-search-command
-			 engine qstring query groups))
-	       thread-ids proc)
-	  (with-current-buffer proc-buffer
-	    (erase-buffer)
-	    (setq proc (apply #'start-process (format "search-%s" server)
-			      proc-buffer program cp-list))
-	    (while (process-live-p proc)
-	      (accept-process-output proc))
-            (goto-char (point-min))
-	    (while (re-search-forward
-                    "^thread:\\([^[:space:]\n]+\\)"
-                    (point-max) t)
-	      (cl-pushnew (match-string 1) thread-ids :test #'equal)))
-	  (cl-call-next-method
-	   engine server
-	   ;; If we found threads, completely replace the query with
-	   ;; our new thread-based one.
-           (if thread-ids
-               `((query . ,(mapconcat (lambda (thrd)
-                                        (concat "thread:" thrd))
-                                      thread-ids " or ")))
-             query)
-	   nil)))
-    (cl-call-next-method engine server query groups)))
-
 (cl-defmethod gnus-search-indexed-search-command ((engine gnus-search-notmuch)
 						  (qstring string)
 						  query &optional _groups)
@@ -1721,13 +1684,14 @@ gnus-search-indexed-search-command
       (append
        (list (format "--config=%s" config-file)
              "search"
-             (if thread
-                 "--output=threads"
-             "--output=files"))
+             "--output=files")
        (unless thread '("--duplicate=1"))
        (when limit (list (format "--limit=%d" limit)))
        switches
-       (list qstring)))))
+       (list (if thread
+                 (format "thread:\"{%s}\""
+                         (string-replace "\"" "\"\"" qstring))
+               qstring))))))
 
 ;;; Mairix interface
 
-- 
2.30.2


[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: bug#56442: gnus-search-run-search: Hits notmuch command line length limits

[Eric Abrahamsen]

Sean Whitton <spwhitton@spwhitton.name> writes:

> Hello,
>
> I'm running a pretty innocent notmuch query over a fairly small Maildir:
>
> "((List:debian-devel.lists.debian.org) or ... or
> (List:debian-haskell.lists.debian.org) or
> (List:debconf-discuss.lists.debian.org)) and (not path:annex/**)"
>
> but gnus-search-run-search fails to return any results.  The reason is
> that Gnus first runs the query with --output=threads to obtain a list of
> thread ids, and then runs another query with --output=files and a query
> constructed from the output of the first query: "thread:000000000000d9d0
> or thread:000000000000d9e0 or thread:000000000000d268 or ..."
>
> The resulting command fails completely:
>
>     emacs: /usr/bin/notmuch: Argument list too long
>
> Instead of running two searches like this, we can just surround the
> whole query like this: "thread:{QUERY}".  The manual says it's exactly
> equivalent:
>
>     ... the user should think of the query thread:{<something>} as
>     expanding to all of the thread IDs which match <something>; not‐
>     much then performs a second search using the expanded query.
>
> This should be faster, too, with only running a single external command.
> Here is the patch I'm thinking I'll apply, if anyone has comments.

Huh, I tried this a couple months ago with a more complicated query,
like:

thread:{from:bob or from:jane}

and I remember notmuch barking at me about spaces or the "or" or
something -- anyway I got the impression that it couldn't accept
multi-part queries inside the "thread:{}" syntax. But looking at your
patch maybe I just needed to quote differently?

Anyway I'll give this a test this weekend. It would be *very* nice if we
could use this syntax rather than the ugly home-grown one.

Thanks for the report!

Eric\r

Re: bug#56442: gnus-search-run-search: Hits notmuch command line length limits

[Sean Whitton]

Hello,

On Sat 09 Jul 2022 at 09:11AM -07, Eric Abrahamsen wrote:

> But looking at your patch maybe I just needed to quote differently?

Right, notmuch has this unusual " -> "" quoting thing.

> Anyway I'll give this a test this weekend. It would be *very* nice if
> we could use this syntax rather than the ugly home-grown one.

Let me know the results of your test, and if it's also continuing to
work for me by then, I'll install the change.  Thanks for taking a look.

-- 
Sean Whitton