unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* bug#56442: gnus-search-run-search: Hits notmuch command line length limits
@ 2022-07-07 16:56 Sean Whitton
  0 siblings, 0 replies; 4+ messages in thread
From: Sean Whitton @ 2022-07-07 16:56 UTC (permalink / raw)
  To: 56442; +Cc: notmuch, Eric Abrahamsen

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

Hello,

I'm running a pretty innocent notmuch query over a fairly small Maildir:

"((List:debian-devel.lists.debian.org) or ... or
(List:debian-haskell.lists.debian.org) or
(List:debconf-discuss.lists.debian.org)) and (not path:annex/**)"

but gnus-search-run-search fails to return any results.  The reason is
that Gnus first runs the query with --output=threads to obtain a list of
thread ids, and then runs another query with --output=files and a query
constructed from the output of the first query: "thread:000000000000d9d0
or thread:000000000000d9e0 or thread:000000000000d268 or ..."

The resulting command fails completely:

    emacs: /usr/bin/notmuch: Argument list too long

Instead of running two searches like this, we can just surround the
whole query like this: "thread:{QUERY}".  The manual says it's exactly
equivalent:

    ... the user should think of the query thread:{<something>} as
    expanding to all of the thread IDs which match <something>; not‐
    much then performs a second search using the expanded query.

This should be faster, too, with only running a single external command.
Here is the patch I'm thinking I'll apply, if anyone has comments.

-- 
Sean Whitton

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Use-notmuch-thread-operator-instead-of-performing-tw.patch --]
[-- Type: text/x-diff, Size: 3370 bytes --]

From d62398cc27abd317d31d8f07e151a82081444217 Mon Sep 17 00:00:00 2001
From: Sean Whitton <spwhitton@spwhitton.name>
Date: Thu, 7 Jul 2022 09:34:04 -0700
Subject: [PATCH] Use notmuch thread:{} operator instead of performing two
 searches

This also avoids some command line length limit problems.

* gnus-search.el
(gnus-search-run-search (engine gnus-search-notmuch)): Delete method.
(gnus-search-indexed-search-command (engine gnus-search-notmuch)):
When searching for threads, wrap whole query in thread:{} operator.
Always use --output=files, never --output=threads.
---
 lisp/gnus/gnus-search.el | 46 +++++-----------------------------------
 1 file changed, 5 insertions(+), 41 deletions(-)

diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el
index 369df81d9b..53b6d1b4c6 100644
--- a/lisp/gnus/gnus-search.el
+++ b/lisp/gnus/gnus-search.el
@@ -1672,43 +1672,6 @@ gnus-search-transform-expression
       (format "date:%s.." (notmuch-date (cdr expr))))
      (t (ignore-errors (cl-call-next-method))))))
 
-(cl-defmethod gnus-search-run-search :around ((engine gnus-search-notmuch)
-					      server query groups)
-  "Handle notmuch's thread-search routine."
-  ;; Notmuch allows for searching threads, but only using its own
-  ;; thread ids.  That means a thread search is a \"double-bounce\":
-  ;; once to find the relevant thread ids, and again to find the
-  ;; actual messages.  This method performs the first \"bounce\".
-  (if (alist-get 'thread query)
-      (with-slots (program proc-buffer) engine
-	(let* ((qstring
-		(gnus-search-make-query-string engine query))
-	       (cp-list (gnus-search-indexed-search-command
-			 engine qstring query groups))
-	       thread-ids proc)
-	  (with-current-buffer proc-buffer
-	    (erase-buffer)
-	    (setq proc (apply #'start-process (format "search-%s" server)
-			      proc-buffer program cp-list))
-	    (while (process-live-p proc)
-	      (accept-process-output proc))
-            (goto-char (point-min))
-	    (while (re-search-forward
-                    "^thread:\\([^[:space:]\n]+\\)"
-                    (point-max) t)
-	      (cl-pushnew (match-string 1) thread-ids :test #'equal)))
-	  (cl-call-next-method
-	   engine server
-	   ;; If we found threads, completely replace the query with
-	   ;; our new thread-based one.
-           (if thread-ids
-               `((query . ,(mapconcat (lambda (thrd)
-                                        (concat "thread:" thrd))
-                                      thread-ids " or ")))
-             query)
-	   nil)))
-    (cl-call-next-method engine server query groups)))
-
 (cl-defmethod gnus-search-indexed-search-command ((engine gnus-search-notmuch)
 						  (qstring string)
 						  query &optional _groups)
@@ -1721,13 +1684,14 @@ gnus-search-indexed-search-command
       (append
        (list (format "--config=%s" config-file)
              "search"
-             (if thread
-                 "--output=threads"
-             "--output=files"))
+             "--output=files")
        (unless thread '("--duplicate=1"))
        (when limit (list (format "--limit=%d" limit)))
        switches
-       (list qstring)))))
+       (list (if thread
+                 (format "thread:\"{%s}\""
+                         (string-replace "\"" "\"\"" qstring))
+               qstring))))))
 
 ;;; Mairix interface
 
-- 
2.30.2


[-- Attachment #3: Type: text/plain, Size: 0 bytes --]



^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: bug#56442: gnus-search-run-search: Hits notmuch command line length limits
       [not found] <875yk850be.fsf__40907.3059143097$1657213041$gmane$org@athena.silentflame.com>
@ 2022-07-09 16:11 ` Eric Abrahamsen
  2022-07-09 18:01   ` Sean Whitton
  2022-07-09 19:19   ` quoting: (was: Re: bug#56442: gnus-search-run-search: Hits notmuch command line length limits) Tomi Ollila
  0 siblings, 2 replies; 4+ messages in thread
From: Eric Abrahamsen @ 2022-07-09 16:11 UTC (permalink / raw)
  To: Sean Whitton; +Cc: 56442, notmuch

Sean Whitton <spwhitton@spwhitton.name> writes:

> Hello,
>
> I'm running a pretty innocent notmuch query over a fairly small Maildir:
>
> "((List:debian-devel.lists.debian.org) or ... or
> (List:debian-haskell.lists.debian.org) or
> (List:debconf-discuss.lists.debian.org)) and (not path:annex/**)"
>
> but gnus-search-run-search fails to return any results.  The reason is
> that Gnus first runs the query with --output=threads to obtain a list of
> thread ids, and then runs another query with --output=files and a query
> constructed from the output of the first query: "thread:000000000000d9d0
> or thread:000000000000d9e0 or thread:000000000000d268 or ..."
>
> The resulting command fails completely:
>
>     emacs: /usr/bin/notmuch: Argument list too long
>
> Instead of running two searches like this, we can just surround the
> whole query like this: "thread:{QUERY}".  The manual says it's exactly
> equivalent:
>
>     ... the user should think of the query thread:{<something>} as
>     expanding to all of the thread IDs which match <something>; not‐
>     much then performs a second search using the expanded query.
>
> This should be faster, too, with only running a single external command.
> Here is the patch I'm thinking I'll apply, if anyone has comments.

Huh, I tried this a couple months ago with a more complicated query,
like:

thread:{from:bob or from:jane}

and I remember notmuch barking at me about spaces or the "or" or
something -- anyway I got the impression that it couldn't accept
multi-part queries inside the "thread:{}" syntax. But looking at your
patch maybe I just needed to quote differently?

Anyway I'll give this a test this weekend. It would be *very* nice if we
could use this syntax rather than the ugly home-grown one.

Thanks for the report!

Eric\r

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: bug#56442: gnus-search-run-search: Hits notmuch command line length limits
  2022-07-09 16:11 ` bug#56442: gnus-search-run-search: Hits notmuch command line length limits Eric Abrahamsen
@ 2022-07-09 18:01   ` Sean Whitton
  2022-07-09 19:19   ` quoting: (was: Re: bug#56442: gnus-search-run-search: Hits notmuch command line length limits) Tomi Ollila
  1 sibling, 0 replies; 4+ messages in thread
From: Sean Whitton @ 2022-07-09 18:01 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: 56442, notmuch

Hello,

On Sat 09 Jul 2022 at 09:11AM -07, Eric Abrahamsen wrote:

> But looking at your patch maybe I just needed to quote differently?

Right, notmuch has this unusual " -> "" quoting thing.

> Anyway I'll give this a test this weekend. It would be *very* nice if
> we could use this syntax rather than the ugly home-grown one.

Let me know the results of your test, and if it's also continuing to
work for me by then, I'll install the change.  Thanks for taking a look.

-- 
Sean Whitton

^ permalink raw reply	[flat|nested] 4+ messages in thread

* quoting: (was: Re: bug#56442: gnus-search-run-search: Hits notmuch command line length limits)
  2022-07-09 16:11 ` bug#56442: gnus-search-run-search: Hits notmuch command line length limits Eric Abrahamsen
  2022-07-09 18:01   ` Sean Whitton
@ 2022-07-09 19:19   ` Tomi Ollila
  1 sibling, 0 replies; 4+ messages in thread
From: Tomi Ollila @ 2022-07-09 19:19 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: notmuch

On Sat, Jul 09 2022, Eric Abrahamsen wrote:

>
> Huh, I tried this a couple months ago with a more complicated query,
> like:
>
> thread:{from:bob or from:jane}
>
> and I remember notmuch barking at me about spaces or the "or" or
> something -- anyway I got the impression that it couldn't accept
> multi-part queries inside the "thread:{}" syntax. But looking at your
> patch maybe I just needed to quote differently?

I am not sure whether this is relevant, but I give a shot...

I tried the following queries from command line:

$ notmuch search 'thread:"{from:bob or from:jane}"'

thread:0000000000000b91   2009-01-20 [6/6] ...
thread:00000000000015fa   2008-11-26 [2/2] ...

and

$ notmuch search 'from:bob or from:jane}':

thread:0000000000000b91   2009-01-20 [1/6] ...
thread:00000000000015fa   2008-11-26 [1/2] ...


The outer '':s protect from shell expansion so what is
inside those goes verbatim to notmuch and from there to
xapian query parser (IIRC). 

Former includes all messages in thread where at least
one message matched to the query and latter just those
messages that match the query.

Like Sean mentioned in another message, Xapian converts ""
to " in query string (probably something irrelevant in this
discussion) the thread:"{...}" syntax binds the whole expression
to this thread: prefix (Why {...} is not enough is beyond me ;D)

Tomi

>
> Anyway I'll give this a test this weekend. It would be *very* nice if we
> could use this syntax rather than the ugly home-grown one.
>
> Thanks for the report!
>
> Eric

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2022-07-09 19:19 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <875yk850be.fsf__40907.3059143097$1657213041$gmane$org@athena.silentflame.com>
2022-07-09 16:11 ` bug#56442: gnus-search-run-search: Hits notmuch command line length limits Eric Abrahamsen
2022-07-09 18:01   ` Sean Whitton
2022-07-09 19:19   ` quoting: (was: Re: bug#56442: gnus-search-run-search: Hits notmuch command line length limits) Tomi Ollila
2022-07-07 16:56 bug#56442: gnus-search-run-search: Hits notmuch command line length limits Sean Whitton

Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).