* bug#56442: gnus-search-run-search: Hits notmuch command line length limits
@ 2022-07-07 16:56 Sean Whitton
2022-07-09 16:11 ` Eric Abrahamsen
0 siblings, 1 reply; 5+ messages in thread
From: Sean Whitton @ 2022-07-07 16:56 UTC (permalink / raw)
To: 56442; +Cc: notmuch, Eric Abrahamsen
[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]
Hello,
I'm running a pretty innocent notmuch query over a fairly small Maildir:
"((List:debian-devel.lists.debian.org) or ... or
(List:debian-haskell.lists.debian.org) or
(List:debconf-discuss.lists.debian.org)) and (not path:annex/**)"
but gnus-search-run-search fails to return any results. The reason is
that Gnus first runs the query with --output=threads to obtain a list of
thread ids, and then runs another query with --output=files and a query
constructed from the output of the first query: "thread:000000000000d9d0
or thread:000000000000d9e0 or thread:000000000000d268 or ..."
The resulting command fails completely:
emacs: /usr/bin/notmuch: Argument list too long
Instead of running two searches like this, we can just surround the
whole query like this: "thread:{QUERY}". The manual says it's exactly
equivalent:
... the user should think of the query thread:{<something>} as
expanding to all of the thread IDs which match <something>; not‐
much then performs a second search using the expanded query.
This should be faster, too, with only running a single external command.
Here is the patch I'm thinking I'll apply, if anyone has comments.
--
Sean Whitton
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Use-notmuch-thread-operator-instead-of-performing-tw.patch --]
[-- Type: text/x-diff, Size: 3370 bytes --]
From d62398cc27abd317d31d8f07e151a82081444217 Mon Sep 17 00:00:00 2001
From: Sean Whitton <spwhitton@spwhitton.name>
Date: Thu, 7 Jul 2022 09:34:04 -0700
Subject: [PATCH] Use notmuch thread:{} operator instead of performing two
searches
This also avoids some command line length limit problems.
* gnus-search.el
(gnus-search-run-search (engine gnus-search-notmuch)): Delete method.
(gnus-search-indexed-search-command (engine gnus-search-notmuch)):
When searching for threads, wrap whole query in thread:{} operator.
Always use --output=files, never --output=threads.
---
lisp/gnus/gnus-search.el | 46 +++++-----------------------------------
1 file changed, 5 insertions(+), 41 deletions(-)
diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el
index 369df81d9b..53b6d1b4c6 100644
--- a/lisp/gnus/gnus-search.el
+++ b/lisp/gnus/gnus-search.el
@@ -1672,43 +1672,6 @@ gnus-search-transform-expression
(format "date:%s.." (notmuch-date (cdr expr))))
(t (ignore-errors (cl-call-next-method))))))
-(cl-defmethod gnus-search-run-search :around ((engine gnus-search-notmuch)
- server query groups)
- "Handle notmuch's thread-search routine."
- ;; Notmuch allows for searching threads, but only using its own
- ;; thread ids. That means a thread search is a \"double-bounce\":
- ;; once to find the relevant thread ids, and again to find the
- ;; actual messages. This method performs the first \"bounce\".
- (if (alist-get 'thread query)
- (with-slots (program proc-buffer) engine
- (let* ((qstring
- (gnus-search-make-query-string engine query))
- (cp-list (gnus-search-indexed-search-command
- engine qstring query groups))
- thread-ids proc)
- (with-current-buffer proc-buffer
- (erase-buffer)
- (setq proc (apply #'start-process (format "search-%s" server)
- proc-buffer program cp-list))
- (while (process-live-p proc)
- (accept-process-output proc))
- (goto-char (point-min))
- (while (re-search-forward
- "^thread:\\([^[:space:]\n]+\\)"
- (point-max) t)
- (cl-pushnew (match-string 1) thread-ids :test #'equal)))
- (cl-call-next-method
- engine server
- ;; If we found threads, completely replace the query with
- ;; our new thread-based one.
- (if thread-ids
- `((query . ,(mapconcat (lambda (thrd)
- (concat "thread:" thrd))
- thread-ids " or ")))
- query)
- nil)))
- (cl-call-next-method engine server query groups)))
-
(cl-defmethod gnus-search-indexed-search-command ((engine gnus-search-notmuch)
(qstring string)
query &optional _groups)
@@ -1721,13 +1684,14 @@ gnus-search-indexed-search-command
(append
(list (format "--config=%s" config-file)
"search"
- (if thread
- "--output=threads"
- "--output=files"))
+ "--output=files")
(unless thread '("--duplicate=1"))
(when limit (list (format "--limit=%d" limit)))
switches
- (list qstring)))))
+ (list (if thread
+ (format "thread:\"{%s}\""
+ (string-replace "\"" "\"\"" qstring))
+ qstring))))))
;;; Mairix interface
--
2.30.2
^ permalink raw reply related [flat|nested] 5+ messages in thread
* bug#56442: gnus-search-run-search: Hits notmuch command line length limits
2022-07-07 16:56 bug#56442: gnus-search-run-search: Hits notmuch command line length limits Sean Whitton
@ 2022-07-09 16:11 ` Eric Abrahamsen
2022-07-09 18:01 ` Sean Whitton
0 siblings, 1 reply; 5+ messages in thread
From: Eric Abrahamsen @ 2022-07-09 16:11 UTC (permalink / raw)
To: Sean Whitton; +Cc: notmuch, 56442
Sean Whitton <spwhitton@spwhitton.name> writes:
> Hello,
>
> I'm running a pretty innocent notmuch query over a fairly small Maildir:
>
> "((List:debian-devel.lists.debian.org) or ... or
> (List:debian-haskell.lists.debian.org) or
> (List:debconf-discuss.lists.debian.org)) and (not path:annex/**)"
>
> but gnus-search-run-search fails to return any results. The reason is
> that Gnus first runs the query with --output=threads to obtain a list of
> thread ids, and then runs another query with --output=files and a query
> constructed from the output of the first query: "thread:000000000000d9d0
> or thread:000000000000d9e0 or thread:000000000000d268 or ..."
>
> The resulting command fails completely:
>
> emacs: /usr/bin/notmuch: Argument list too long
>
> Instead of running two searches like this, we can just surround the
> whole query like this: "thread:{QUERY}". The manual says it's exactly
> equivalent:
>
> ... the user should think of the query thread:{<something>} as
> expanding to all of the thread IDs which match <something>; not‐
> much then performs a second search using the expanded query.
>
> This should be faster, too, with only running a single external command.
> Here is the patch I'm thinking I'll apply, if anyone has comments.
Huh, I tried this a couple months ago with a more complicated query,
like:
thread:{from:bob or from:jane}
and I remember notmuch barking at me about spaces or the "or" or
something -- anyway I got the impression that it couldn't accept
multi-part queries inside the "thread:{}" syntax. But looking at your
patch maybe I just needed to quote differently?
Anyway I'll give this a test this weekend. It would be *very* nice if we
could use this syntax rather than the ugly home-grown one.
Thanks for the report!
Eric
^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#56442: gnus-search-run-search: Hits notmuch command line length limits
2022-07-09 16:11 ` Eric Abrahamsen
@ 2022-07-09 18:01 ` Sean Whitton
2022-07-14 18:50 ` Sean Whitton
0 siblings, 1 reply; 5+ messages in thread
From: Sean Whitton @ 2022-07-09 18:01 UTC (permalink / raw)
To: Eric Abrahamsen; +Cc: notmuch, 56442
Hello,
On Sat 09 Jul 2022 at 09:11AM -07, Eric Abrahamsen wrote:
> But looking at your patch maybe I just needed to quote differently?
Right, notmuch has this unusual " -> "" quoting thing.
> Anyway I'll give this a test this weekend. It would be *very* nice if
> we could use this syntax rather than the ugly home-grown one.
Let me know the results of your test, and if it's also continuing to
work for me by then, I'll install the change. Thanks for taking a look.
--
Sean Whitton
^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#56442: gnus-search-run-search: Hits notmuch command line length limits
2022-07-09 18:01 ` Sean Whitton
@ 2022-07-14 18:50 ` Sean Whitton
2022-07-14 22:11 ` Eric Abrahamsen
0 siblings, 1 reply; 5+ messages in thread
From: Sean Whitton @ 2022-07-14 18:50 UTC (permalink / raw)
To: 56442-done, Eric Abrahamsen
Hello,
On Sat 09 Jul 2022 at 11:01AM -07, Sean Whitton wrote:
> On Sat 09 Jul 2022 at 09:11AM -07, Eric Abrahamsen wrote:
>
>> Anyway I'll give this a test this weekend. It would be *very* nice if
>> we could use this syntax rather than the ugly home-grown one.
>
> Let me know the results of your test, and if it's also continuing to
> work for me by then, I'll install the change. Thanks for taking a look.
I've been testing it with a variety of complex queries for a week now,
and you indicated you were happy with making a change like this, so I've
gone ahead and pushed to master.
--
Sean Whitton
^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#56442: gnus-search-run-search: Hits notmuch command line length limits
2022-07-14 18:50 ` Sean Whitton
@ 2022-07-14 22:11 ` Eric Abrahamsen
0 siblings, 0 replies; 5+ messages in thread
From: Eric Abrahamsen @ 2022-07-14 22:11 UTC (permalink / raw)
To: Sean Whitton; +Cc: 56442-done
On 07/14/22 11:50 AM, Sean Whitton wrote:
> Hello,
>
> On Sat 09 Jul 2022 at 11:01AM -07, Sean Whitton wrote:
>
>> On Sat 09 Jul 2022 at 09:11AM -07, Eric Abrahamsen wrote:
>>
>>> Anyway I'll give this a test this weekend. It would be *very* nice if
>>> we could use this syntax rather than the ugly home-grown one.
>>
>> Let me know the results of your test, and if it's also continuing to
>> work for me by then, I'll install the change. Thanks for taking a look.
>
> I've been testing it with a variety of complex queries for a week now,
> and you indicated you were happy with making a change like this, so I've
> gone ahead and pushed to master.
Sounds good, thanks!
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-07-14 22:11 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-07-07 16:56 bug#56442: gnus-search-run-search: Hits notmuch command line length limits Sean Whitton
2022-07-09 16:11 ` Eric Abrahamsen
2022-07-09 18:01 ` Sean Whitton
2022-07-14 18:50 ` Sean Whitton
2022-07-14 22:11 ` Eric Abrahamsen
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.