From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id UMaiGWYQx2JfbwAAbAwnHQ (envelope-from ) for ; Thu, 07 Jul 2022 18:57:10 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id AJOKGWYQx2KRggAAauVa8A (envelope-from ) for ; Thu, 07 Jul 2022 18:57:10 +0200 Received: from mail.notmuchmail.org (yantan.tethera.net [135.181.149.255]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 17C9F838 for ; Thu, 7 Jul 2022 18:57:09 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 4F0BD5E545; Thu, 7 Jul 2022 16:57:06 +0000 (UTC) Received: from debbugs.gnu.org (debbugs.gnu.org [209.51.188.43]) by mail.notmuchmail.org (Postfix) with ESMTPS id 4970C5DD92 for ; Thu, 7 Jul 2022 16:57:04 +0000 (UTC) Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1o9UoI-0002T2-Jl; Thu, 07 Jul 2022 12:57:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#56442: gnus-search-run-search: Hits notmuch command line length limits Resent-From: Sean Whitton Original-Sender: "Debbugs-submit" Resent-CC: notmuch@notmuchmail.org, eric@ericabrahamsen.net, bug-gnu-emacs@gnu.org Resent-Date: Thu, 07 Jul 2022 16:57:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 56442 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: To: 56442@debbugs.gnu.org X-Debbugs-Original-To: bug-gnu-emacs@gnu.org X-Debbugs-Original-Xcc: notmuch@notmuchmail.org, Eric Abrahamsen Received: via spool by submit@debbugs.gnu.org id=B.16572129809434 (code B ref -1); Thu, 07 Jul 2022 16:57:02 +0000 Received: (at submit) by debbugs.gnu.org; 7 Jul 2022 16:56:20 +0000 Received: from localhost ([127.0.0.1]:57746 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o9Unb-0002S5-V2 for submit@debbugs.gnu.org; Thu, 07 Jul 2022 12:56:20 -0400 Received: from lists.gnu.org ([209.51.188.17]:42762) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o9UnW-0002Rt-C9 for submit@debbugs.gnu.org; Thu, 07 Jul 2022 12:56:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55146) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o9UnW-0005P6-3S for bug-gnu-emacs@gnu.org; Thu, 07 Jul 2022 12:56:14 -0400 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:46581) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o9UnU-0003RN-3L for bug-gnu-emacs@gnu.org; Thu, 07 Jul 2022 12:56:13 -0400 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id E9E553200302 for ; Thu, 7 Jul 2022 12:56:08 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Thu, 07 Jul 2022 12:56:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=spwhitton.name; h=cc:content-type:date:date:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to; s=fm2; t= 1657212968; x=1657299368; bh=OxU+Ily5TkckCY4kzVnSzv4SklrY5aEasGb SUEhThfM=; b=JgleLBUhQtZYGiLCVq/a2K8uHcyRcpLm09nplOT4IzPjqbALVQc TlpLeiWlXUqRWnKnWG3mYy7pYb+ej8DrXRxkSU2lZUNtT3WL5ihkWV/evmat71IY SK4rrlDhECyUm5HGpyK93EzJpkd12eIBAST/iOca1T5CBAz0T3t5Ks+zWyfc/pnR TTrm1Q+5awXU5Zl8ynladXdFhto0J8S143ctS9rscHNrtlW6wNwUac41UQXokI4A Jx3+K4dL9gHEVnM5cZMy8vJFQMGBrhz9bnZ0JmdlmLoj8YorsS1cvli39CKwl7Ii Onvbnz0SDGiBf0VZ+zKUwixHVeEPVdicjYA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:message-id:mime-version :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1657212968; x= 1657299368; bh=OxU+Ily5TkckCY4kzVnSzv4SklrY5aEasGbSUEhThfM=; b=U nQT3JdKFr3GEq3iEodCwo7GbtiySEvoOcYQEQg8+f2BEOdONXgc5RI37JKEHMxEp IE/eTbGJ/0Zsfqo1JzhgApC9FVgNUVC8lCVEg64RaHfZ/zLi/g8RdayZ+Jrwt6vI aYnEkYqHWBw/BnUs0b5hPPxRWZd3U3rv00YJh9Ry6IffktdhYYEtVH1IsMZH/4H6 NIo5UAZaEdkPSjc0SvSka2mEVwJ91d/msivTP+TTglY2mscEhBsl/kspweno5CHn aaiUdo091XVauMcmOCcqzUXdae37wFILfMP2HeadCRyPbHiDxeu01cfzeIiMHfSG c/77GRTMAiynreYptdwOQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrudeihedguddthecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhephffvufffkfgfgggtsehmtddtre dtreejnecuhfhrohhmpefuvggrnhcuhghhihhtthhonhcuoehsphifhhhithhtohhnsehs phifhhhithhtohhnrdhnrghmvgeqnecuggftrfgrthhtvghrnhepveeiheehuedtveehge dvfffgkeffuedtffevvefgtddugeehjedvteejteefffegnecuffhomhgrihhnpeguvggs ihgrnhdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfh hrohhmpehsphifhhhithhtohhnsehsphifhhhithhtohhnrdhnrghmvg X-ME-Proxy: Feedback-ID: i23c04076:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 7 Jul 2022 12:56:08 -0400 (EDT) Received: by athena.silentflame.com (Postfix, from userid 1000) id E68C61B5A61; Thu, 7 Jul 2022 16:56:05 +0000 (UTC) From: Sean Whitton Date: Thu, 07 Jul 2022 09:56:05 -0700 Message-ID: <875yk850be.fsf@athena.silentflame.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/29.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=64.147.123.19; envelope-from=spwhitton@spwhitton.name; helo=wout3-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Archive: Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" Message-ID-Hash: PGGTHV6O6XSULQIVBB6SURL43FO2XEL5 X-Message-ID-Hash: PGGTHV6O6XSULQIVBB6SURL43FO2XEL5 X-MailFrom: Debian-debbugs@debbugs.gnu.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: notmuch@notmuchmail.org, Eric Abrahamsen List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: DE ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1657213030; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:list-id:list-help:list-owner:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=lcvqPDaWVMqsimii7SxlVFvuxkHDrHxZNLSZ/p1dz/M=; b=GXdsGNpH5SzHGX1CdDAi55neLb+Ok4pCRYr8rXIiW01G8Q5uIe76GDPy4VNh2OKx9saAiu a3J2ALQMnFW49K2ZUYo2wI+/jOvqe4xMvd2/TP1ZyImsXJDYXcUSivkcPbWq+MCZzjdGd0 BENz4Il3EP80m5ywjI3N1EyNh46Pf6s7BSYOlDgDCG0ZdcrGqYxdHdruayYxLRS2A6IyPt blzGo6ScaxUxQqUg/QdNtYB4li9vRw84OaNdUgWQHwVM6Hf1bk7TBBphNNjJHTWIpXfH9q qEHj4cIa7pxGru/iAY6TkGN1nqXT7zzxuKBg8vkVIUz8cciif0vjjqAuTUx5Xg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1657213030; a=rsa-sha256; cv=none; b=Urgwj0KgXMgbe3RRUBEJocr84dnE3DG8h+6YQcCImCuNXeLi81hmE7P7NQy3qvN88dqxaZ rG/hLhq/KSU6C8hlddlDw2gF/SGY++O+klhFtkUFNbcWtdzDiREtY/mkXPPX9zdB1hkR/E G/bQBqjsz+vUgj+PL7LXo6PI5AOg8vWAE6pr9FoxZnApLk46O/FECZ6kngJIv2GMa5Z65p nZv1hv96l+Q4KDWc9HCMyWSI/u/6S9+OW6tmAHb0GbHzLYn79gjIBiaViAo4W0j7CNMPeI zIBS52j+HZDx8yA1a2qhQMGGzARw8zzTvW2unZq4pKZU7RE1S28sW3opJelS1A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=spwhitton.name header.s=fm2 header.b=JgleLBUh; dkim=fail ("body hash did not verify") header.d=messagingengine.com header.s=fm3 header.b="U nQT3Jd"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: 1.01 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=spwhitton.name header.s=fm2 header.b=JgleLBUh; dkim=fail ("body hash did not verify") header.d=messagingengine.com header.s=fm3 header.b="U nQT3Jd"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: 17C9F838 X-Spam-Score: 1.01 X-Migadu-Scanner: scn1.migadu.com X-TUID: RbF/lsi0zj5z --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello, I'm running a pretty innocent notmuch query over a fairly small Maildir: "((List:debian-devel.lists.debian.org) or ... or (List:debian-haskell.lists.debian.org) or (List:debconf-discuss.lists.debian.org)) and (not path:annex/**)" but gnus-search-run-search fails to return any results. The reason is that Gnus first runs the query with --output=3Dthreads to obtain a list of thread ids, and then runs another query with --output=3Dfiles and a query constructed from the output of the first query: "thread:000000000000d9d0 or thread:000000000000d9e0 or thread:000000000000d268 or ..." The resulting command fails completely: emacs: /usr/bin/notmuch: Argument list too long Instead of running two searches like this, we can just surround the whole query like this: "thread:{QUERY}". The manual says it's exactly equivalent: ... the user should think of the query thread:{} as expanding to all of the thread IDs which match ; not=E2=80=90 much then performs a second search using the expanded query. This should be faster, too, with only running a single external command. Here is the patch I'm thinking I'll apply, if anyone has comments. --=20 Sean Whitton --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-Use-notmuch-thread-operator-instead-of-performing-tw.patch >From d62398cc27abd317d31d8f07e151a82081444217 Mon Sep 17 00:00:00 2001 From: Sean Whitton Date: Thu, 7 Jul 2022 09:34:04 -0700 Subject: [PATCH] Use notmuch thread:{} operator instead of performing two searches This also avoids some command line length limit problems. * gnus-search.el (gnus-search-run-search (engine gnus-search-notmuch)): Delete method. (gnus-search-indexed-search-command (engine gnus-search-notmuch)): When searching for threads, wrap whole query in thread:{} operator. Always use --output=files, never --output=threads. --- lisp/gnus/gnus-search.el | 46 +++++----------------------------------- 1 file changed, 5 insertions(+), 41 deletions(-) diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el index 369df81d9b..53b6d1b4c6 100644 --- a/lisp/gnus/gnus-search.el +++ b/lisp/gnus/gnus-search.el @@ -1672,43 +1672,6 @@ gnus-search-transform-expression (format "date:%s.." (notmuch-date (cdr expr)))) (t (ignore-errors (cl-call-next-method)))))) -(cl-defmethod gnus-search-run-search :around ((engine gnus-search-notmuch) - server query groups) - "Handle notmuch's thread-search routine." - ;; Notmuch allows for searching threads, but only using its own - ;; thread ids. That means a thread search is a \"double-bounce\": - ;; once to find the relevant thread ids, and again to find the - ;; actual messages. This method performs the first \"bounce\". - (if (alist-get 'thread query) - (with-slots (program proc-buffer) engine - (let* ((qstring - (gnus-search-make-query-string engine query)) - (cp-list (gnus-search-indexed-search-command - engine qstring query groups)) - thread-ids proc) - (with-current-buffer proc-buffer - (erase-buffer) - (setq proc (apply #'start-process (format "search-%s" server) - proc-buffer program cp-list)) - (while (process-live-p proc) - (accept-process-output proc)) - (goto-char (point-min)) - (while (re-search-forward - "^thread:\\([^[:space:]\n]+\\)" - (point-max) t) - (cl-pushnew (match-string 1) thread-ids :test #'equal))) - (cl-call-next-method - engine server - ;; If we found threads, completely replace the query with - ;; our new thread-based one. - (if thread-ids - `((query . ,(mapconcat (lambda (thrd) - (concat "thread:" thrd)) - thread-ids " or "))) - query) - nil))) - (cl-call-next-method engine server query groups))) - (cl-defmethod gnus-search-indexed-search-command ((engine gnus-search-notmuch) (qstring string) query &optional _groups) @@ -1721,13 +1684,14 @@ gnus-search-indexed-search-command (append (list (format "--config=%s" config-file) "search" - (if thread - "--output=threads" - "--output=files")) + "--output=files") (unless thread '("--duplicate=1")) (when limit (list (format "--limit=%d" limit))) switches - (list qstring))))) + (list (if thread + (format "thread:\"{%s}\"" + (string-replace "\"" "\"\"" qstring)) + qstring)))))) ;;; Mairix interface -- 2.30.2 --=-=-= Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --=-=-=--