From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eric Abrahamsen Newsgroups: gmane.emacs.bugs Subject: bug#44509: 28.0.50; Error querying with new gnus-search and notmuch Date: Wed, 11 Nov 2020 11:11:06 -0800 Message-ID: <87wnyremk5.fsf@ericabrahamsen.net> References: <87imag7kkh.fsf@gnus.jao.io> <87361kd1sg.fsf@ericabrahamsen.net> <87v9eg7ec3.fsf@gnus.jao.io> <874km0bgat.fsf@ericabrahamsen.net> <87blg8729u.fsf@gnus.jao.io> <875z6cxyue.fsf@gnus.jao.io> <877dqrhjea.fsf@ericabrahamsen.net> <87eekzwxuh.fsf@gnus.jao.io> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="30886"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: 44509@debbugs.gnu.org To: jao Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Nov 11 20:28:31 2020 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kcvnD-0007tv-19 for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 11 Nov 2020 20:28:31 +0100 Original-Received: from localhost ([::1]:34612 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kcvnC-0004as-1h for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 11 Nov 2020 14:28:30 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:51396) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kcvXG-0005QX-PX for bug-gnu-emacs@gnu.org; Wed, 11 Nov 2020 14:12:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:59533) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kcvXG-0002uF-FQ for bug-gnu-emacs@gnu.org; Wed, 11 Nov 2020 14:12:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kcvXG-0006S4-9y for bug-gnu-emacs@gnu.org; Wed, 11 Nov 2020 14:12:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Eric Abrahamsen Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 11 Nov 2020 19:12:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44509 X-GNU-PR-Package: emacs Original-Received: via spool by 44509-submit@debbugs.gnu.org id=B44509.160512187824744 (code B ref 44509); Wed, 11 Nov 2020 19:12:02 +0000 Original-Received: (at 44509) by debbugs.gnu.org; 11 Nov 2020 19:11:18 +0000 Original-Received: from localhost ([127.0.0.1]:42845 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kcvWX-0006R1-Gz for submit@debbugs.gnu.org; Wed, 11 Nov 2020 14:11:18 -0500 Original-Received: from ericabrahamsen.net ([52.70.2.18]:50112 helo=mail.ericabrahamsen.net) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kcvWU-0006Qm-Cu for 44509@debbugs.gnu.org; Wed, 11 Nov 2020 14:11:16 -0500 Original-Received: from localhost (24-113-150-48.wavecable.com [24.113.150.48]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id 35EDDFA16E; Wed, 11 Nov 2020 19:11:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericabrahamsen.net; s=mail; t=1605121868; bh=Ley9NMHZM26zfDWO0Prq0/p+XvN+Oz6N/wn9eOCL1C8=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=WNbhnO8HwR8o9BECHs/Jeu8xNO7zMf8mh9cTjqvGF24JqmUfzwjrz6sMJDhU98GYd Zrd7HDm215MZwSYqytbJXYgX89AzVV53/GrU1BQITOQuPU1lOFfOLTbKGIp31TaU8i F8eTbYjlS1E2WBvUdWyQIOXf+DCgIciZlStaoGEQ= In-Reply-To: <87eekzwxuh.fsf@gnus.jao.io> (jao's message of "Wed, 11 Nov 2020 18:29:58 +0000") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:193131 Archived-At: jao writes: > On Wed, Nov 11 2020, Eric Abrahamsen wrote: > > [...] > >> This code that comes straight from the old nnir.el, and has always >> looked a bit fragile to me. The problem is that it really just expects >> each message to be in a regular file, with groups as folders. >> >> So you're using notmuch as a search engine for a local leafnode nntp >> server, and indexing its message store directly? > > Yes. > >> Is there any leafnode setting that could influence how it stores its >> messages? Can it be convinced to store them in hierarchical folders? > > It is exactly what it is doing. Leafnode messages in the nntp group > gmane.emacs.bugs are stored in /prefix/gmane/emac/bugs/
> exactly as nnmail would do. And the result of the search by notmuch is > displaying those paths correctly. The problem is that, in the code i > mentioned, when that output is processed, the code is expecting to see > in the results list gmane.emacs.bugs instead of pathnames. So i don't > understand how that code has ever worked, even on the old nnir. I see, I had your problem backwards. My guess is that no one has tried to do this before, so the issue has never come up. BTW, while there's no longer a "gmane" search engine, it would be nice to provide an engine for searching other nntp servers. In your case you're running it locally, but do you know if a remote leafnode server handles any sort of search functionality? (I might as well google this myself.) >> I suppose I could change the group-regexp to munge periods, but that >> could cause breakage in other cases, and I would be hesitant to do that. > > In may case, the problem is the other way around: the regexp should > accept /, but is only accepting periods (because it's constructed > directly from the group name, which, for nntp at least, contains > periods, not slashes). > >> Otherwise, all the indexed search engines have a >> `gnus-search-indexed-extract' method that's used to actually return the >> file name from the results buffer. > > See above. It's really very easy for me to fix this with an around > method like this one: > > (cl-defmethod gnus-search-indexed-parse-output :around ((e gnus-search-notmuch) s q groups) > (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g)) > groups))) > (cl-call-next-method e s q gs))) > > because the problem in my case is the group name, not the search > results. Note that, for cases (if any) where the group names already > look like "gmane/emacs/bug" (which i suspect is what nnml and nnmaildir > are doing and that's why it works), that's a no-op. > > So yes, for me it's a solved problem, even without defining a new > engine. Okay! Glad that's sorted. You might still want a new engine, if you have other notmuch engines that shouldn't inherit this behavior. >>> On other news, i was trying to find a way in Gnus to go from Message-ID >>> to article no. for IMAP groups or nnmaildirs (which would make using >>> notmuch with dovecot really trivial), but without luck: anyone knows of >>> an easy way? >> >> Come to think of it, nnimap can already accept article numbers >> as message-ids. So if notmuch returns its results as message-ids, it >> should work transparently. > > Yes, i thought that at first. i got stuck when i discovered that > nnselect apparently only accepts article numbers to build its groups. i > guess one could extend nnselect to also accept message ids, and that > that would be useful also in other contexts. It looks like nnselect can accept message-ids when requesting individual articles, but not when categorizing groups/articles, nor when requesting headers. My guess is that might be quite a bit of work to implement, particularly since the id->number routine will be different for each backend. You can see that happening in `nnselect-request-article'. >> The problem is that while the mechanism is there, it works by searching >> each message-id and getting the proper article number that way. My guess >> is that you'd be negating most of the speed advantage of using notmuch >> like this, and you'd still be better off using dovecot's own full-text >> indexing. You don't have to use xapian! >> >> https://doc.dovecot.org/configuration_manual/fts/ > > Ah, thanks for that, i'll definitely take a look. The nice thing about > using notmuch would be that is one less moving piece to maintain, and > that then one could move very, very transparently between notmuch's > emacs interface and gnus, using the same message files and indexes. > >> BUT, if you really wanted to do this, it would be relatively easy to >> check for "--output=messages" in 'switches, and use that instead of >> "--output=files". > > Yes, it's trivial to get a vector of results of the form > [group message-id score] ... a full engine like that is just a few > lines: > > (require 'notmuch-query) > > (defclass jao-gnus-notmuch (gnus-search-notmuch) ()) > > (cl-defmethod gnus-search-run-search ((engine jao-gnus-notmuch) srv query groups) > (let ((qstring (gnus-search-make-query-string engine query)) > (res)) > (dolist (group groups) > (let* ((folder (format "folder:%s" > (replace-regexp-in-string "\\." "/" group))) > (ids (notmuch-query-get-message-ids qstring folder))) > (dolist (id ids) > (push (vector (gnus-group-full-name group srv) id 100) res)))) > res)) Hmm, I guess it wouldn't be useful for me to implement this in the general case, since the ids aren't going to be accepted by nnselect. > That's all (well, obviating thread seaches, but those are not hard > either). Except that it doesn't work because nnselect wants > [group article-no score]. If you tell me how to obtain article-no from > message-id, i'm done :) For nnimap, it's `nnimap-find-article-by-message-id'. Eric