Gnus and notmuch

Gnus and notmuch

[Alexis Praga]

Hi guys,

I've tried to setup Gnus to use notmuch as a search engine
(configuration below) but running G G on the group buffer yields an
error "(wrong-type-argument number-or-marker-p nil)" (backtrace below).

My setup uses 2 maildir. If I set only the "gmail" one, I can run
searches but there are never any results. Using the "free" one, I have
the previous error.

Any help appreciated !

Configuration :
>(setq gnus-select-method '(nnnil "")
>      gnus-secondary-select-methods
>      '((nnmaildir "gmail" (directory "~/mail/gmail")
>		   (gnus-search-engine gnus-search-notmuch)
>		   (config-file "/home/alex/.notmuch-config")
>                   (get-new-mail nil)
>		   )
>	(nnmaildir "free" (directory "~/mail/free")
>		   (gnus-search-engine gnus-search-notmuch)
>		   (config-file "/home/alex/.notmuch-config")
>                   (get-new-mail nil)
>		   ))
>      gnus-search-use-parsed-queries nil) ; Emacs 28

Config: GNU Emacs 28.0.50

Backtrace:
>  <(nil nil)
>  nnselect-compress-artlist([["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.free.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.free.archive" nil 100] ["nnmaildir+free:home.alex.mail.free.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.free.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ["nnmaildir+free:home.alex.mail.gmail.archive" nil 100] ...])
>  nnselect-request-group("nnselect-864ke6gm6o.fsf" "nnselect-ephemeral" t ("nnselect:nnselect-864ke6gm6o.fsf" 3 nil nil (nnselect "nnselect-ephemeral" (nnselect-address "nnselect")) ((quit-config #<buffer *Group*> . group) (nnselect-specs (nnselect-function . gnus-search-run-query) (nnselect-args (search-query-spec (query . "soutenance") (raw)) (search-group-spec ("nnmaildir:free" "nnmaildir+free:archive")))) (nnselect-artlist))))
>  gnus-request-group("nnselect:nnselect-864ke6gm6o.fsf" t nil ("nnselect:nnselect-864ke6gm6o.fsf" 3 nil nil (nnselect "nnselect-ephemeral" (nnselect-address "nnselect")) ((quit-config #<buffer *Group*> . group) (nnselect-specs (nnselect-function . gnus-search-run-query) (nnselect-args (search-query-spec (query . "soutenance") (raw)) (search-group-spec ("nnmaildir:free" "nnmaildir+free:archive")))) (nnselect-artlist))))
>  gnus-select-newsgroup("nnselect:nnselect-864ke6gm6o.fsf" t nil)
>  gnus-summary-read-group-1("nnselect:nnselect-864ke6gm6o.fsf" t t nil nil nil)
>  gnus-summary-read-group("nnselect:nnselect-864ke6gm6o.fsf" t t nil nil nil nil)
>  gnus-group-read-group(t t "nnselect:nnselect-864ke6gm6o.fsf" nil)
>  gnus-group-read-ephemeral-group("nnselect-864ke6gm6o.fsf" (nnselect "nnselect") nil (#<buffer *Group*> . group) nil nil ((nnselect-specs (nnselect-function . gnus-search-run-query) (nnselect-args (search-query-spec (query . "soutenance") (raw)) (search-group-spec ("nnmaildir:free" "nnmaildir+free:archive")))) (nnselect-artlist)))
>  gnus-group-read-ephemeral-search-group(nil)
>  funcall-interactively(gnus-group-read-ephemeral-search-group nil)
>  command-execute(gnus-group-read-ephemeral-search-group)

-- 
Alexis Praga

Re: Gnus and notmuch

[Eric Abrahamsen]

Alexis Praga <alexis.praga@gmail.com> writes:

> Hi guys,
>
> I've tried to setup Gnus to use notmuch as a search engine
> (configuration below) but running G G on the group buffer yields an
> error "(wrong-type-argument number-or-marker-p nil)" (backtrace below).
>
> My setup uses 2 maildir. If I set only the "gmail" one, I can run
> searches but there are never any results. Using the "free" one, I have
> the previous error.
>
> Any help appreciated !
>
> Configuration :
>>(setq gnus-select-method '(nnnil "")
>>      gnus-secondary-select-methods
>>      '((nnmaildir "gmail" (directory "~/mail/gmail")
>>		   (gnus-search-engine gnus-search-notmuch)
>>		   (config-file "/home/alex/.notmuch-config")
>>                   (get-new-mail nil)
>>		   )
>>	(nnmaildir "free" (directory "~/mail/free")
>>		   (gnus-search-engine gnus-search-notmuch)
>>		   (config-file "/home/alex/.notmuch-config")
>>                   (get-new-mail nil)
>>		   ))
>>      gnus-search-use-parsed-queries nil) ; Emacs 28

The first thing will be to get the (config-file "file") bits inside the
search engine config, like so:

(nnmaildir "gmail" (directory "~/mail/gmail")
	   (gnus-search-engine gnus-search-notmuch
	     (config-file "/home/alex/.notmuch-config")
             (remove-prefix "~/mail/gmail"))
           (get-new-mail nil))

Note that you'll also need a remove-prefix setting for each engine, also
inside the gnus-search-engine sexp. That's a required setting, so I'll
add it to the example in the search section of the manual, it's
confusing to not have it in there.

Try with updated config as above, and let me know if that fixes it.

Yours,
Eric

Re: Gnus and notmuch

[Alexis Praga]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:
> Well, one problem at a time :)
>
> While it should be possible to use a single notmuch installation to
> search two nnmaildir servers, as you're doing, it's also possible that
> this is the source of the problem. Could you also show me your
> ~/.notmuch-config?
>
> These two lines are the source of the problem: gnus-search was able to
> find the article numbers from search hits in the "archive" group, but
> was not able to turn search hits in the "home.alex.mail.free.archive"
> group into article numbers. Usually this means it wasn't actually able
> to locate the files on disk, which usually means there's a mismatch with
> the 'remove-prefix setting.
>
> From the traceback, it looks like you were only searching the
> nnmaildir+gmail:archive group. Do you have any idea why you were getting
> results from home.alex.mail.free.archive?
>
> Eric

(Cc'ing the mailing-list...)

Notmuch is configured to parse a folder containing two maildirs (gmail
and free) like this:

> [database]
> path=/home/alex/mail
> [user]
> name=Alexis Praga
> primary_email=$GMAIL
> other_email=$FREE
> 
> [new]
> tags=inbox;unread;new
> ignore=.nnmaildir
> 
> [search]
> exclude_tags=deleted;spam;
> 
> [maildir]
> synchronize_flags=true
> 
> [crypto]
> gpg_path=gpg

If I setup notmuch to take care of one maildir only and using the same
setup with notmuch, I don't have any errors but I don't have any results
either:
> Group nnselect:nnselect-86eed9eef9.fsf contains no messages
Also, removing "ignore=.nnmaildir" from the configuration of notmuch does not help.

I don'tt really know why it is searching for the other maildir. That's
why I have been restarting emacs and removing ~/.newrrc.el*, hoping it
would resolve some conflict with my previous attempts.

Thanks,

-- 
Alexis Praga

Re: Gnus and notmuch

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 2255 bytes --]

Alexis Praga <alexis.praga@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>> Well, one problem at a time :)
>>
>> While it should be possible to use a single notmuch installation to
>> search two nnmaildir servers, as you're doing, it's also possible that
>> this is the source of the problem. Could you also show me your
>> ~/.notmuch-config?
>>
>> These two lines are the source of the problem: gnus-search was able to
>> find the article numbers from search hits in the "archive" group, but
>> was not able to turn search hits in the "home.alex.mail.free.archive"
>> group into article numbers. Usually this means it wasn't actually able
>> to locate the files on disk, which usually means there's a mismatch with
>> the 'remove-prefix setting.
>>
>> From the traceback, it looks like you were only searching the
>> nnmaildir+gmail:archive group. Do you have any idea why you were getting
>> results from home.alex.mail.free.archive?
> I don'tt really know why it is searching for the other maildir. That's
> why I have been restarting emacs and removing ~/.newrrc.el*, hoping it
> would resolve some conflict with my previous attempts.

I think see the problem, it's yet more fragility in parsing the output
of search results.

Notmuch is returning all results in all directories, and Gnus filters
out the results it wants depending on which paths match the group name.
Because "archive" is a substring of "home.alex.mail.free.archive", it
thinks it's a relevant search result.

I guess I'm not entirely sure why it fails to find the articles from
that group -- I would expect the articles from the other group to be
located correctly -- but this is probably still the source of the
problem.

This stuff is so fragile that I dread to think of what else might break
with this change, but... If you're comfortable patching and
re-evaluating, would you please apply the attached, then re-evaluate the
`gnus-search-indexed-parse-output' method, and see if that fixes the
problem?

Another proper solution (in addition to this) would be to figure out how
to pass the groups parameter to notmuch in either its folder: or path:
key. That would avoid duplicate work, and presumably run faster. I'll
put this on the todo list.

Eric



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnus-search-groups.diff --]
[-- Type: text/x-patch, Size: 971 bytes --]

diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el
index fc9f8684f6..71a65ab95d 100644
--- a/lisp/gnus/gnus-search.el
+++ b/lisp/gnus/gnus-search.el
@@ -1347,15 +1347,18 @@ gnus-search-indexed-parse-output
 						server query &optional groups)
   (let ((prefix (slot-value engine 'remove-prefix))
 	(group-regexp (when groups
-			(mapconcat
-			 (lambda (group-name)
-			   (mapconcat #'regexp-quote
-				      (split-string
-				       (gnus-group-real-name group-name)
-				       "[.\\/]")
-				      "[.\\\\/]"))
-			 groups
-			 "\\|")))
+                        (concat
+                         "\\`"
+			 (mapconcat
+			  (lambda (group-name)
+			    (mapconcat #'regexp-quote
+				       (split-string
+				        (gnus-group-real-name group-name)
+				        "[.\\/]")
+				       "[.\\\\/]"))
+			  groups
+			  "\\|")
+                         "\\'")))
 	artlist vectors article group)
     (goto-char (point-min))
     (while (not (or (eobp)

Re: Gnus and notmuch

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Alexis Praga <alexis.praga@gmail.com> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>> Well, one problem at a time :)
>>>
>>> While it should be possible to use a single notmuch installation to
>>> search two nnmaildir servers, as you're doing, it's also possible that
>>> this is the source of the problem. Could you also show me your
>>> ~/.notmuch-config?
>>>
>>> These two lines are the source of the problem: gnus-search was able to
>>> find the article numbers from search hits in the "archive" group, but
>>> was not able to turn search hits in the "home.alex.mail.free.archive"
>>> group into article numbers. Usually this means it wasn't actually able
>>> to locate the files on disk, which usually means there's a mismatch with
>>> the 'remove-prefix setting.
>>>
>>> From the traceback, it looks like you were only searching the
>>> nnmaildir+gmail:archive group. Do you have any idea why you were getting
>>> results from home.alex.mail.free.archive?
>> I don'tt really know why it is searching for the other maildir. That's
>> why I have been restarting emacs and removing ~/.newrrc.el*, hoping it
>> would resolve some conflict with my previous attempts.
>
> I think see the problem, it's yet more fragility in parsing the output
> of search results.
>
> Notmuch is returning all results in all directories, and Gnus filters
> out the results it wants depending on which paths match the group name.
> Because "archive" is a substring of "home.alex.mail.free.archive", it
> thinks it's a relevant search result.
>
> I guess I'm not entirely sure why it fails to find the articles from
> that group -- I would expect the articles from the other group to be
> located correctly -- but this is probably still the source of the
> problem.
>
> This stuff is so fragile that I dread to think of what else might break
> with this change, but... If you're comfortable patching and
> re-evaluating, would you please apply the attached, then re-evaluate the
> `gnus-search-indexed-parse-output' method, and see if that fixes the
> problem?

Ignore this patch! It will choke on the maildir "cur" directory segment.
Hold on for a bit here while I try to think of a proper solution for the
problem.

Eric

Re: Gnus and notmuch

[Alexis Praga]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>> I think see the problem, it's yet more fragility in parsing the output
>> of search results.
>>
>> Notmuch is returning all results in all directories, and Gnus filters
>> out the results it wants depending on which paths match the group name.
>> Because "archive" is a substring of "home.alex.mail.free.archive", it
>> thinks it's a relevant search result.

Here's a naïve suggestion: can the argument of "remove-prefix" be used
to filter out the search results outside the maildir ?

-- 
Alexis Praga

Re: Gnus and notmuch

[Eric Abrahamsen]

Alexis Praga <alexis.praga@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>> I think see the problem, it's yet more fragility in parsing the output
>>> of search results.
>>>
>>> Notmuch is returning all results in all directories, and Gnus filters
>>> out the results it wants depending on which paths match the group name.
>>> Because "archive" is a substring of "home.alex.mail.free.archive", it
>>> thinks it's a relevant search result.
>
> Here's a naïve suggestion: can the argument of "remove-prefix" be used
> to filter out the search results outside the maildir ?

Those are two separate issues: the remove-prefix gets rid of everything
from the root of your filesystem to the top-level of your mail store.
All remaining path segments are interpreted as a (potentially-nested)
group name. Then the filename itself is converted to an article number.

Re: Gnus and notmuch

[Alexandr Vityazev]

On 2021-06-10, 10:35 +0200, Alexis Praga <alexis.praga@gmail.com> wrote:

> Hi guys,
>
> I've tried to setup Gnus to use notmuch as a search engine
> (configuration below) but running G G on the group buffer yields an
> error "(wrong-type-argument number-or-marker-p nil)" (backtrace below).
>
> My setup uses 2 maildir. If I set only the "gmail" one, I can run
> searches but there are never any results. Using the "free" one, I have
> the previous error.
>
> Any help appreciated !
>
> Configuration :
>>(setq gnus-select-method '(nnnil "")
>>      gnus-secondary-select-methods
>>      '((nnmaildir "gmail" (directory "~/mail/gmail")
>>		   (gnus-search-engine gnus-search-notmuch)
>>		   (config-file "/home/alex/.notmuch-config")
>>                   (get-new-mail nil)
>>		   )
>>	(nnmaildir "free" (directory "~/mail/free")
>>		   (gnus-search-engine gnus-search-notmuch)
>>		   (config-file "/home/alex/.notmuch-config")
>>                   (get-new-mail nil)
>>		   ))
>>      gnus-search-use-parsed-queries nil) ; Emacs 28
You can use the environment variable NOTMUCH_PROFILE and define
two different config files for each email address. 
-- 
Best regards,
Alexandr Vityazev

Re: Gnus and notmuch

[Alexis Praga]

Alexandr Vityazev <avityazev@posteo.net> writes:

> You can use the environment variable NOTMUCH_PROFILE and define
> two different config files for each email address. 

Hi Alexandr,

So I've setup two maildirs and used 2 different notmuch
config (see below). No more errors now, but all my searches returns empty
results.

> (setq gnus-select-method '(nnnil "")
>       gnus-secondary-select-methods
>       '((nnmaildir "gmail" (directory "~/mail/gmail")
> 		   (gnus-search-engine gnus-search-notmuch
> 				       (config-file "/home/alex/.notmuch-config-gmail")
> 				       (remove-prefix "~/mail/gmail"))
> 		   (get-new-mail nil))
> 	(nnmaildir "free" (directory "~/mail/free")
> 		   (gnus-search-engine gnus-search-notmuch
> 				       (config-file "/home/alex/.notmuch-config-free")
> 				       (remove-prefix "~/mail/free"))
> 		   (get-new-mail nil))
>       ))

-- 
Alexis Praga

Re: Gnus and notmuch

[Eric Abrahamsen]

Alexis Praga <alexis.praga@gmail.com> writes:

> Alexandr Vityazev <avityazev@posteo.net> writes:
>
>> You can use the environment variable NOTMUCH_PROFILE and define
>> two different config files for each email address. 
>
> Hi Alexandr,
>
> So I've setup two maildirs and used 2 different notmuch
> config (see below). No more errors now, but all my searches returns empty
> results.

If you look in *Messages* and search for nnselect-run, do you see any
more specific error messages?

Re: Gnus and notmuch

[Alexis Praga]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:
> If you look in *Messages* and search for nnselect-run, do you see any
> more specific error messages?

No, there's only
> Group nnselect:nnselect-86a6nvw3hv.fsf contains no messages

Setting "gnus-verbose" to true before starting gnus results in gnus
refusing to start with
> gnus-check-server: Wrong type argument: number-or-marker-p, nil

Setting it after starting gnus make me unabel to read groups:
>  gnus-summary-read-group-1: Wrong type argument: number-or-marker-p, t

I'm also unable to exit gnus:
> gnus-save-newsrc-file: Wrong type argument: number-or-marker-p, t

-- 
Alexis Praga

Re: Gnus and notmuch

[Alexandr Vityazev]

On 2021-06-12, 16:45 +0200, Alexis Praga <alexis.praga@gmail.com> wrote:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>> If you look in *Messages* and search for nnselect-run, do you see any
>> more specific error messages?
>
> No, there's only
>> Group nnselect:nnselect-86a6nvw3hv.fsf contains no messages
>
> Setting "gnus-verbose" to true before starting gnus results in gnus
> refusing to start with
>> gnus-check-server: Wrong type argument: number-or-marker-p, nil
>
> Setting it after starting gnus make me unabel to read groups:
>>  gnus-summary-read-group-1: Wrong type argument: number-or-marker-p, t
>
> I'm also unable to exit gnus:
>> gnus-save-newsrc-file: Wrong type argument: number-or-marker-p, t

do you have a [Gmail] dir in the ~/mail/gmail directory? If so then try the
following config:

(nnmaildir "gmail" (directory "~/mail/gmail/[Gmail]")
 		   (gnus-search-engine gnus-search-notmuch
 				       (config-file "/home/alex/.notmuch-config-gmail")
 				       (remove-prefix "~/mail/gmail/[Gmail]"))
-- 

Alexandr Vityazev

Re: Gnus and notmuch

[Alexis Praga]

> Le 12 juin 2021 à 18:50, Alexandr Vityazev <avityazev@posteo.net> a écrit :
> 
> On 2021-06-12, 16:45 +0200, Alexis Praga <alexis.praga@gmail.com> wrote:
> 
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>> If you look in *Messages* and search for nnselect-run, do you see any
>>> more specific error messages?
>> 
>> No, there's only
>>> Group nnselect:nnselect-86a6nvw3hv.fsf contains no messages
>> 
>> Setting "gnus-verbose" to true before starting gnus results in gnus
>> refusing to start with
>>> gnus-check-server: Wrong type argument: number-or-marker-p, nil
>> 
>> Setting it after starting gnus make me unabel to read groups:
>>> gnus-summary-read-group-1: Wrong type argument: number-or-marker-p, t
>> 
>> I'm also unable to exit gnus:
>>> gnus-save-newsrc-file: Wrong type argument: number-or-marker-p, t
> 
> do you have a [Gmail] dir in the ~/mail/gmail directory? If so then try the
> following config:
> 
> (nnmaildir "gmail" (directory "~/mail/gmail/[Gmail]")
>           (gnus-search-engine gnus-search-notmuch
>                       (config-file "/home/alex/.notmuch-config-gmail")
>                       (remove-prefix "~/mail/gmail/[Gmail]"))
> -- 
> 
> Alexandr Vityazev

Hi,

No, I don’t have a [Gmail] directory. I use mbsync to map the gmail account to three maildir : inbox, archive and sent (inside the gmail folder). 

Thanks