bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Hello,

this is probably a member of one class of my repeated Gnus bug reports
about one and the same thing that breaks and breaks again.

Some years ago I had setup nnmairix for Gnus to be able to search
mails.  It was hard.

It worked for some months.  Since then, every single time when I tried to
use it, it was broken, and needed manual intervention to fix.  Most of
the time, I had to repeat the setup, and that worked.

Now, I tried several hours to get it work, but failed, no chance
whatever I tried.  I see that mairix works fine in the background, but
Gnus doesn't.  I only get empty or inaccessible groups corresponding to
the search.  Recreating or editing the related nnml group(s) doesn't
help.  Gnus doesn't let me edit the nnmairix server definition, it says
I had to do it "in my configuration".  I have no clue in which.  I don't
create that server in my normal setup.

.newsrc.eld maybe?  Thanks to the overlong lines, Emacs is not even able
to edit that file.  I could try longlines or so, but I gave up for now,
since I don't even know if it is the right file and what I should try.

Is anybody using this?  Should be a common task to search mails in
Gnus.  Why is it so hard?

TIA,

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Hello,
> [rants about nnmairix not working in Gnus...]

Forgot to say this is in Emacs master.

Thanks,

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> > [rants about nnmairix not working in Gnus...]
>
> Forgot to say this is in Emacs master.

Some more experimentation: Gnus calls the mairix process, mairix creates
and fills directories with search results as expected, these also seem
to appear as newly created search-related groups, but visiting these
groups always fails with "contains no messages".  G D with this dir
works.

Anyone any idea?

TIA,

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
>> > [rants about nnmairix not working in Gnus...]
>>
>> Forgot to say this is in Emacs master.
>
> Some more experimentation: Gnus calls the mairix process, mairix creates
> and fills directories with search results as expected, these also seem
> to appear as newly created search-related groups, but visiting these
> groups always fails with "contains no messages".  G D with this dir
> works.
>
> Anyone any idea?

My only advice is to not use nnmairix (which no one is maintaining), and
just use mairix as a regular backend to gnus-search (which should do
everything that nnmairix ever did).

Not helpful, I know, but that's all I've got...

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> My only advice is to not use nnmairix (which no one is maintaining), and
> just use mairix as a regular backend to gnus-search (which should do
> everything that nnmairix ever did).
>
> Not helpful, I know, but that's all I've got...

Oh -- no, very helpful!

Can we then maybe make it a bit more discoverable that nnmairix is
obsolete (and if it's unmaintained and doesn't work, it is, in reality,
very obsolete)?  If I don't read the introducing paragraph of the large
part that covers nnmairix in the Gnus manual (and why should I when I
know what it does), I can't obtain that information (or can I?).  I have
wasted hours of trying to get nnmairx work and debug it the last two
days.


Thanks,

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Eric Abrahamsen]

On 03/16/21 01:35 AM, Michael Heerdegen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> My only advice is to not use nnmairix (which no one is maintaining), and
>> just use mairix as a regular backend to gnus-search (which should do
>> everything that nnmairix ever did).
>>
>> Not helpful, I know, but that's all I've got...
>
> Oh -- no, very helpful!
>
> Can we then maybe make it a bit more discoverable that nnmairix is
> obsolete (and if it's unmaintained and doesn't work, it is, in reality,
> very obsolete)?  If I don't read the introducing paragraph of the large
> part that covers nnmairix in the Gnus manual (and why should I when I
> know what it does), I can't obtain that information (or can I?).  I have
> wasted hours of trying to get nnmairx work and debug it the last two
> days.

Ugh, I'm sorry. I knew that nnmairix's functionality was obsoleted by
the newer libraries, but didn't know that it simply didn't work anymore.
Sorry for your wasted time!

I think it might be time to move nnmairix to lisp/obsolete? Let's see
what Lars has to say about procedure/protocol.

Eric

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I think it might be time to move nnmairix to lisp/obsolete? Let's see
> what Lars has to say about procedure/protocol.

There's probably users of nnmairix out there, so I think obsoleting it
would be premature.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Eric Abrahamsen]

On 03/18/21 04:42 AM, Lars Ingebrigtsen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I think it might be time to move nnmairix to lisp/obsolete? Let's see
>> what Lars has to say about procedure/protocol.
>
> There's probably users of nnmairix out there, so I think obsoleting it
> would be premature.

Fair enough!

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> My only advice is to not use nnmairix (which no one is maintaining), and
> just use mairix as a regular backend to gnus-search (which should do
> everything that nnmairix ever did).

Seems to be cool - I tried it, but didn't get it work.  I tried to
search my (locally) archived messages with G G and mairix.

This is my mairixrc:

| base=~/Mail/archive
| mfolder=/home/micha/mairix
| mh=...
| mformat=mh
| omit=zz_mairix-*
| database=~/.mairixdatabase

and I use this config for Gnus:

#+begin_src emacs-lisp
(setq gnus-search-mairix-remove-prefix
      (expand-file-name "mairix/" "~")) ; correct?
(setf (alist-get 'nnml gnus-search-default-engines)
      'gnus-search-mairix)
#+end_src

When searching with G G mairix seems to be called; I get a prompt asking
how many of a plausible varying number of articles I want to see in the
created group; I hit RET to accept the default and Gnus just says

  "No unread news"

and that's it.  Am I doing something wrong?

TIA,

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> My only advice is to not use nnmairix (which no one is maintaining), and
>> just use mairix as a regular backend to gnus-search (which should do
>> everything that nnmairix ever did).
>
> Seems to be cool - I tried it, but didn't get it work.  I tried to
> search my (locally) archived messages with G G and mairix.
>
> This is my mairixrc:
>
> | base=~/Mail/archive
> | mfolder=/home/micha/mairix
> | mh=...
> | mformat=mh
> | omit=zz_mairix-*
> | database=~/.mairixdatabase
>
> and I use this config for Gnus:
>
> #+begin_src emacs-lisp
> (setq gnus-search-mairix-remove-prefix
>       (expand-file-name "mairix/" "~")) ; correct?
> (setf (alist-get 'nnml gnus-search-default-engines)
>       'gnus-search-mairix)
> #+end_src
>
> When searching with G G mairix seems to be called; I get a prompt asking
> how many of a plausible varying number of articles I want to see in the
> created group; I hit RET to accept the default and Gnus just says
>
>   "No unread news"
>
> and that's it.  Am I doing something wrong?

So maybe you're running into the exact same bug as before, via a
different path? :)

I don't have mairix installed and haven't used it before, let me get
things set up and see if I can reproduce the error, using your config etc.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> So maybe you're running into the exact same bug as before, via a
> different path? :)

Yes, seems so...

> I don't have mairix installed and haven't used it before, let me get
> things set up and see if I can reproduce the error, using your config
> etc.

Great, thanks.  I hope it's not my fault at the end.  Tell me if you
have any question.

Setting up mairix is easy.  You only need to save the .mairixrc.  Note
that the "mfolder" path is not ~-exanded, however, must be
/home/user/etc.  "base" is where you stored mails live.

Then run the "mairix" binary and see if it creates an index.  If you
don't have thousands of messages that will terminate more or less
instantly.  That's already all.

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Eric Abrahamsen]

On 03/16/21 05:03 AM, Michael Heerdegen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> So maybe you're running into the exact same bug as before, via a
>> different path? :)
>
> Yes, seems so...
>
>> I don't have mairix installed and haven't used it before, let me get
>> things set up and see if I can reproduce the error, using your config
>> etc.
>
> Great, thanks.  I hope it's not my fault at the end.  Tell me if you
> have any question.
>
> Setting up mairix is easy.  You only need to save the .mairixrc.  Note
> that the "mfolder" path is not ~-exanded, however, must be
> /home/user/etc.  "base" is where you stored mails live.
>
> Then run the "mairix" binary and see if it creates an index.  If you
> don't have thousands of messages that will terminate more or less
> instantly.  That's already all.

Well this is embarrassing, I can't get it to do anything at all. I
dumped a bunch of messages in ~/Mail/archive, set up my .mairixrc like:

base=/home/eric/Mail/archive
mfolder=/home/eric/mairix
mh=...
mformat=mh
omit=zz_mairix-*
database=~/.mairixdatabase

I also created the mfolder directory in case mairix wasn't going to do
that for me (what's that actually for?). But running "mairix -v" tells
me "No new messages found", and the ~/.mairixdatabase isn't created.

My messages have numerical file names. Is there anything else I need to
set up?

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Well this is embarrassing, I can't get it to do anything at all. I
> dumped a bunch of messages in ~/Mail/archive [...]

It seems you also need to have a .overview file in an mh mail
directory, else mairix does not consider it as mh mail folder.  Dunno
where to get it from by hand - when storing messages with Gnus Gnus
creates it automatically.  Maybe you should do that.

Or use a different folder format, e.g. mbox.

> I also created the mfolder directory in case mairix wasn't going to do
> that for me (what's that actually for?).

That's the folder which mairix will fill with (symlinks to) matching
messages.  Should normally be created automatically I think.  I also
think that Gnus tells mairix to use a different directory anyway.

HTH,

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> So maybe you're running into the exact same bug as before, via a
> different path? :)

Tried my luck on debugging, but it's really hard for me.

What I can say is that `nnselect-retrieve-headers' does not retrieve any
headers but gives up instead.  I'm not sure if there is a problem with
paths or whatever.  While debugging, in one of 10 trials then the thing
suddenly worked (!), I could see the expected matches but I did not find
a pattern what was different at that moment.  And quitting the group
then gave the error

| gnus-list-of-unread-articles: Group nnml+archive:home.micha.Mail.archive..emacs couldn't be activated

Hope somebody knowing the code can reproduce the issue.

Regards,

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Eric Abrahamsen]

On 03/17/21 02:02 AM, Michael Heerdegen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> So maybe you're running into the exact same bug as before, via a
>> different path? :)
>
> Tried my luck on debugging, but it's really hard for me.
>
> What I can say is that `nnselect-retrieve-headers' does not retrieve any
> headers but gives up instead.  I'm not sure if there is a problem with
> paths or whatever.  While debugging, in one of 10 trials then the thing
> suddenly worked (!), I could see the expected matches but I did not find
> a pattern what was different at that moment.  And quitting the group
> then gave the error
>
> | gnus-list-of-unread-articles: Group
> nnml+archive:home.micha.Mail.archive..emacs couldn't be activated
>
> Hope somebody knowing the code can reproduce the issue.

I was finally able to get mairix to index a test server, and searching
worked just fine. One thing I realized afterwards is that I had set the
remove-prefix to the directory where the actual mails are kept
("/home/eric/Mail"), whereas it looks like you've set it to the
directory where mairix is supposed to put its search-result symlinks
("/home/micha/mairix").

This is the main difference between nnmairix and gnus-search with
mairix: nnmairix actually creates new groups and reads them directly,
gnus-search returns a list of results that refer to messages on the
original server. The "-r" option is passed to mairix, which tells it not
to create its own folders but return the original file names of matched
results -- that's why remove-prefix has to refer to the underlying
server.

Hope that makes sense. I suppose I should add a note to the
documentation?

Eric

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I was finally able to get mairix to index a test server, and searching
> worked just fine. One thing I realized afterwards is that I had set the
> remove-prefix to the directory where the actual mails are kept
> ("/home/eric/Mail"), whereas it looks like you've set it to the
> directory where mairix is supposed to put its search-result symlinks
> ("/home/micha/mairix").

Indeed: I changed that and now searching works well!  Many thanks, dunno
how long I would have had to continue debugging until I had found that.

> This is the main difference between nnmairix and gnus-search with
> mairix: nnmairix actually creates new groups and reads them directly,
> gnus-search returns a list of results that refer to messages on the
> original server. The "-r" option is passed to mairix, which tells it not
> to create its own folders but return the original file names of matched
> results -- that's why remove-prefix has to refer to the underlying
> server.
>
> Hope that makes sense.

Makes a lot of sense, yes.

> I suppose I should add a note to the documentation?

Yes, it would be good to make that clearer.

BTW, I didn't even understand why it is even necessary to provide that.
If Gnus gets absolute names from mairix, why isn't it able to resolve
them in this case?

Thanks,

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Eric Abrahamsen]

On 03/17/21 23:39 PM, Michael Heerdegen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I was finally able to get mairix to index a test server, and searching
>> worked just fine. One thing I realized afterwards is that I had set the
>> remove-prefix to the directory where the actual mails are kept
>> ("/home/eric/Mail"), whereas it looks like you've set it to the
>> directory where mairix is supposed to put its search-result symlinks
>> ("/home/micha/mairix").
>
> Indeed: I changed that and now searching works well!  Many thanks, dunno
> how long I would have had to continue debugging until I had found that.

You would have had to find your way to the definition of
`gnus-search-indexed-parse-output', implemented for the
`gnus-search-indexed' engine class. Mairix isn't mentioned anywhere
around there. Generic functions are awesome, but there can definitely be
a discoverability problem.

>> This is the main difference between nnmairix and gnus-search with
>> mairix: nnmairix actually creates new groups and reads them directly,
>> gnus-search returns a list of results that refer to messages on the
>> original server. The "-r" option is passed to mairix, which tells it not
>> to create its own folders but return the original file names of matched
>> results -- that's why remove-prefix has to refer to the underlying
>> server.
>>
>> Hope that makes sense.
>
> Makes a lot of sense, yes.
>
>> I suppose I should add a note to the documentation?
>
> Yes, it would be good to make that clearer.

I will add a note to the manual, and probably also some code comments at
the top of gnus-search.el, giving hints for bug-hunters. I can close
this bug report when that's done.

> BTW, I didn't even understand why it is even necessary to provide that.
> If Gnus gets absolute names from mairix, why isn't it able to resolve
> them in this case?

The local-indexed search code conflates the two issues of 1) where is
the file that represents a single search results, and 2) how do we get
from that file path to an article number that a Gnus server will
understand.

If you look at the code in the function mentioned above, you'll see the
whole process is surprisingly hacky. The "remove-prefix" is chopped off
the front of the filepath, then the next part of the filepath is
interpreted as a group name, then it attempts to turn the non-directory
filename into an article number somehow. If the remove-prefix is wrong,
the later code gets a full filepath and doesn't know what to do with it.

I think all nnmail-type servers have some config for "directory" or
"nnmh-directory" or whatever, and I think there's a good chance that the
remove-prefix can be made redundant with this config.

I'm a little surprised it works as well as it does!

Eric

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> > I suppose I should add a note to the documentation?

You have done that now - thanks.

I have another question: The docstrings of the *-remove-prefix options
say that "This is a regular expression.".  AFAIU that's not true: It's
just a string that is given to `string-remove-prefix'.  Correct?

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Eric Abrahamsen]

On 02/15/22 00:51 AM, Michael Heerdegen wrote:
> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
>> > I suppose I should add a note to the documentation?
>
> You have done that now - thanks.
>
> I have another question: The docstrings of the *-remove-prefix options
> say that "This is a regular expression.".  AFAIU that's not true: It's
> just a string that is given to `string-remove-prefix'.  Correct?

Good catch! I'll fix that now. This `getenv' stuff is bad, too...

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Good catch! I'll fix that now. This `getenv' stuff is bad, too...

Thanks.  Then, from my side this report can be closed when you are done
with that.

Regards,

Michael.

bug#47130: 28.0.50; Gnus: mairix doesn't work anymore

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Good catch! I'll fix that now. This `getenv' stuff is bad, too...
>
> Thanks.  Then, from my side this report can be closed when you are done
> with that.

Done! Thanks.