bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

I have not been able to get to the root of the following problem.  Yet I
thought I post it anyway as it is rather annoying.

I am fairly new to Gnus when I first used it with Emacs 27 (for about
half a year).  Since Emacs 28.0.91 came out, I have been using it and
this gives me the strange problem that Gnus downloads some mail messages
from the IMAP server multiple times without deleting them on the server
(typically three to five times) till Gnus (or the IMAP server?) decides
it has downloaded this message often enough.  These messages are marked
in Gnus as duplicates of the first instance of this message.  Yet
stranger, it only happens with spam emails including many emails that
are not marked as spam by the spam filter of the IMAP server.  But only
when I look into these emails I see that I do not want or need them.

First, I thought that something had changed on the IMAP server
(Office 365) that triggered this.  Then I switched back from Emacs
28.0.91 to Emacs 27.2, when the problem disappeared again.  Then I went
back to 28.0.91, when I am annoyed again by these duplicate messages.
So from all I can tell, it appears to be related to some difference
between how Emacs 27 and 28.0.91 fetch mail from the IMAP server.

Is anybody else seeing something similar?  Let me know what kind of
details might help to debug this further.  (I am not yet a Gnus expert
and yet less an expert of the IMAP protocol that may also play a role
here.)


In GNU Emacs 28.0.91 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.5, cairo version 1.16.0)
 of 2022-01-16 built on regnitz
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description: Debian GNU/Linux 10 (buster)

Configured using:
 'configure --with-native-compilation'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LC_COLLATE: C
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

Roland Winkler <winkler@gnu.org> writes:

> I have not been able to get to the root of the following problem.  Yet I
> thought I post it anyway as it is rather annoying.
>
> I am fairly new to Gnus when I first used it with Emacs 27 (for about
> half a year).  Since Emacs 28.0.91 came out, I have been using it and
> this gives me the strange problem that Gnus downloads some mail messages
> from the IMAP server multiple times without deleting them on the server
> (typically three to five times) till Gnus (or the IMAP server?) decides
> it has downloaded this message often enough.  These messages are marked
> in Gnus as duplicates of the first instance of this message.  Yet
> stranger, it only happens with spam emails including many emails that
> are not marked as spam by the spam filter of the IMAP server.  But only
> when I look into these emails I see that I do not want or need them.

You're using the IMAP server as a mail source, right? Can you post your
value of `mail-sources'?

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

On Fri, Feb 25 2022, Eric Abrahamsen wrote:
> You're using the IMAP server as a mail source, right? Can you post
> your value of `mail-sources'?

mail-sources is

       ((imap :server "localhost"
              :port 1143
              :user "winkler@foo.com"
              :authentication 'login
              :predicate "1:*"
              :mailbox ("INBOX" "JUNK EMAIL"))
        (pop :server "fencepost.gnu.org"
             :port 995
             :user "winkler"
             :authentication 'apop))
 
Server is "localhost" because I am using davmail in between emacs and
the actual IMAP server (Office 365).  There is no problem with
fencepost.

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

Roland Winkler <winkler@gnu.org> writes:

> On Fri, Feb 25 2022, Eric Abrahamsen wrote:
>> You're using the IMAP server as a mail source, right? Can you post
>> your value of `mail-sources'?
>
> mail-sources is
>
>        ((imap :server "localhost"
>               :port 1143
>               :user "winkler@foo.com"
>               :authentication 'login
>               :predicate "1:*"
>               :mailbox ("INBOX" "JUNK EMAIL"))

I went down a bit of a rabbit hole here, and my best guess is that
commit daa4e0120 (which is in emacs-28 but not emacs-27) might have
resulted in a bug in the dynamic binding of your imap mail-source
definition data.

More specifically, the body of `mail-source-fetch-imap' is wrapped in
`mail-source-bind', which perpetuates some voodoo to locally bind all of
the data in your definition to local variables (e.g. :mailbox turns into
the variable `mailbox'). `mail-source-fetch-imap' hasn't changed much,
but `mail-source-bind' has, so maybe something's going wrong in the
binding of imap source data?

Something to try would be to edebug `mail-source-fetch-imap' and make
sure that the dynamic variables `dontexpunge' and `fetchflag' are what
you'd expect them to be, ie nil and "\Deleted". And watch what happens
for both "INBOX" and "JUNK EMAIL" (why are you fetching your junk mail,
anyway?) when you get to this bit of the code:

(when (and remove fetchflag)
  (setq remove (nreverse remove))
  (imap-message-flags-add
   (imap-range-to-message-set (gnus-compress-sequence remove))
   fetchflag nil buf))
(if dontexpunge
    (imap-mailbox-unselect buf)
  (imap-mailbox-close nil buf))

To be honest I don't see why `imap-mailbox-close' would expunge mail
anyway: its docstring says it does, but no expunge command is given.
There's a `imap-mailbox-expunge' function, but nothing ever calls it.
Anyway, that part of the code hasn't changed since emacs-27, so <shrug>.

If something funny is happening with the values of fetchflag or
dontexpunge, we can drag in Stefan M, who changed `mail-source-bind'.

Eric

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

On Fri, Feb 25 2022, Eric Abrahamsen wrote:
> I went down a bit of a rabbit hole here, and my best guess is that
> commit daa4e0120 (which is in emacs-28 but not emacs-27) might have
> resulted in a bug in the dynamic binding of your imap mail-source
> definition data.

Thank you!  Unfortunately, I am very busy the next couple of days.
Later this week I'll try to debug what you have suggested.

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

On Fri, Feb 25 2022, Eric Abrahamsen wrote:
> I went down a bit of a rabbit hole here, and my best guess is that
> commit daa4e0120 (which is in emacs-28 but not emacs-27) might have
> resulted in a bug in the dynamic binding of your imap mail-source
> definition data.

...First I thought "let me revert this patch and see what happens", but
that's not a meaningful step.

> Something to try would be to edebug `mail-source-fetch-imap' and make
> sure that the dynamic variables `dontexpunge' and `fetchflag' are what
> you'd expect them to be, ie nil and "\Deleted". And watch what happens
> for both "INBOX" and "JUNK EMAIL"

The values of these variables are always what you said they should be,
nil and "\Deleted".

> (why are you fetching your junk mail, anyway?)

It's the usual problem that the folder "JUNK EMAIL" contains messages
that should not be there.  And I haven't figured out how the server
identifies mail as spam, nor whether it would allow me to fine-tune
this.  I find it easier to do all this on the client side.

> To be honest I don't see why `imap-mailbox-close' would expunge mail
> anyway: its docstring says it does, but no expunge command is given.
> There's a `imap-mailbox-expunge' function, but nothing ever calls it.
> Anyway, that part of the code hasn't changed since emacs-27, so
> <shrug>.

I looked at this code.  Unfortunately, the wrappers it uses make it very
difficult to understand what it is doing, also because I do not know
either what it needs to do to get things right.

> If something funny is happening with the values of fetchflag or
> dontexpunge, we can drag in Stefan M, who changed `mail-source-bind'.

Following your suggestions of monitoring the variables `dontexpunge' and
`fetchflag', I do not see anything unusual in mail-source-fetch-imap.

My understanding is that IMAP servers permit to expunge individual
messages while keeping others, though I do not want to use this feature
(I want to expunge everything after fetching it).  However, I do not see
where mail-source-fetch-imap would implement such fine control.  But my
problem of multiple downloads exists only for *some*, but not all
messages.  So it seems to me we may be looking at the wrong part of the
code altogether.  But that's only a wild guess of someone who does not
know much about all of this.

Any help is appreciated!

Roland

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

Roland Winkler <winkler@gnu.org> writes:

> On Fri, Feb 25 2022, Eric Abrahamsen wrote:
>> I went down a bit of a rabbit hole here, and my best guess is that
>> commit daa4e0120 (which is in emacs-28 but not emacs-27) might have
>> resulted in a bug in the dynamic binding of your imap mail-source
>> definition data.
>
> ...First I thought "let me revert this patch and see what happens", but
> that's not a meaningful step.
>
>> Something to try would be to edebug `mail-source-fetch-imap' and make
>> sure that the dynamic variables `dontexpunge' and `fetchflag' are what
>> you'd expect them to be, ie nil and "\Deleted". And watch what happens
>> for both "INBOX" and "JUNK EMAIL"
>
> The values of these variables are always what you said they should be,
> nil and "\Deleted".
>
>> (why are you fetching your junk mail, anyway?)
>
> It's the usual problem that the folder "JUNK EMAIL" contains messages
> that should not be there.  And I haven't figured out how the server
> identifies mail as spam, nor whether it would allow me to fine-tune
> this.  I find it easier to do all this on the client side.
>
>> To be honest I don't see why `imap-mailbox-close' would expunge mail
>> anyway: its docstring says it does, but no expunge command is given.
>> There's a `imap-mailbox-expunge' function, but nothing ever calls it.
>> Anyway, that part of the code hasn't changed since emacs-27, so
>> <shrug>.
>
> I looked at this code.  Unfortunately, the wrappers it uses make it very
> difficult to understand what it is doing, also because I do not know
> either what it needs to do to get things right.
>
>> If something funny is happening with the values of fetchflag or
>> dontexpunge, we can drag in Stefan M, who changed `mail-source-bind'.
>
> Following your suggestions of monitoring the variables `dontexpunge' and
> `fetchflag', I do not see anything unusual in mail-source-fetch-imap.

My intuitive shots-in-the-dark are pretty much always wrong :(

> My understanding is that IMAP servers permit to expunge individual
> messages while keeping others, though I do not want to use this feature
> (I want to expunge everything after fetching it).  However, I do not see
> where mail-source-fetch-imap would implement such fine control.  But my
> problem of multiple downloads exists only for *some*, but not all
> messages.  So it seems to me we may be looking at the wrong part of the
> code altogether.  But that's only a wild guess of someone who does not
> know much about all of this.

I don't know that much, either. I'll set up a local IMAP account and do
some testing with this, but it will take me a week or so to get to it.
Can you tell me what IMAP server you're talking to here?

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

On Thu, Mar 10 2022, Eric Abrahamsen wrote:
>> My understanding is that IMAP servers permit to expunge individual
>> messages while keeping others, though I do not want to use this feature
>> (I want to expunge everything after fetching it).  However, I do not see
>> where mail-source-fetch-imap would implement such fine control.  But my
>> problem of multiple downloads exists only for *some*, but not all
>> messages.  So it seems to me we may be looking at the wrong part of the
>> code altogether.  But that's only a wild guess of someone who does not
>> know much about all of this.
>
> I don't know that much, either. I'll set up a local IMAP account and do
> some testing with this, but it will take me a week or so to get to it.
> Can you tell me what IMAP server you're talking to here?

The IMAP server is Office365 that's accessed via a local davmail (for
Office365 interactive authentication).

I was always surprised that the duplicate emails are "mostly spam".
Previously, I sent you my settting of `mail-sources'.  I changed the
line

  :mailbox ("INBOX" "JUNK EMAIL")

to

  :mailbox ("JUNK EMAIL" "INBOX")

Strange enough, I didn't observe any duplicate message downloads when I
tested this for a few days.

To be more specific: if I understand the office365 spam filter
correctly, it puts messages quite generously into the "JUNK EMAIL"
folder, which is why I prefer to review these messages locally.  But
these messages in "JUNK EMAIL" are tagged as spam more or less
aggressively as spam.  Also, I believe gnus won't tell me whether it
downloaded a message from the INBOX or "JUNK EMAIL" folder.  That's why
I cannot tell with certainty for some of the duplicate downloads from
which folder on the server they originated.  But it seems that all my
duplicate messages are coming from the "JUNK EMAIL" folder on the IMAP
server.

I believe that "INBOX" is really the default folder on an IMAP server
that also appears in some gnus defaults.  Then, adding another folder to
:mailbox may be less common.  So my problems arise only in what may be a
less frequent use case.  Certainly, this becomes yet more obscure if it
matters whether a second folder is listed in :mailbox before or after
the default "INBOX".

All the above is describing symptoms.  The root of the problem remains
unclear to me.

Roland

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

On Sat, Mar 12 2022, Roland Winkler wrote:
> I was always surprised that the duplicate emails are "mostly spam".
> Previously, I sent you my settting of `mail-sources'.  I changed the
> line
>
>   :mailbox ("INBOX" "JUNK EMAIL")
>
> to
>
>   :mailbox ("JUNK EMAIL" "INBOX")
>
> Strange enough, I didn't observe any duplicate message downloads when I
> tested this for a few days.

One more observation:

The bug appears to be related to :mailbox being a list of folders.
I just installed a rule on the IMAP server that puts part of my regular
incoming mail in another folder FOO.  Now downloading the messages from
the IMAP folder FOO into Gnus gives me more duplicates.  Any IMAP folder
beyond INBOX appears to show this behavior.

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

On 03/18/22 11:09 AM, Roland Winkler wrote:
> On Sat, Mar 12 2022, Roland Winkler wrote:
>> I was always surprised that the duplicate emails are "mostly spam".
>> Previously, I sent you my settting of `mail-sources'.  I changed the
>> line
>>
>>   :mailbox ("INBOX" "JUNK EMAIL")
>>
>> to
>>
>>   :mailbox ("JUNK EMAIL" "INBOX")
>>
>> Strange enough, I didn't observe any duplicate message downloads when I
>> tested this for a few days.
>
> One more observation:
>
> The bug appears to be related to :mailbox being a list of folders.
> I just installed a rule on the IMAP server that puts part of my regular
> incoming mail in another folder FOO.  Now downloading the messages from
> the IMAP folder FOO into Gnus gives me more duplicates.  Any IMAP folder
> beyond INBOX appears to show this behavior.

Okay, that's helpful, thanks. I'll try to concentrate on finding
differences in behavior between a single :mailbox and multiple.

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

On Fri, Mar 18 2022, Eric Abrahamsen wrote:
> Okay, that's helpful, thanks. I'll try to concentrate on finding
> differences in behavior between a single :mailbox and multiple.

One more thought:

My Gnus talks to the imap server on localhost (davmail).  Is it possible
to look at this communication?  Then I could try to find out in which
way Emacs 27 is communicating differently with the imap server compared
with Emacs 28.

(I just convinced myself once more that the problems with duplicate mail
downloads for multiple folders on the IMAP server go away when I use
Emacs 27.)

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Robert Pluim]

>>>>> On Fri, 18 Mar 2022 11:34:25 -0500, Roland Winkler <winkler@gnu.org> said:

    Roland> On Fri, Mar 18 2022, Eric Abrahamsen wrote:
    >> Okay, that's helpful, thanks. I'll try to concentrate on finding
    >> differences in behavior between a single :mailbox and multiple.

    Roland> One more thought:

    Roland> My Gnus talks to the imap server on localhost (davmail).  Is it possible
    Roland> to look at this communication?  Then I could try to find out in which
    Roland> way Emacs 27 is communicating differently with the imap server compared
    Roland> with Emacs 28.

You mean (setq imap-log t)?

Robert
-- 

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

On Fri, Mar 18 2022, Robert Pluim wrote:
> You mean (setq imap-log t)?

YES, thank you!  So finally I can be more specific:

(1) When the message from Imap folder FOO is not properly deleted on the
    server, the log buffer says (Emacs 28)

11 OK UID FETCH completed%d
12 UID STORE 135750,27 +FLAGS (\Deleted)%d
12 OK STORE completed%d
13 CLOSE%d
13 OK CLOSE completed%d

(2) When Emacs 28 does properly delete the message in Imap folder FOO,
    the log buffer says

9 OK UID FETCH completed%d
10 UID STORE 27 +FLAGS (\Deleted)%d
* 1 FETCH (UID 27 FLAGS (\Deleted))%d
10 OK STORE completed%d
11 CLOSE%d
11 OK CLOSE completed%d

The details are interesting:

(1) I got the first snippet when Emacs 28 first downloaded and properly
    deleted a message from INBOX (this message had UID 135750), then
    things failed as shown for the message downloaded from folder FOO.

    It seems the UID of the message from INBOX reappeared when Gnus
    wanted to download the message from FOO.

(2) In the second IMAP session, there was only the message in FOO and
    nothing in INBOX.  Then Emacs 28 properly downloaded and deleted the
    message from folder FOO.

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

On 03/18/22 11:34 AM, Roland Winkler wrote:
> On Fri, Mar 18 2022, Eric Abrahamsen wrote:
>> Okay, that's helpful, thanks. I'll try to concentrate on finding
>> differences in behavior between a single :mailbox and multiple.
>
> One more thought:
>
> My Gnus talks to the imap server on localhost (davmail).  Is it possible
> to look at this communication?  Then I could try to find out in which
> way Emacs 27 is communicating differently with the imap server compared
> with Emacs 28.
>
> (I just convinced myself once more that the problems with duplicate mail
> downloads for multiple folders on the IMAP server go away when I use
> Emacs 27.)

You can set `nnimap-record-commands' non-nil, and should get a buffer
containing the client-server conversation to look at.

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> On 03/18/22 11:34 AM, Roland Winkler wrote:
>> On Fri, Mar 18 2022, Eric Abrahamsen wrote:
>>> Okay, that's helpful, thanks. I'll try to concentrate on finding
>>> differences in behavior between a single :mailbox and multiple.
>>
>> One more thought:
>>
>> My Gnus talks to the imap server on localhost (davmail).  Is it possible
>> to look at this communication?  Then I could try to find out in which
>> way Emacs 27 is communicating differently with the imap server compared
>> with Emacs 28.
>>
>> (I just convinced myself once more that the problems with duplicate mail
>> downloads for multiple folders on the IMAP server go away when I use
>> Emacs 27.)
>
> You can set `nnimap-record-commands' non-nil, and should get a buffer
> containing the client-server conversation to look at.

Sorry, this is only for nnimap servers. Robert's suggestion is the right
one (imap-log no longer exists, but I think it's still there in Emacs 28.1?)

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> If something funny is happening with the values of fetchflag or
> dontexpunge, we can drag in Stefan M, who changed `mail-source-bind'.

FWIW, I just took a look at the macroexpanded form of
`mail-source-fetch-imap` and my "fishiness" detector didn't notice
anything out of the expected.

I don't understand the code enough to detect if there's something wrong,
but at least the new expanded code looks the way I wanted to make it
look (e.g. `dontexpunge` is indeed dynamically bound inside this
function).


        Stefan

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

On 03/18/22 11:27 AM, Stefan Monnier wrote:
>> If something funny is happening with the values of fetchflag or
>> dontexpunge, we can drag in Stefan M, who changed `mail-source-bind'.
>
> FWIW, I just took a look at the macroexpanded form of
> `mail-source-fetch-imap` and my "fishiness" detector didn't notice
> anything out of the expected.
>
> I don't understand the code enough to detect if there's something wrong,
> but at least the new expanded code looks the way I wanted to make it
> look (e.g. `dontexpunge` is indeed dynamically bound inside this
> function).

Yeah, my "hunches" are wrong almost 100% of the time. It's too bad,
though, because otherwise I don't have the faintest idea where to start
debugging this. I was just going to set up a local IMAP account to use
as a mail source, and hope that something presented itself.

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Andrew Cohen]

(I sent this to Roland directly since I'm a bit shy about communicating
on public lists, but here goes anyway)

I think the problem is that the code in mail-source-fetch-imap doesn't
handle a list of mailboxes properly (I don't know why it would have
worked in emacs-27 though). Adam added a loop over a list of mailboxes
way back in 2015, but it seems to me that some of the variables bound
outside the loop should be bound inside (imap usually works on a per
mailbox basis)

So I think this might fix it (although I don't have a setup available to
test it).

Of course I might be totally off-base. If so, sorry in advance. 

diff --git a/lisp/gnus/mail-source.el b/lisp/gnus/mail-source.el
index 5d0c0e2654..01cca831f2 100644
--- a/lisp/gnus/mail-source.el
+++ b/lisp/gnus/mail-source.el
@@ -1066,9 +1066,7 @@ mail-source-fetch-imap
     (let ((from (format "%s:%s:%s" server user port))
 	  (found 0)
 	  (buf (generate-new-buffer " *imap source*"))
-	  (mail-source-string (format "imap:%s:%s" server mailbox))
-	  (imap-shell-program (or (list program) imap-shell-program))
-	  remove)
+	  (imap-shell-program (or (list program) imap-shell-program)))
       (if (and (imap-open server port stream authentication buf)
 	       (imap-authenticate
 		user (or (cdr (assoc from mail-source-password-cache))
@@ -1078,7 +1076,8 @@ mail-source-fetch-imap
             (dolist (mailbox mailbox-list)
               (when (imap-mailbox-select mailbox nil buf)
 	  (let ((coding-system-for-write mail-source-imap-file-coding-system)
-		str)
+                (mail-source-string (format "imap:%s:%s" server mailbox))
+		str remove)
             (message "Fetching from %s..." mailbox)
 	    (with-temp-file mail-source-crash-box
 	      ;; Avoid converting 8-bit chars from inserted strings to


-- 
Andrew Cohen

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

On Sat, Mar 19 2022, Andrew Cohen wrote:
> I think the problem is that the code in mail-source-fetch-imap doesn't
> handle a list of mailboxes properly (I don't know why it would have
> worked in emacs-27 though). Adam added a loop over a list of mailboxes
> way back in 2015, but it seems to me that some of the variables bound
> outside the loop should be bound inside (imap usually works on a per
> mailbox basis)
>
> So I think this might fix it (although I don't have a setup available
> to test it).

Thank you!  I can confirm: I identified one scenario with emails I send
to myself such that they go into different folders on the IMAP server.
Then downloading these messages with Gnus from Emacs 28.0.91 gives me
reproducibly the duplicate mail downloads as described in the original
post.  When I apply Andrew's patch to `mail-source-fetch-imap' the
downloads happen as expected, properly deleting the messages on the
server after download, as confirmed also with imap-log bound to to.
So for this one test case the patch appears to solve the problem.

The patch may require more testing and also checking the code by someone
who knows what is supposed to happen in such an imap session.  I am
surprised that `mail-source-fetch-imap' in Gnus from Emacs 27.2 and from
Emacs 28.0.91 are essentially the same (from all I can tell).  So I am
wondering why `mail-source-fetch-imap' in Gnus from Emacs 27.2 was
behaving as expected; but the bug showed up only in Emacs 28.  Can this
be in subtle ways related to the fact that mail-source.el in Emacs 27
used dynamic binding, whereas in Emacs 28 it uses lexical binding, as
speculated previously by Eric?

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

Roland Winkler <winkler@gnu.org> writes:

> On Sat, Mar 19 2022, Andrew Cohen wrote:
>> I think the problem is that the code in mail-source-fetch-imap doesn't
>> handle a list of mailboxes properly (I don't know why it would have
>> worked in emacs-27 though). Adam added a loop over a list of mailboxes
>> way back in 2015, but it seems to me that some of the variables bound
>> outside the loop should be bound inside (imap usually works on a per
>> mailbox basis)
>>
>> So I think this might fix it (although I don't have a setup available
>> to test it).
>
> Thank you!  I can confirm: I identified one scenario with emails I send
> to myself such that they go into different folders on the IMAP server.
> Then downloading these messages with Gnus from Emacs 28.0.91 gives me
> reproducibly the duplicate mail downloads as described in the original
> post.  When I apply Andrew's patch to `mail-source-fetch-imap' the
> downloads happen as expected, properly deleting the messages on the
> server after download, as confirmed also with imap-log bound to to.
> So for this one test case the patch appears to solve the problem.
>
> The patch may require more testing and also checking the code by someone
> who knows what is supposed to happen in such an imap session.  I am
> surprised that `mail-source-fetch-imap' in Gnus from Emacs 27.2 and from
> Emacs 28.0.91 are essentially the same (from all I can tell).  So I am
> wondering why `mail-source-fetch-imap' in Gnus from Emacs 27.2 was
> behaving as expected; but the bug showed up only in Emacs 28.  Can this
> be in subtle ways related to the fact that mail-source.el in Emacs 27
> used dynamic binding, whereas in Emacs 28 it uses lexical binding, as
> speculated previously by Eric?

FWIW this patch looks right to me, thanks to Andy for figuring it out.
Is someone going to push this? It would be nice at the same time to fix
the indentation of the mailbox loop so it's clearer what's actually
happening.

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> FWIW this patch looks right to me, thanks to Andy for figuring it out.
> Is someone going to push this? It would be nice at the same time to fix
> the indentation of the mailbox loop so it's clearer what's actually
> happening.

I think Andrew pushed it in 01336a2582?  (To the trunk.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

On 03/21/22 20:38 PM, Lars Ingebrigtsen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> FWIW this patch looks right to me, thanks to Andy for figuring it out.
>> Is someone going to push this? It would be nice at the same time to fix
>> the indentation of the mailbox loop so it's clearer what's actually
>> happening.
>
> I think Andrew pushed it in 01336a2582?  (To the trunk.)

Oh! I didn't look far enough back. Didn't we want it in 28.1, too?

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Oh! I didn't look far enough back. Didn't we want it in 28.1, too?

Since this didn't work in Emacs 27, either, it's not a regression, so in
Emacs 29 it goes.  😀

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

On 03/21/22 20:50 PM, Lars Ingebrigtsen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Oh! I didn't look far enough back. Didn't we want it in 28.1, too?
>
> Since this didn't work in Emacs 27, either, it's not a regression, so in
> Emacs 29 it goes.  😀

Spoken like a true maintainer :)

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

On Mon, Mar 21 2022, Eric Abrahamsen wrote:
> It would be nice at the same time to fix the indentation of the
> mailbox loop so it's clearer what's actually happening.

Much agreed!  Can you do that, Eric?

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Eric Abrahamsen]

Roland Winkler <winkler@gnu.org> writes:

> On Mon, Mar 21 2022, Eric Abrahamsen wrote:
>> It would be nice at the same time to fix the indentation of the
>> mailbox loop so it's clearer what's actually happening.
>
> Much agreed!  Can you do that, Eric?

Sure, will do in a bit.

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Andrew Cohen]

On Sat, Mar 19 2022, Roland Winkler wrote:
>On Sat, Mar 19 2022, Andrew Cohen wrote:

[...]

>> So I think this might fix it (although I don't have a setup available
>> to test it).

> Thank you!  I can confirm: I identified one scenario with emails I send
>to myself such that they go into different folders on the IMAP server.
>Then downloading these messages with Gnus from Emacs 28.0.91 gives me
>reproducibly the duplicate mail downloads as described in the original
>post.  When I apply Andrew's patch to `mail-source-fetch-imap' the
>downloads happen as expected, properly deleting the messages on the
>server after download, as confirmed also with imap-log bound to to.
>So for this one test case the patch appears to solve the problem.

Great! 

>The patch may require more testing and also checking the code by someone
>who knows what is supposed to happen in such an imap session.  

More testing would be good---but I suspect the number of users of a list
of mailboxes with imap is small :) Fortunately the patch is quite minor
and I'm pretty confident that this change is the right thing.

My patch also fixes another bug---the header identifying the originating
mailbox was getting set improperly (to the list of mailboxes rather than
the actual mailbox).

>I am surprised that `mail-source-fetch-imap' in Gnus from Emacs 27.2
>and from Emacs 28.0.91 are essentially the same (from all I can tell).
>So I am wondering why `mail-source-fetch-imap' in Gnus from Emacs 27.2
>was behaving as expected; but the bug showed up only in Emacs 28.  Can
>this be in subtle ways related to the fact that mail-source.el in Emacs
>27 used dynamic binding, whereas in Emacs 28 it uses lexical binding,
>as speculated previously by Eric?

This is indeed a mystery. If you are excited about getting to the bottom
of it you could try enabling the log with emacs-27 and post the result
of your scenario. The problem was the variable 'remove was accumulating
UIDs from prior iterations of the loop. 



-- 
Andrew Cohen

bug#54158: 28.0.91; duplicate mail downloads in Gnus (IMAP)

[Roland Winkler]

On Sun, Mar 20 2022, Andrew Cohen wrote:
>>I am surprised that `mail-source-fetch-imap' in Gnus from Emacs 27.2
>>and from Emacs 28.0.91 are essentially the same (from all I can tell).
>>So I am wondering why `mail-source-fetch-imap' in Gnus from Emacs 27.2
>>was behaving as expected; but the bug showed up only in Emacs 28.  Can
>>this be in subtle ways related to the fact that mail-source.el in Emacs
>>27 used dynamic binding, whereas in Emacs 28 it uses lexical binding,
>>as speculated previously by Eric?
>
> This is indeed a mystery. If you are excited about getting to the bottom
> of it you could try enabling the log with emacs-27 and post the result
> of your scenario. The problem was the variable 'remove was accumulating
> UIDs from prior iterations of the loop. 

I am perplexed: Certainly, I struggled for the longest time to identify
a reproducible recipe for this bug till I learned about the variable
imap-log (see my earlier posts in this thread).  Indeed, with this
recipe the bug does show up both with Emacs 28.0.91 and Emacs 27.2.  So
I need to take back what I said before that this bug is a regression.

Reading the code of `mail-source-fetch-imap' more carefully, I agree
with you: it would be a mystery if the code had ever worked as intended.

A possible explanation from my side is: while I started using Gnus about
mid of last year when I had Emacs 27.2, I added a second IMAP folder to
my `mail-sources' configuration more recently, possibly around the time
when I switched from 27.2 to 28.0.91.  Then I got fooled by this bug
because I didn't know its actual cause.  While I went back temporarily
from 28.0.91 to 27.2 to see whether this changed anything, I may have
done that for too short an amount of time to run into the particular
situation that triggers this bug.  Oh well.

Thanks to everyone who has helped to find the cause of this bug!