bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Ulrich Mueller]

In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit)
 of 2019-04-21 built on a1i15
Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
System Description: Gentoo/Linux

When fetching e-mail from a dovecot-2.3.5.1 IMAP server, Gnus/nnimap
gets confused and displays ghost messages with address "nobody" and
subject "(none)" in the summary buffer, like in this example:

*Summary nnimap+dev.gentoo.org:INBOX*
----------------------------------------------------------------------
R. [   1: Ulrich Mueller         ] test 1
 . [   ?: nobody                 ] (none)
 . [   1: Ulrich Mueller         ] test 2
----------------------------------------------------------------------

My gnus-secondary-select-methods looks like this:

      '((nnimap "dev.gentoo.org"
		(nnimap-stream tls)
		(nnimap-user "ulm"))
	;; ... further methods omitted ...
	)

For debugging, I have enabled logging with nnimap-record-commands, and
set debug-on-entry for function nnimap-transform-headers.

I then get the following "*imap log*" buffer:
----------------------------------------------------------------------
07:41:57 [dev.gentoo.org] (inhibited)
07:41:59 [dev.gentoo.org] 10105 CAPABILITY
07:41:59 [dev.gentoo.org] 10106 ENABLE QRESYNC
07:41:59 [dev.gentoo.org] 10107 LIST "" "*"
[... other servers ...]
07:42:00 [dev.gentoo.org] 10147 EXAMINE "INBOX" (QRESYNC (1364468255 63360))
[... other servers ...]
07:42:08 [dev.gentoo.org] 10192 SELECT "INBOX"
07:42:08 [dev.gentoo.org] 10193 SELECT "INBOX"
07:42:08 [dev.gentoo.org] 10194 UID FETCH 32409:32410 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Diary-Time-Zone X-Diary-Dow X-Diary-Year X-Diary-Month X-Diary-Dom X-Diary-Hour X-Diary-Minute To Newsgroups Cc)])
----------------------------------------------------------------------

Buffer " *nnimap dev.gentoo.org nil  *nntpd**" looks like this, upon
entering nnimap-transform-headers:
----------------------------------------------------------------------
* 583 FETCH (UID 32409 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165}
From: Ulrich Mueller <ulm@gentoo.org>
To: ulm@gentoo.org
Subject: test 1
Date: Sat, 27 Apr 2019 07:39:56 +0200
Message-ID: <w6gd0l8vwer.fsf@kph.uni-mainz.de>

)
* 584 FETCH (UID 32410 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165}
From: Ulrich Mueller <ulm@gentoo.org>
To: ulm@gentoo.org
Subject: test 2
Date: Sat, 27 Apr 2019 07:40:15 +0200
Message-ID: <w6g8svwvwe8.fsf@kph.uni-mainz.de>

)
* 583 FETCH (UID 32409 MODSEQ (63364) FLAGS ($HasNoAttachment))
* 584 FETCH (UID 32410 MODSEQ (63364) FLAGS ($HasNoAttachment))
10194 OK Fetch completed (0.003 + 0.000 + 0.002 secs).
----------------------------------------------------------------------

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

Ulrich Mueller <ulm@gentoo.org> writes:

> In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit)
>  of 2019-04-21 built on a1i15
> Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
> System Description: Gentoo/Linux
>
> When fetching e-mail from a dovecot-2.3.5.1 IMAP server, Gnus/nnimap
> gets confused and displays ghost messages with address "nobody" and
> subject "(none)" in the summary buffer, like in this example:

Are you able to build an earlier version of Emacs (say, from a couple of
months ago) and test if you see the same thing? I made some fairly deep
changes to Gnus over the past month, and while I don't see any reason
why that would cause this behavior, it would be nice to rule it out.

Thanks,
Eric

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Ulrich Mueller]

>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:

> Are you able to build an earlier version of Emacs (say, from a couple
> of months ago) and test if you see the same thing? I made some fairly
> deep changes to Gnus over the past month, and while I don't see any
> reason why that would cause this behavior, it would be nice to rule it
> out.

Yes, I had tried with Emacs 26.2 and it has the same problem.

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

Ulrich Mueller <ulm@gentoo.org> writes:

>>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:
>
>> Are you able to build an earlier version of Emacs (say, from a couple
>> of months ago) and test if you see the same thing? I made some fairly
>> deep changes to Gnus over the past month, and while I don't see any
>> reason why that would cause this behavior, it would be nice to rule it
>> out.
>
> Yes, I had tried with Emacs 26.2 and it has the same problem.

Huh. Can you show us the value of gnus-newsgroup-data and
gnus-newsgroup-headers in this group? Does the dummy article always show
up? Only in this group? Is this a newly-added IMAP server, and did it
display this problem right from the beginning?

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Ulrich Mueller]

>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:

> Huh. Can you show us the value of gnus-newsgroup-data and
> gnus-newsgroup-headers in this group?

See below, for the current group, whose summary buffer shows your
article and a dummy article.

> Does the dummy article always show up?

It always shows up when new articles have been fetched. It doesn't
show up when entering a group that doesn't have new mail (i.e., if a
group previously had a dummy article, it will be gone when reentering
that group).

> Only in this group?

All groups for that particular IMAP server. 

> Is this a newly-added IMAP server, and did it display this problem
> right from the beginning?

AFAICS, the problem appeared after dovecot on the server side was
upgraded from version 2.2.34 to 2.3.5.1. But I see it only with Gnus.
I don't have any issues with Mozilla Thunderbird (on GNU/Linux) nor with
K-9 Mail (on Android/Linux).

----------------------------------------------------------------------
gnus-newsgroup-data is a variable defined in ‘gnus-sum.el’.
Its value is shown below.

Documentation:
Not documented as a variable.

Value:
((32415 82 2
	[32415 "Re: bug#35443: 27.0.50; Gnus (nnimap) shows \"ghost\" messages in summary buffer" "Eric Abrahamsen <eric@ericabrahamsen.net>" "Sat, 27 Apr 2019 13:47:18 -0700" "<87h8ajjhux.fsf@ericabrahamsen.net>" "<w6g1s1ovuew.fsf@kph.uni-mainz.de> <87wojfjxry.fsf@ericabrahamsen.net> <w6gwojfo3cy.fsf@kph.uni-mainz.de>" 2609 16 nil
	       ((Cc . "35443@debbugs.gnu.org")
		(To . "Ulrich Mueller <ulm@gentoo.org>"))]
	0)
 (32415 32 116
	[32415 "(none)" "(nobody)" "" "fake+none+nnimap+dev.gentoo.org:INBOX+32415" nil -1 -1 nil nil]
	0))
Local in buffer *Summary nnimap+dev.gentoo.org:INBOX*; global value is the same.
----------------------------------------------------------------------
gnus-newsgroup-headers is a variable defined in ‘gnus-sum.el’.
Its value is
([32415 "Re: bug#35443: 27.0.50; Gnus (nnimap) shows \"ghost\" messages in summary buffer" "Eric Abrahamsen <eric@ericabrahamsen.net>" "Sat, 27 Apr 2019 13:47:18 -0700" "<87h8ajjhux.fsf@ericabrahamsen.net>" "<w6g1s1ovuew.fsf@kph.uni-mainz.de> <87wojfjxry.fsf@ericabrahamsen.net> <w6gwojfo3cy.fsf@kph.uni-mainz.de>" 2609 16 nil
	((Cc . "35443@debbugs.gnu.org")
	 (To . "Ulrich Mueller <ulm@gentoo.org>"))]
 [32415 "(none)" "(nobody)" "" "fake+none+nnimap+dev.gentoo.org:INBOX+32415" nil -1 -1 nil nil])
Local in buffer *Summary nnimap+dev.gentoo.org:INBOX*; global value is nil

Documentation:
List of article headers in the current newsgroup.
----------------------------------------------------------------------

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

Ulrich Mueller <ulm@gentoo.org> writes:

>>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:
>
>> Huh. Can you show us the value of gnus-newsgroup-data and
>> gnus-newsgroup-headers in this group?
>
> See below, for the current group, whose summary buffer shows your
> article and a dummy article.
>
>> Does the dummy article always show up?
>
> It always shows up when new articles have been fetched. It doesn't
> show up when entering a group that doesn't have new mail (i.e., if a
> group previously had a dummy article, it will be gone when reentering
> that group).
>
>> Only in this group?
>
> All groups for that particular IMAP server. 
>
>> Is this a newly-added IMAP server, and did it display this problem
>> right from the beginning?
>
> AFAICS, the problem appeared after dovecot on the server side was
> upgraded from version 2.2.34 to 2.3.5.1. But I see it only with Gnus.
> I don't have any issues with Mozilla Thunderbird (on GNU/Linux) nor with
> K-9 Mail (on Android/Linux).

Thanks, this should be enough to make some progress in tracking the
problem down. Give me a couple of days...

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

Ulrich Mueller <ulm@gentoo.org> writes:

> In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit)
>  of 2019-04-21 built on a1i15
> Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
> System Description: Gentoo/Linux
>
> When fetching e-mail from a dovecot-2.3.5.1 IMAP server, Gnus/nnimap
> gets confused and displays ghost messages with address "nobody" and
> subject "(none)" in the summary buffer, like in this example:
>
> *Summary nnimap+dev.gentoo.org:INBOX*
> ----------------------------------------------------------------------
> R. [   1: Ulrich Mueller         ] test 1
>  . [   ?: nobody                 ] (none)
>  . [   1: Ulrich Mueller         ] test 2
> ----------------------------------------------------------------------

[...]

> Buffer " *nnimap dev.gentoo.org nil *nntpd**" looks like this, upon
> entering nnimap-transform-headers:
>----------------------------------------------------------------------
> * 583 FETCH (UID 32409 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165}
> From: Ulrich Mueller <ulm@gentoo.org>
> To: ulm@gentoo.org
> Subject: test 1
> Date: Sat, 27 Apr 2019 07:39:56 +0200
> Message-ID: <w6gd0l8vwer.fsf@kph.uni-mainz.de>
>
> )
> * 584 FETCH (UID 32410 RFC822.SIZE 698 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 8 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {165}
> From: Ulrich Mueller <ulm@gentoo.org>
> To: ulm@gentoo.org
> Subject: test 2
> Date: Sat, 27 Apr 2019 07:40:15 +0200
> Message-ID: <w6g8svwvwe8.fsf@kph.uni-mainz.de>
>
> )
> * 583 FETCH (UID 32409 MODSEQ (63364) FLAGS ($HasNoAttachment))
> * 584 FETCH (UID 32410 MODSEQ (63364) FLAGS ($HasNoAttachment))
> 10194 OK Fetch completed (0.003 + 0.000 + 0.002 secs).
> ----------------------------------------------------------------------

Okay, I've made a bit of progress on this.

Locally I'm using Dovecot 2.3.6, and it does not output those last two
lines before the OK.

`nnimap-transform-headers' is not expecting those two lines -- it
deletes all but the last, leaving a buffer that looks like:

211 32409 Article retrieved.
Chars: 698
Lines: 1
From: Ulrich Mueller <ulm@gentoo.org>
To: ulm@gentoo.org
Subject: test 1
Date: Sat, 27 Apr 2019 07:39:56 +0200
Message-ID: <w6gd0l8vwer.fsf@kph.uni-mainz.de>

.
211 32410 Article retrieved.
Chars: 698
Lines: 1
From: Ulrich Mueller <ulm@gentoo.org>
To: ulm@gentoo.org
Subject: test 2
Date: Sat, 27 Apr 2019 07:40:15 +0200
Message-ID: <w6g8svwvwe8.fsf@kph.uni-mainz.de>

.
211 32409 Article retrieved.
* 584 FETCH (UID 32410 MODSEQ (63364) FLAGS ($HasNoAttachment))
10194 OK Fetch completed (0.003 + 0.000 + 0.002 secs).
.

Which Gnus then parses as an _extra_ article 32409, but then there's no
header data for it, which is why you get all the "nobody" "none"
nonsense.

Essentially, we're not set up to parse this particular return value.

I guess what I'll do is ask on the dovecot mailing list under what
circumstances/versions we'd get a string like that, and try to help Gnus
parse it correctly.

Thanks for your patience,
Eric

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

On 04/28/19 05:53 AM, Ulrich Mueller wrote:
>>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:
>
>> Huh. Can you show us the value of gnus-newsgroup-data and
>> gnus-newsgroup-headers in this group?
>
> See below, for the current group, whose summary buffer shows your
> article and a dummy article.

Can you also tell me your value for `nnmail-extra-headers'?

Thanks,
Eric

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Ulrich Mueller]

>>>>> On Tue, 07 May 2019, Eric Abrahamsen wrote:

> Can you also tell me your value for `nnmail-extra-headers'?

(X-Diary-Time-Zone X-Diary-Dow X-Diary-Year X-Diary-Month X-Diary-Dom X-Diary-Hour X-Diary-Minute To Newsgroups Cc)

Not sure where the X-Diary-* ones come from, since I don't have that
variable in my config files.

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Ulrich Mueller]

>>>>> On Tue, 07 May 2019, Eric Abrahamsen wrote:

> Locally I'm using Dovecot 2.3.6, and it does not output those last two
> lines before the OK.

Hm, on the server side here dovecot was upgraded to 2.3.6 today, but
the problem persists. Upon entering nnimap-transform-headers, the
" *nnimap dev.gentoo.org nil  *nntpd**" buffer shows the same pattern
as in my original report:

* 590 FETCH (UID 32471 RFC822.SIZE 694 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 6 1 NIL NIL NIL NIL) BODY[HEADER.FIELDS (SUBJECT FROM DATE MESSAGE-ID REFERENCES IN-REPLY-TO XREF X-DIARY-TIME-ZONE X-DIARY-DOW X-DIARY-YEAR X-DIARY-MONTH X-DIARY-DOM X-DIARY-HOUR X-DIARY-MINUTE TO NEWSGROUPS CC)] {163}
From: Ulrich Mueller <ulm@gentoo.org>
To: ulm@gentoo.org
Subject: test
Date: Wed, 08 May 2019 13:59:32 +0200
Message-ID: <w6gzhnxkvh7.fsf@kph.uni-mainz.de>

)
* 590 FETCH (UID 32471 MODSEQ (63547) FLAGS ($HasNoAttachment))
16742 OK Fetch completed (0.003 + 0.000 + 0.002 secs).

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

--8<---------------cut here---------------start------------->8---
> Ulrich Mueller <ulm@gentoo.org> writes:
>
>>>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:
>>
>>> Huh. Can you show us the value of gnus-newsgroup-data and
>>> gnus-newsgroup-headers in this group?
>>
>> See below, for the current group, whose summary buffer shows your
>> article and a dummy article.
>>
>>> Does the dummy article always show up?
>>
>> It always shows up when new articles have been fetched. It doesn't
>> show up when entering a group that doesn't have new mail (i.e., if a
>> group previously had a dummy article, it will be gone when reentering
>> that group).
>>
>>> Only in this group?
>>
>> All groups for that particular IMAP server.
>>
>>> Is this a newly-added IMAP server, and did it display this problem
>>> right from the beginning?
>>
>> AFAICS, the problem appeared after dovecot on the server side was
>> upgraded from version 2.2.34 to 2.3.5.1. But I see it only with Gnus.
>> I don't have any issues with Mozilla Thunderbird (on GNU/Linux) nor with
>> K-9 Mail (on Android/Linux).
>
> Thanks, this should be enough to make some progress in tracking the
> problem down. Give me a couple of days...

Okay, here's what I was told on irc:

--8<---------------cut here---------------start------------->8---
<tss> looks like it's the new feature that adds $HasAttachment or
      $HasNoAttachment flags to mails. they're actually supposed to be added
      only during mail delivery, and there's a setting needed to enable them:
      mail_attachment_detection_options = add-flags-on-save
<tss> but .. there's an "unintentional feature" :) that they also get added
      during some FETCH commands if they're not already there
<tss> but even without this, imap protocol allows sending FETCH FLAGS updates
      (or any updates really) as a response to any command. and with
      concurrent imap access dovecot does this.
<tss> for example if another client adds a \Seen flag, that same thing could
      happen
<jkt> either way, a client should really be able to process these
      "unexpeceted" FETCH updates, that's what the standard is about
--8<---------------cut here---------------end--------------->8---

Ultimately, the "real" fix would be to teach Gnus to handle all possible
responses from IMAP servers, probably using a proper parser.

But if you have access to the dovecot config in your case, you might try
removing the mail_attachment_detection_options setting above, if it's
set. If it's not set, then you might be getting the "unintentional
feature".

I suppose in the interim I could mess with `nnimap-transform-headers'
and try to add some bookkeeping so that multiple FETCH responses for the
same article UID get merged together. On the other hand, this feature
only seems to relate to has/hasnoattachment flags, which Gnus doesn't
handle anyway -- we could safely drop those lines altogether.

(Though it sure would be nice to handle hasattachment flags, that's
something that users have requested in the past...)

Well, that's where we're at.

Eric

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Ulrich Mueller]

>>>>> On Thu, 09 May 2019, Eric Abrahamsen wrote:

> Okay, here's what I was told on irc:

> <tss> looks like it's the new feature that adds $HasAttachment or
>       $HasNoAttachment flags to mails. they're actually supposed to be added
>       only during mail delivery, and there's a setting needed to enable them:
>       mail_attachment_detection_options = add-flags-on-save
> <tss> but .. there's an "unintentional feature" :) that they also get added
>       during some FETCH commands if they're not already there
> <tss> but even without this, imap protocol allows sending FETCH FLAGS updates
>       (or any updates really) as a response to any command. and with
>       concurrent imap access dovecot does this.
> <tss> for example if another client adds a \Seen flag, that same thing could
>       happen
> <jkt> either way, a client should really be able to process these
>       "unexpeceted" FETCH updates, that's what the standard is about
> --8<---------------cut here---------------end--------------->8---

> Ultimately, the "real" fix would be to teach Gnus to handle all possible
> responses from IMAP servers, probably using a proper parser.

> But if you have access to the dovecot config in your case, you might try
> removing the mail_attachment_detection_options setting above, if it's
> set. If it's not set, then you might be getting the "unintentional
> feature".

Thank you very much for your help. Indeed the add-flags-on-save option
was enabled in the config file. Gentoo's infrastructure team has agreed
to disable it for now, which made the problem go away.

> I suppose in the interim I could mess with `nnimap-transform-headers'
> and try to add some bookkeeping so that multiple FETCH responses for
> the same article UID get merged together. On the other hand, this
> feature only seems to relate to has/hasnoattachment flags, which Gnus
> doesn't handle anyway -- we could safely drop those lines altogether.

> (Though it sure would be nice to handle hasattachment flags, that's
> something that users have requested in the past...)

Maybe drop the extra FETCH responses for now, and add proper bookkeeping
later when implementing the requested feature?

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

Ulrich Mueller <ulm@gentoo.org> writes:

>>>>>> On Thu, 09 May 2019, Eric Abrahamsen wrote:
>
>> Okay, here's what I was told on irc:
>
>> <tss> looks like it's the new feature that adds $HasAttachment or
>>       $HasNoAttachment flags to mails. they're actually supposed to be added
>>       only during mail delivery, and there's a setting needed to enable them:
>>       mail_attachment_detection_options = add-flags-on-save
>> <tss> but .. there's an "unintentional feature" :) that they also get added
>>       during some FETCH commands if they're not already there
>> <tss> but even without this, imap protocol allows sending FETCH FLAGS updates
>>       (or any updates really) as a response to any command. and with
>>       concurrent imap access dovecot does this.
>> <tss> for example if another client adds a \Seen flag, that same thing could
>>       happen
>> <jkt> either way, a client should really be able to process these
>>       "unexpeceted" FETCH updates, that's what the standard is about
>> --8<---------------cut here---------------end--------------->8---
>
>> Ultimately, the "real" fix would be to teach Gnus to handle all possible
>> responses from IMAP servers, probably using a proper parser.
>
>> But if you have access to the dovecot config in your case, you might try
>> removing the mail_attachment_detection_options setting above, if it's
>> set. If it's not set, then you might be getting the "unintentional
>> feature".
>
> Thank you very much for your help. Indeed the add-flags-on-save option
> was enabled in the config file. Gentoo's infrastructure team has agreed
> to disable it for now, which made the problem go away.

Well that was lucky.

>> I suppose in the interim I could mess with `nnimap-transform-headers'
>> and try to add some bookkeeping so that multiple FETCH responses for
>> the same article UID get merged together. On the other hand, this
>> feature only seems to relate to has/hasnoattachment flags, which Gnus
>> doesn't handle anyway -- we could safely drop those lines altogether.
>
>> (Though it sure would be nice to handle hasattachment flags, that's
>> something that users have requested in the past...)
>
> Maybe drop the extra FETCH responses for now, and add proper bookkeeping
> later when implementing the requested feature?

Okay, I'll work something up in the next week that goes halfway with
this.

Glad it's partially fixed.

Eric

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>> Maybe drop the extra FETCH responses for now, and add proper bookkeeping
>> later when implementing the requested feature?
>
> Okay, I'll work something up in the next week that goes halfway with
> this.

Did you get any further in fixing this nnimap parsing bug?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>> Maybe drop the extra FETCH responses for now, and add proper bookkeeping
>>> later when implementing the requested feature?
>>
>> Okay, I'll work something up in the next week that goes halfway with
>> this.
>
> Did you get any further in fixing this nnimap parsing bug?

No I didn't! I looked at it for a while, had an internal conflict about
just dropping the lines vs actually doing something with them, imagined
all the work that would go into a more fully-featured parser, then got
distracted and forgot about it.

I still hope that in some distant future we could have a real parser
consuming these buffers. So far as I know the only in-emacs options are
in cedet -- wisent and the other one -- but I've never been able to make
them work. And now it sounds like cedet might not even stay in-tree?

Anyway, I'll make a real todo for dropping the lines, though it seems a
shame...

Eric

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 1067 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>> Maybe drop the extra FETCH responses for now, and add proper bookkeeping
>>> later when implementing the requested feature?
>>
>> Okay, I'll work something up in the next week that goes halfway with
>> this.
>
> Did you get any further in fixing this nnimap parsing bug?

Here's a whack at it. I tried to make sure that it would handle unwanted
FETCH responses whether they came before or after (or in the middle of)
the wanted FETCH responses, but I'm not in love with checking the header
regexp this way.

Because this IMAP server feature is very closely focused on adding a
flag in case of attachment (and because Gnus never explicitly requests
this flag, though I'd sure like to in the future), another more targeted
approach would be to simply delete any lines containing
$Has\(No\)?Attachment, assuming that these FETCH responses will only
take up one line.

I've configured my local Dovecot with the offending setting, and will
test it out for a bit.

Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: imap-header-transform-fix.diff --]
[-- Type: text/x-patch, Size: 3949 bytes --]

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index 9e52abc1ca..7478fcb3e5 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -232,7 +232,7 @@ nnimap-retrieve-headers
 
 (defun nnimap-transform-headers ()
   (goto-char (point-min))
-  (let (article lines size string labels)
+  (let (seen-articles article lines size string labels)
     (cl-block nil
       (while (not (eobp))
 	(while (not (looking-at "\\* [0-9]+ FETCH"))
@@ -261,45 +261,56 @@ nnimap-transform-headers
 	      (and (re-search-forward "UID \\([0-9]+\\)" (line-end-position)
 				      t)
 		   (match-string 1)))
-	(setq lines nil)
-	(beginning-of-line)
-	(setq size
-	      (and (re-search-forward "RFC822.SIZE \\([0-9]+\\)"
-				      (line-end-position)
-				      t)
-		   (match-string 1)))
-	(beginning-of-line)
-	(when (search-forward "X-GM-LABELS" (line-end-position) t)
-	  (setq labels (ignore-errors (read (current-buffer)))))
-	(beginning-of-line)
-	(when (search-forward "BODYSTRUCTURE" (line-end-position) t)
-	  (let ((structure (ignore-errors
-			     (read (current-buffer)))))
-	    (while (and (consp structure)
-			(not (atom (car structure))))
-	      (setq structure (car structure)))
-	    (setq lines (if (and
-			     (stringp (car structure))
-			     (equal (upcase (nth 0 structure)) "MESSAGE")
-			     (equal (upcase (nth 1 structure)) "RFC822"))
-			    (nth 9 structure)
-			  (nth 7 structure)))))
-	(delete-region (line-beginning-position) (line-end-position))
-	(insert (format "211 %s Article retrieved." article))
-	(forward-line 1)
-	(when size
-	  (insert (format "Chars: %s\n" size)))
-	(when lines
-	  (insert (format "Lines: %s\n" lines)))
-	(when labels
-	  (insert (format "X-GM-LABELS: %s\n" labels)))
-	;; Most servers have a blank line after the headers, but
-	;; Davmail doesn't.
-	(unless (re-search-forward "^\r$\\|^)\r?$" nil t)
-	  (goto-char (point-max)))
-	(delete-region (line-beginning-position) (line-end-position))
-	(insert ".")
-	(forward-line 1)))))
+	;; If we've already got headers for this article, or this
+	;; FETCH line doesn't provide headers for the article, skip
+	;; it.  See bug#35433.
+	(if (or (member article seen-articles)
+		(save-excursion
+		  (forward-line)
+		  (null (looking-at-p
+			 "\\(From\\|To\\|Subject\\|Date\\|Message-ID\\)"))))
+	    (delete-region (line-beginning-position)
+			   (1+ (line-end-position)))
+	  (setq lines nil)
+	  (beginning-of-line)
+	  (setq size
+		(and (re-search-forward "RFC822.SIZE \\([0-9]+\\)"
+					(line-end-position)
+					t)
+		     (match-string 1)))
+	  (beginning-of-line)
+	  (when (search-forward "X-GM-LABELS" (line-end-position) t)
+	    (setq labels (ignore-errors (read (current-buffer)))))
+	  (beginning-of-line)
+	  (when (search-forward "BODYSTRUCTURE" (line-end-position) t)
+	    (let ((structure (ignore-errors
+			       (read (current-buffer)))))
+	      (while (and (consp structure)
+			  (not (atom (car structure))))
+		(setq structure (car structure)))
+	      (setq lines (if (and
+			       (stringp (car structure))
+			       (equal (upcase (nth 0 structure)) "MESSAGE")
+			       (equal (upcase (nth 1 structure)) "RFC822"))
+			      (nth 9 structure)
+			    (nth 7 structure)))))
+	  (delete-region (line-beginning-position) (line-end-position))
+	  (insert (format "211 %s Article retrieved." article))
+	  (forward-line 1)
+	  (when size
+	    (insert (format "Chars: %s\n" size)))
+	  (when lines
+	    (insert (format "Lines: %s\n" lines)))
+	  (when labels
+	    (insert (format "X-GM-LABELS: %s\n" labels)))
+	  ;; Most servers have a blank line after the headers, but
+	  ;; Davmail doesn't.
+	  (unless (re-search-forward "^\r$\\|^)\r?$" nil t)
+	    (goto-char (point-max)))
+	  (delete-region (line-beginning-position) (line-end-position))
+	  (insert ".")
+	  (forward-line 1)
+	  (push article seen-articles))))))
 
 (defun nnimap-unfold-quoted-lines ()
   ;; Unfold quoted {number} strings.

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> No I didn't! I looked at it for a while, had an internal conflict about
> just dropping the lines vs actually doing something with them, imagined
> all the work that would go into a more fully-featured parser, then got
> distracted and forgot about it.

OK, I'll take a look at it...

> I still hope that in some distant future we could have a real parser
> consuming these buffers. So far as I know the only in-emacs options are
> in cedet -- wisent and the other one -- but I've never been able to make
> them work. And now it sounds like cedet might not even stay in-tree?

I had to add a parsing feature to wisent, and that was kinda more
painful than you'd expect.  :-/

But I haven't heard anything about dumping cedet from the tree.  Is that
the plan?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>> Did you get any further in fixing this nnimap parsing bug?
>
> Here's a whack at it. I tried to make sure that it would handle unwanted
> FETCH responses whether they came before or after (or in the middle of)
> the wanted FETCH responses, but I'm not in love with checking the header
> regexp this way.

Well, I think it's OK...

> Because this IMAP server feature is very closely focused on adding a
> flag in case of attachment (and because Gnus never explicitly requests
> this flag, though I'd sure like to in the future), another more targeted
> approach would be to simply delete any lines containing
> $Has\(No\)?Attachment, assuming that these FETCH responses will only
> take up one line.

That sounds a bit brittle -- I'm sure there'll be other extensions like
this in the future to the IMAP protocol.

> I've configured my local Dovecot with the offending setting, and will
> test it out for a bit.

Great!

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

On 06/23/19 14:13 PM, Lars Ingebrigtsen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> No I didn't! I looked at it for a while, had an internal conflict about
>> just dropping the lines vs actually doing something with them, imagined
>> all the work that would go into a more fully-featured parser, then got
>> distracted and forgot about it.
>
> OK, I'll take a look at it...
>
>> I still hope that in some distant future we could have a real parser
>> consuming these buffers. So far as I know the only in-emacs options are
>> in cedet -- wisent and the other one -- but I've never been able to make
>> them work. And now it sounds like cedet might not even stay in-tree?
>
> I had to add a parsing feature to wisent, and that was kinda more
> painful than you'd expect.  :-/

Oh, I thought that was most of the point of wisent to begin with. No
wonder I couldn't get it to do anything. In theory, do you have any
recommendations in the parsing direction.

> But I haven't heard anything about dumping cedet from the tree. Is that
> the plan?

No, I was skimming your threads about cleaning up the build process and
got the impression that it was unmaintained. I probably misinterpreted.

On 06/23/19 14:23 PM, Lars Ingebrigtsen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>> Did you get any further in fixing this nnimap parsing bug?
>>
>> Here's a whack at it. I tried to make sure that it would handle unwanted
>> FETCH responses whether they came before or after (or in the middle of)
>> the wanted FETCH responses, but I'm not in love with checking the header
>> regexp this way.
>
> Well, I think it's OK...

Cool.

>> Because this IMAP server feature is very closely focused on adding a
>> flag in case of attachment (and because Gnus never explicitly requests
>> this flag, though I'd sure like to in the future), another more targeted
>> approach would be to simply delete any lines containing
>> $Has\(No\)?Attachment, assuming that these FETCH responses will only
>> take up one line.
>
> That sounds a bit brittle -- I'm sure there'll be other extensions like
> this in the future to the IMAP protocol.

Sure, I'll stick with this.

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>> I had to add a parsing feature to wisent, and that was kinda more
>> painful than you'd expect.  :-/
>
> Oh, I thought that was most of the point of wisent to begin with. No
> wonder I couldn't get it to do anything. In theory, do you have any
> recommendations in the parsing direction.

Yes, that's the point of wisent, but as with any lex-like thing I've
dealt with, it's...  painful.  :-)  At least to me.

>> But I haven't heard anything about dumping cedet from the tree. Is that
>> the plan?
>
> No, I was skimming your threads about cleaning up the build process and
> got the impression that it was unmaintained. I probably misinterpreted.

We are all the maintainer of cedet now.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

On 06/23/19 18:58 PM, Lars Ingebrigtsen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>> I had to add a parsing feature to wisent, and that was kinda more
>>> painful than you'd expect.  :-/
>>
>> Oh, I thought that was most of the point of wisent to begin with. No
>> wonder I couldn't get it to do anything. In theory, do you have any
>> recommendations in the parsing direction.
>
> Yes, that's the point of wisent, but as with any lex-like thing I've
> dealt with, it's...  painful.  :-)  At least to me.

Huh. There's a package in the repos called parsec that's pretty easy to
use, but nnimap couldn't rely on that....

>>> But I haven't heard anything about dumping cedet from the tree. Is that
>>> the plan?
>>
>> No, I was skimming your threads about cleaning up the build process and
>> got the impression that it was unmaintained. I probably misinterpreted.
>
> We are all the maintainer of cedet now.  :-)

Ugh.

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

On 06/23/19 14:23 PM, Lars Ingebrigtsen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>> Did you get any further in fixing this nnimap parsing bug?
>>
>> Here's a whack at it. I tried to make sure that it would handle unwanted
>> FETCH responses whether they came before or after (or in the middle of)
>> the wanted FETCH responses, but I'm not in love with checking the header
>> regexp this way.
>
> Well, I think it's OK...
>
>> Because this IMAP server feature is very closely focused on adding a
>> flag in case of attachment (and because Gnus never explicitly requests
>> this flag, though I'd sure like to in the future), another more targeted
>> approach would be to simply delete any lines containing
>> $Has\(No\)?Attachment, assuming that these FETCH responses will only
>> take up one line.
>
> That sounds a bit brittle -- I'm sure there'll be other extensions like
> this in the future to the IMAP protocol.
>
>> I've configured my local Dovecot with the offending setting, and will
>> test it out for a bit.
>
> Great!

(NB This patch is very broken and no one should use it; I'm still
working on it. I want a proper parser!)

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> On 06/23/19 14:23 PM, Lars Ingebrigtsen wrote:
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>>> Did you get any further in fixing this nnimap parsing bug?
>>>
>>> Here's a whack at it. I tried to make sure that it would handle unwanted
>>> FETCH responses whether they came before or after (or in the middle of)
>>> the wanted FETCH responses, but I'm not in love with checking the header
>>> regexp this way.
>>
>> Well, I think it's OK...
>>
>>> Because this IMAP server feature is very closely focused on adding a
>>> flag in case of attachment (and because Gnus never explicitly requests
>>> this flag, though I'd sure like to in the future), another more targeted
>>> approach would be to simply delete any lines containing
>>> $Has\(No\)?Attachment, assuming that these FETCH responses will only
>>> take up one line.
>>
>> That sounds a bit brittle -- I'm sure there'll be other extensions like
>> this in the future to the IMAP protocol.
>>
>>> I've configured my local Dovecot with the offending setting, and will
>>> test it out for a bit.
>>
>> Great!
>
> (NB This patch is very broken and no one should use it; I'm still
> working on it. I want a proper parser!)

Turns out the dumb thing was simple: I was testing if the FETCH line was
followed by mail headers by looking for:

"\\(From\\|To\\|Subject\\|Date\\|Message-ID\\)"

Which is obviously insufficient. I've since changed it to:

"\\([A-Z][[:ascii:]-]+: \\)"

Which works better, but also seems susceptible to breakage due to
ignorance. A quick scan doesn't turn up any `mail-header-regexp'
variables or `this-is-a-mail-header-p' functions -- does the above
regexp seem reasonable?

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Which is obviously insufficient. I've since changed it to:
>
> "\\([A-Z][[:ascii:]-]+: \\)"
>
> Which works better, but also seems susceptible to breakage due to
> ignorance. A quick scan doesn't turn up any `mail-header-regexp'
> variables or `this-is-a-mail-header-p' functions -- does the above
> regexp seem reasonable?

Headers are case insensitive...  but not all characters in the [:ascii:]
set are allowed; far from it.  [:alnum:] is closer, but with - and...
er...  stuff...  One of the RFCs probably has the definition.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> (NB This patch is very broken and no one should use it; I'm still
> working on it. I want a proper parser!)

Yes, that would be nice.  But there wasn't any patch attached.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Which is obviously insufficient. I've since changed it to:
>>
>> "\\([A-Z][[:ascii:]-]+: \\)"
>>
>> Which works better, but also seems susceptible to breakage due to
>> ignorance. A quick scan doesn't turn up any `mail-header-regexp'
>> variables or `this-is-a-mail-header-p' functions -- does the above
>> regexp seem reasonable?
>
> Headers are case insensitive...  but not all characters in the [:ascii:]
> set are allowed; far from it.  [:alnum:] is closer, but with - and...
> er...  stuff...  One of the RFCs probably has the definition.  :-)

Oh, good point. RFC2822 says chars 33 to 126, inclusive, minus the
colon. So I guess it's "[!-9,;-~]+: ", unlikely as some of those
characters seem.

Now I'm working with the attached diff.

Eric

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: imap-header-transform-fix.diff --]
[-- Type: text/x-patch, Size: 3916 bytes --]

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index 9e52abc1ca..802254fc14 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -232,7 +232,7 @@ nnimap-retrieve-headers
 
 (defun nnimap-transform-headers ()
   (goto-char (point-min))
-  (let (article lines size string labels)
+  (let (seen-articles article lines size string labels)
     (cl-block nil
       (while (not (eobp))
 	(while (not (looking-at "\\* [0-9]+ FETCH"))
@@ -261,45 +261,56 @@ nnimap-transform-headers
 	      (and (re-search-forward "UID \\([0-9]+\\)" (line-end-position)
 				      t)
 		   (match-string 1)))
-	(setq lines nil)
-	(beginning-of-line)
-	(setq size
-	      (and (re-search-forward "RFC822.SIZE \\([0-9]+\\)"
-				      (line-end-position)
-				      t)
-		   (match-string 1)))
-	(beginning-of-line)
-	(when (search-forward "X-GM-LABELS" (line-end-position) t)
-	  (setq labels (ignore-errors (read (current-buffer)))))
-	(beginning-of-line)
-	(when (search-forward "BODYSTRUCTURE" (line-end-position) t)
-	  (let ((structure (ignore-errors
-			     (read (current-buffer)))))
-	    (while (and (consp structure)
-			(not (atom (car structure))))
-	      (setq structure (car structure)))
-	    (setq lines (if (and
-			     (stringp (car structure))
-			     (equal (upcase (nth 0 structure)) "MESSAGE")
-			     (equal (upcase (nth 1 structure)) "RFC822"))
-			    (nth 9 structure)
-			  (nth 7 structure)))))
-	(delete-region (line-beginning-position) (line-end-position))
-	(insert (format "211 %s Article retrieved." article))
-	(forward-line 1)
-	(when size
-	  (insert (format "Chars: %s\n" size)))
-	(when lines
-	  (insert (format "Lines: %s\n" lines)))
-	(when labels
-	  (insert (format "X-GM-LABELS: %s\n" labels)))
-	;; Most servers have a blank line after the headers, but
-	;; Davmail doesn't.
-	(unless (re-search-forward "^\r$\\|^)\r?$" nil t)
-	  (goto-char (point-max)))
-	(delete-region (line-beginning-position) (line-end-position))
-	(insert ".")
-	(forward-line 1)))))
+	;; If we've already got headers for this article, or this
+	;; FETCH line doesn't provide headers for the article, skip
+	;; it.  See bug#35433.
+	(if (or (member article seen-articles)
+		(save-excursion
+		  (forward-line)
+		  (null (looking-at-p
+			 "[!-9,;-~]+: "))))
+	    (delete-region (line-beginning-position)
+			   (1+ (line-end-position)))
+	  (setq lines nil)
+	  (beginning-of-line)
+	  (setq size
+		(and (re-search-forward "RFC822.SIZE \\([0-9]+\\)"
+					(line-end-position)
+					t)
+		     (match-string 1)))
+	  (beginning-of-line)
+	  (when (search-forward "X-GM-LABELS" (line-end-position) t)
+	    (setq labels (ignore-errors (read (current-buffer)))))
+	  (beginning-of-line)
+	  (when (search-forward "BODYSTRUCTURE" (line-end-position) t)
+	    (let ((structure (ignore-errors
+			       (read (current-buffer)))))
+	      (while (and (consp structure)
+			  (not (atom (car structure))))
+		(setq structure (car structure)))
+	      (setq lines (if (and
+			       (stringp (car structure))
+			       (equal (upcase (nth 0 structure)) "MESSAGE")
+			       (equal (upcase (nth 1 structure)) "RFC822"))
+			      (nth 9 structure)
+			    (nth 7 structure)))))
+	  (delete-region (line-beginning-position) (line-end-position))
+	  (insert (format "211 %s Article retrieved." article))
+	  (forward-line 1)
+	  (when size
+	    (insert (format "Chars: %s\n" size)))
+	  (when lines
+	    (insert (format "Lines: %s\n" lines)))
+	  (when labels
+	    (insert (format "X-GM-LABELS: %s\n" labels)))
+	  ;; Most servers have a blank line after the headers, but
+	  ;; Davmail doesn't.
+	  (unless (re-search-forward "^\r$\\|^)\r?$" nil t)
+	    (goto-char (point-max)))
+	  (delete-region (line-beginning-position) (line-end-position))
+	  (insert ".")
+	  (forward-line 1)
+	  (push article seen-articles))))))
 
 (defun nnimap-unfold-quoted-lines ()
   ;; Unfold quoted {number} strings.

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Which is obviously insufficient. I've since changed it to:
>>>
>>> "\\([A-Z][[:ascii:]-]+: \\)"
>>>
>>> Which works better, but also seems susceptible to breakage due to
>>> ignorance. A quick scan doesn't turn up any `mail-header-regexp'
>>> variables or `this-is-a-mail-header-p' functions -- does the above
>>> regexp seem reasonable?
>>
>> Headers are case insensitive...  but not all characters in the [:ascii:]
>> set are allowed; far from it.  [:alnum:] is closer, but with - and...
>> er...  stuff...  One of the RFCs probably has the definition.  :-)
>
> Oh, good point. RFC2822 says chars 33 to 126, inclusive, minus the
> colon. So I guess it's "[!-9,;-~]+: ", unlikely as some of those
> characters seem.
>
> Now I'm working with the attached diff.

Sorry, this one...


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: imap-header-transform-fix.diff --]
[-- Type: text/x-patch, Size: 3916 bytes --]

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index 9e52abc1ca..1fe6d32c4f 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -232,7 +232,7 @@ nnimap-retrieve-headers
 
 (defun nnimap-transform-headers ()
   (goto-char (point-min))
-  (let (article lines size string labels)
+  (let (seen-articles article lines size string labels)
     (cl-block nil
       (while (not (eobp))
 	(while (not (looking-at "\\* [0-9]+ FETCH"))
@@ -261,45 +261,56 @@ nnimap-transform-headers
 	      (and (re-search-forward "UID \\([0-9]+\\)" (line-end-position)
 				      t)
 		   (match-string 1)))
-	(setq lines nil)
-	(beginning-of-line)
-	(setq size
-	      (and (re-search-forward "RFC822.SIZE \\([0-9]+\\)"
-				      (line-end-position)
-				      t)
-		   (match-string 1)))
-	(beginning-of-line)
-	(when (search-forward "X-GM-LABELS" (line-end-position) t)
-	  (setq labels (ignore-errors (read (current-buffer)))))
-	(beginning-of-line)
-	(when (search-forward "BODYSTRUCTURE" (line-end-position) t)
-	  (let ((structure (ignore-errors
-			     (read (current-buffer)))))
-	    (while (and (consp structure)
-			(not (atom (car structure))))
-	      (setq structure (car structure)))
-	    (setq lines (if (and
-			     (stringp (car structure))
-			     (equal (upcase (nth 0 structure)) "MESSAGE")
-			     (equal (upcase (nth 1 structure)) "RFC822"))
-			    (nth 9 structure)
-			  (nth 7 structure)))))
-	(delete-region (line-beginning-position) (line-end-position))
-	(insert (format "211 %s Article retrieved." article))
-	(forward-line 1)
-	(when size
-	  (insert (format "Chars: %s\n" size)))
-	(when lines
-	  (insert (format "Lines: %s\n" lines)))
-	(when labels
-	  (insert (format "X-GM-LABELS: %s\n" labels)))
-	;; Most servers have a blank line after the headers, but
-	;; Davmail doesn't.
-	(unless (re-search-forward "^\r$\\|^)\r?$" nil t)
-	  (goto-char (point-max)))
-	(delete-region (line-beginning-position) (line-end-position))
-	(insert ".")
-	(forward-line 1)))))
+	;; If we've already got headers for this article, or this
+	;; FETCH line doesn't provide headers for the article, skip
+	;; it.  See bug#35433.
+	(if (or (member article seen-articles)
+		(save-excursion
+		  (forward-line)
+		  (null (looking-at-p
+			 "^[!-9;-~]+: "))))
+	    (delete-region (line-beginning-position)
+			   (1+ (line-end-position)))
+	  (setq lines nil)
+	  (beginning-of-line)
+	  (setq size
+		(and (re-search-forward "RFC822.SIZE \\([0-9]+\\)"
+					(line-end-position)
+					t)
+		     (match-string 1)))
+	  (beginning-of-line)
+	  (when (search-forward "X-GM-LABELS" (line-end-position) t)
+	    (setq labels (ignore-errors (read (current-buffer)))))
+	  (beginning-of-line)
+	  (when (search-forward "BODYSTRUCTURE" (line-end-position) t)
+	    (let ((structure (ignore-errors
+			       (read (current-buffer)))))
+	      (while (and (consp structure)
+			  (not (atom (car structure))))
+		(setq structure (car structure)))
+	      (setq lines (if (and
+			       (stringp (car structure))
+			       (equal (upcase (nth 0 structure)) "MESSAGE")
+			       (equal (upcase (nth 1 structure)) "RFC822"))
+			      (nth 9 structure)
+			    (nth 7 structure)))))
+	  (delete-region (line-beginning-position) (line-end-position))
+	  (insert (format "211 %s Article retrieved." article))
+	  (forward-line 1)
+	  (when size
+	    (insert (format "Chars: %s\n" size)))
+	  (when lines
+	    (insert (format "Lines: %s\n" lines)))
+	  (when labels
+	    (insert (format "X-GM-LABELS: %s\n" labels)))
+	  ;; Most servers have a blank line after the headers, but
+	  ;; Davmail doesn't.
+	  (unless (re-search-forward "^\r$\\|^)\r?$" nil t)
+	    (goto-char (point-max)))
+	  (delete-region (line-beginning-position) (line-end-position))
+	  (insert ".")
+	  (forward-line 1)
+	  (push article seen-articles))))))
 
 (defun nnimap-unfold-quoted-lines ()
   ;; Unfold quoted {number} strings.

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

On 07/12/19 09:50 AM, Eric Abrahamsen wrote:
> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Which is obviously insufficient. I've since changed it to:
>>>
>>> "\\([A-Z][[:ascii:]-]+: \\)"
>>>
>>> Which works better, but also seems susceptible to breakage due to
>>> ignorance. A quick scan doesn't turn up any `mail-header-regexp'
>>> variables or `this-is-a-mail-header-p' functions -- does the above
>>> regexp seem reasonable?
>>
>> Headers are case insensitive...  but not all characters in the [:ascii:]
>> set are allowed; far from it.  [:alnum:] is closer, but with - and...
>> er...  stuff...  One of the RFCs probably has the definition.  :-)
>
> Oh, good point. RFC2822 says chars 33 to 126, inclusive, minus the
> colon. So I guess it's "[!-9,;-~]+: ", unlikely as some of those
> characters seem.
>
> Now I'm working with the attached diff.

I think this is the one. I'll push in a bit.

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen]

There goes...

bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> There goes...

:-)

Works for me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no