bug#35443: 27.0.50; Gnus (nnimap) shows "ghost" messages in summary buffer

[Eric Abrahamsen <eric@ericabrahamsen.net>]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

--8<---------------cut here---------------start------------->8---
> Ulrich Mueller <ulm@gentoo.org> writes:
>
>>>>>>> On Sat, 27 Apr 2019, Eric Abrahamsen wrote:
>>
>>> Huh. Can you show us the value of gnus-newsgroup-data and
>>> gnus-newsgroup-headers in this group?
>>
>> See below, for the current group, whose summary buffer shows your
>> article and a dummy article.
>>
>>> Does the dummy article always show up?
>>
>> It always shows up when new articles have been fetched. It doesn't
>> show up when entering a group that doesn't have new mail (i.e., if a
>> group previously had a dummy article, it will be gone when reentering
>> that group).
>>
>>> Only in this group?
>>
>> All groups for that particular IMAP server.
>>
>>> Is this a newly-added IMAP server, and did it display this problem
>>> right from the beginning?
>>
>> AFAICS, the problem appeared after dovecot on the server side was
>> upgraded from version 2.2.34 to 2.3.5.1. But I see it only with Gnus.
>> I don't have any issues with Mozilla Thunderbird (on GNU/Linux) nor with
>> K-9 Mail (on Android/Linux).
>
> Thanks, this should be enough to make some progress in tracking the
> problem down. Give me a couple of days...

Okay, here's what I was told on irc:

--8<---------------cut here---------------start------------->8---
<tss> looks like it's the new feature that adds $HasAttachment or
      $HasNoAttachment flags to mails. they're actually supposed to be added
      only during mail delivery, and there's a setting needed to enable them:
      mail_attachment_detection_options = add-flags-on-save
<tss> but .. there's an "unintentional feature" :) that they also get added
      during some FETCH commands if they're not already there
<tss> but even without this, imap protocol allows sending FETCH FLAGS updates
      (or any updates really) as a response to any command. and with
      concurrent imap access dovecot does this.
<tss> for example if another client adds a \Seen flag, that same thing could
      happen
<jkt> either way, a client should really be able to process these
      "unexpeceted" FETCH updates, that's what the standard is about
--8<---------------cut here---------------end--------------->8---

Ultimately, the "real" fix would be to teach Gnus to handle all possible
responses from IMAP servers, probably using a proper parser.

But if you have access to the dovecot config in your case, you might try
removing the mail_attachment_detection_options setting above, if it's
set. If it's not set, then you might be getting the "unintentional
feature".

I suppose in the interim I could mess with `nnimap-transform-headers'
and try to add some bookkeeping so that multiple FETCH responses for the
same article UID get merged together. On the other hand, this feature
only seems to relate to has/hasnoattachment flags, which Gnus doesn't
handle anyway -- we could safely drop those lines altogether.

(Though it sure would be nice to handle hasattachment flags, that's
something that users have requested in the past...)

Well, that's where we're at.

Eric