bug#39618: 28.0.50; gnus nnimap reports more group articles than actually exist

[Eric Abrahamsen <eric@ericabrahamsen.net>]

On 02/18/20 21:56 PM, Deus Max wrote:
> On Sun, Feb 16 2020, Eric Abrahamsen wrote:
>
>> Deus Max <deusmax@gmx.com> writes:
>>
>>> Recently my gnus started displaying (in the *Group* buffer) some groups
>>> (which had previously no unread articles) with an unread "ghost" article
>>> but which groups could not be normally entered.
>>>
>>> Here is a description of what I observed:
>>>
>>> 1. When pressing <Return>, the cursor simply moves on to the next group
>>>    line. The number of unread articles remains unchanged and non-zero.
>>> 2. When pressing <c> for `gnus-group-catchup-current', the unread
>>>    article number becomes zero and upon <Return>, all read articles are
>>>    displayed in the group. Upon exit from the group, upon "g"
>>>    gnus-refresh the "ghost" unread article reappears.
>>>
>>> Luckily, this being nnimap the situation can be recovered by removing
>>> the .newsrc files and restarting gnus from scratch. This is not optimal,
>>> as all other relevant configurations will be lost, such as group levels,
>>> etc.
>>>
>>> This issue has happened before, it is not the first time.
>>
>> This definitely happens to many of us from time to time. Unfortunately I
>> can't really reproduce the problem, as by the time it appears it's too
>> late to figure out where it came from, though I assume it has to do with
>> Gnus calculating unread messages from a high-low range, and not being
>> aware of "filled in" read messages within that range.
>>
> You think this is a nnimap or a general Gnus issue ?

I don't know. I suspect that it's a general Gnus issue, but it is more
evident with nnimap, since that's pretty much the only (?) server where
local marks must be kept in sync with a remote server. In principle,
there's no reason why Gnus would need to keep local marks for imap
groups.

> Understanding how the newsrc file is written, should be next for me.

It's just, with their lists of marks. The lists are in "gnus range"
format (see gnus-range.el), which is just a compressed list of integers:
'(1 2 3 4 6 12 21 22 23) => '((1. 4) 6 12 (21. 23))

They are treated as sets, with lots of the usual set manipulation
functions in gnus-range.el. Part of the problem is the direct
manipulation of ranges that happens elsewhere in the codebase, typically
impenetrable thickets of "(setcdr (nthcdr 3 range) (cadar range)" etc
etc, often with little or no comments.

On my (long) list of things to do with Gnus is to write several more
macros for range manipulation, so that all the messy stuff happens
inside gnus-range.el, and the rest of the codebase is fairly readable.
My hope is that, in the course of that process, bugs will show
themselves up.

> Is there any type of tracing, or profiling to turn on, until this
> happens again ? Maybe set the gnus and back-end log levels to 10 ?

Nothing that would show you anything about range manipulation, I don't
think.

> The primary thing a mail, em..sorry.. news-reader, should be is stable.
> Not to have any corruption issues.

No argument here...

>> In the meantime, is "M-g" on the problematic group(s) enough to
>> permanently fix the problem? Not a great solution, though better than
>> doctoring your .newsrc.eld file...
>
> No difference. On testing your suggestion, the "M-g" was ignored as
> was/is the regular "g".

Sorry, I don't know where else to look. My only solution is the hard
one: clean up range manipulation until we can see what's going on.