bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Nix <nix@esperi.org.uk>]

On 7 Feb 2016, Lars Ingebrigtsen told this:

> Nick Alcock <nix@esperi.org.uk> writes:
>
>> This happened after my first Emacs update for nearly a year (a bit of a
>> 'big bang'), so it could have come from almost any change to nnimap
>> since September last year. I'll do a bisection and hunt for it. (It's
>> not like I can do any more damage to my imap read marks than I already
>> have! I wanted to get to 'inbox zero', but not this way...)
>
> Are you still seeing this problem?

Just retested with trunk. It still happens. Here's a smoking gun (on a
folder where I don't care about the read marks!). I marked no mails as
read in this time, just went into the group and out of it again.

Have a pile of debugging on the off-chance that you'll be able to figure
out the problem before I can even finish analyzing it :)

Enter group:

16:18:00 [stbeehive.oracle.com] 67 SELECT "Unrelated/Bugs"
16:18:01 [stbeehive.oracle.com] 68 SELECT "Unrelated/Bugs"
16:18:01 [stbeehive.oracle.com] 69 UID FETCH 4380,4910,5279,5788,6273,6790,7314,7610,7615,7619,7623,7629,7633,7637,7641,7645,7650,7653,7657,7661,7665,7669,7673,7677,7681,7685,7689,7695,7699,7703,7707,7711,7715,7720:7721,7725,7729,7733,7737,7741,7744,7749,7753,7757,7761,7765,7768:7769,7773,7777,7781:7782,7786,7790,7795,7802,7807,7812,7819,7824,7828,7832,7836,7841,7845,7849,7853,7857,7861,7865:7866,7869,7873,7877,7881,7885,7889,7894,7900,7904,7908,7911,7916,7920,7924,7928,7933,7937,7941,7945,7949,7954,7958,7962,7966,7971,7975,7979,7983,7987,7992,7996,8001,8005,8009,8013,8018,8022,8028,8032,8036,8040,8044,8048,8052,8059,8063,8067,8071,8075,8079,8085,8089,8093,8098,8102,8106,8110,8114,8118,8122,8126,8130,8134,8138,8142,8146,8150,8152,8156,8160,8164,8168,8172,8176,8180,8184:8185,8189,8194,8198,8202,8205,8212,8216,8220,8224,8228,8232,8237,8242,8248,8253:8254,8258,8262,8269,8275,8279,8283,8288,8294,8300,8304,8310,8315,8319,8323,8329,8333,8338,8342,8348,8352,8355,8360,8365,8367,8373,8377,8381,8386,8390,8395,8399,8404:8405,8409,8414,8419,8423,8427,8431,8436,8441,8445,8451,8455,8461:8462,8467,8471,8475,8479,8484,8488,8494,8498,8502,8506,8510,8514,8518,8522,8527,8531,8535,8540,8544,8548,8552,8556,8560,8564,8568,8572:8573,8577,8581,8584,8588,8591:8592,8596:8597,8601,8606,8610,8615,8619,8623,8628,8632,8636,8642,8646,8650,8655,8659,8669,8673,8677,8681,8685,8689,8694,8700,8705,8713,8717 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Gcc Newsgroups X-GM-LABELS)])

Exit group, having read not a single article:

16:18:26 [stbeehive.oracle.com] 70 SELECT "Unrelated/Bugs"
16:18:26 [stbeehive.oracle.com] 71 UID STORE 4910,5788,6790,7610,7619,7629,7637,7645,7653,7661,7669,7677,7685,7695,7703,7711,7720,7725,7733,7741,7749,7757,7765,7773,7781,7786,7795,7807,7819,7828,7836,7845,7853,7861,7869,7877,7885,7894,7904,7911,7920,7928,7937,7945,7954,7962,7971,7979,7987,7996,8005,8013,8022,8032,8040,8048,8059,8067,8075,8085,8093,8102,8110,8118,8126,8134,8142,8150,8156,8164,8172,8180,8189,8198,8205,8216,8224,8232,8242,8253,8258,8269,8279,8288,8300,8310,8319,8329,8338,8348,8355,8365,8373,8381,8390,8399,8409,8419,8427,8436,8445,8455,8467,8475,8484,8494,8502,8510,8518,8527,8535,8544,8552,8560,8568,8577,8584,8591,8596,8601,8610,8619,8628,8636,8646,8655,8669,8677,8685,8694,8705,8717 +FLAGS.SILENT (\Seen)

Note that it appears to believe the thing has X-GM-LABELS. This is
unfortunate, given that (nnimap-capabilities nnimap-object) returns
("IMAP4REV1" "IDLE" "AUTH=PLAIN"). No X-GM-EXT1.

The articles received, according to the " *nnimap ..." buffer (which I
had to extract from nnimap-process-buffers, I couldn't see it in the
buffer list, what's up with that?) all have initial headers that look
like this, and oh look X-GM-LABELS is there:

Chars: 3533
Lines: 39
X-GM-LABELS: )] {313}

and that latter header frankly looks like something very fubared that
Gnus has added itself. In the absence of the buggy patch, we see these
headers (content redacted):

Chars: 8196
Lines: 148
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1aOM3t-0004jM-O0@verbose.us.oracle.com>
To: [...]

In its presence, we see this:

Chars: 8196
Lines: 179
X-GM-LABELS: )] {278}
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1aOM3t-0004jM-O0@verbose.us.oracle.com>
To: [...]

)

Note the trailing ), extra X-GM-LABELS, and adjusted lines and length:
note also the unchanged Message-ID.

nnimap-header-parameters returns the exact same content before and after
the patch, so that can't be the problem. nnimap-transform-headers is
clearly fubared. If I add a message call into nnimap-transform-headers:

	(when (search-forward "X-GM-LABELS" (line-end-position) t)
	  (setq labels (ignore-errors (read (current-buffer))))
          (message "set labels to %s" labels))

what do I see but

set labels to )] {313}

Oops.

If I dump the line it's searching in, it becomes clear what's going on:

labels: * 4358 FETCH (UID 4380 RFC822.SIZE 3533 BODYSTRUCTURE ("TEXT" "PLAIN" NIL NIL NIL "8BIT" 1869 46 NIL ("INLINE" NIL) NIL NIL) BODY[HEADER.FIELDS ("Subject" "From" "Date" "Message-Id" "References" "In-Reply-To" "Xref" "X-Spam-Status" "To" "Cc" "Keywords" "Gcc" "Newsgroups" "X-GM-LABELS")] {313}

Youwhat? What's that even doing there? That's the header section: it
should have been erased!

Let's look at the *nnimap* buffer for that message right after the first
entry into the group with the buggy patch in place:

* 8196 FETCH (UID 8238 RFC822.SIZE 18125 BODYSTRUCTURE ("TEXT" "PLAIN" NIL NIL NIL "8BIT" 16480 378 NIL ("INLINE" NIL) NIL NIL) BODY[HEADER.FIELDS ("Subject" "From" "Date" "Message-Id" "References" "In-Reply-To" "Xref" "X-Spam-Status" "To" "Cc" "Keywords" "Gcc" "Newsgroups" "X-GM-LABELS")] {303}
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1aOM3t-0004jM-O0@verbose.us.oracle.com>
To: linuxrep_ww@oracle.com, linux_tools_us@oracle.com

)

Not erased. It's clear where the the Chars: 8196 above came from now,
and the X-GM-LABELS header is coming from the same place.

The nnimap buffer after transformation displays an alternating,
on-again, off-again character, which matches the "every other marked as
read" symptoms:

211 4380 Article retrieved.
Chars: 3533
Lines: 14
X-GM-LABELS: )] {313}
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1VjhEK-0003wo-Dk@verbose.us.oracle.com>
To: [...]

)
* 7655 FETCH (UID 7697 RFC822.SIZE 2045 BODYSTRUCTURE (
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1ZRnyM-0002bO-BJ@verbose.us.oracle.com>
To: linuxrep_ww@oracle.com, linux_tools_us@oracle.com
.
* 7656 FETCH (UID 7698 RFC822.SIZE 2492 BODYSTRUCTURE ("TEXT" "PLAIN" NIL NIL NIL "8BIT" 851 30 NIL ("INLINE" NIL) NIL NIL) BODY[HEADER.FIELDS ("Subject" "From" "Date" "Message-Id" "References" "In-Reply-To" "Xref" "X-Spam-Status" "To" "Cc" "Keywords" "Gcc" "Newsgroups" "X-GM-LABELS")] {302}
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1ZRxL0-0004SK-8R@verbose.us.oracle.com>
To: linuxrep_ww@oracle.com, linux_tools_us@oracle.com

)

Note that the brackets in the FETCH are not balanced: the open bracket
after HEADER.FIELDS is closed, but never the BODY[] brackets nor the
FETCH (... list bracket.

-- 
NULL && (void)