bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Nick Alcock]

The bug is fairly easily characterized, and easy to spot in a group with
a great many consecutive unread messages: every time the group is
refreshed, half the unread messages are marked as read, in an
alternating pattern, so after one refresh the result runs (... unread,
read, unread, read...): after more than one refresh the result is harder
to see.

After the first instance of this problem, we also see valid messages
apparently originating from nobody (their name appears as '(none)', and
other similarly bizarre messups indicating something very wrong with
IMAP communication. (Cc:ing Drew because the thing I was communicating
with was an SSL channel to the very same Beehive mailserver he uses -- I
know Drew doesn't use Gnus to post here, but maybe he sometimes looks at
his work mailbox via Gnus, in which case he might appreciate a warning!)

This happened after my first Emacs update for nearly a year (a bit of a
'big bang'), so it could have come from almost any change to nnimap
since September last year. I'll do a bisection and hunt for it. (It's
not like I can do any more damage to my imap read marks than I already
have! I wanted to get to 'inbox zero', but not this way...)

In GNU Emacs 25.0.50.1 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2015-08-01 on spindle
Repository revision: 08d65696c947b09d4c32606f279c3c2594e99e16
Windowing system distributor `The X.Org Foundation', version 11.0.11702000
System LSB Version:	:core-4.1-amd64:core-4.1-noarch

Configured using:
 `configure --without-pop --without-kerberos --without-hesiod
 --with-x-toolkit=lucid --with-wide-int --with-file-notification=inotify
 NO_FAST_MATH=t 'CFLAGS=-Wstrict-overflow=1 -g -gdwarf-4
 -feliminate-unused-debug-types -fvar-tracking-assignments
 -feliminate-dwarf2-dups -O2 -pipe -D__NO_STRING_INLINES
 -D__NO_MATH_INLINES -D_LARGEFILE64_SOURCE'
 'CPPFLAGS=-D__NO_STRING_INLINES -D__NO_MATH_INLINES
 -D_LARGEFILE64_SOURCE' 'LDFLAGS=-Wl,--enable-new-dtags
 -Wl,--hash-style=gnu -Wl,-O1 -Wl,-z,relro -pipe''

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS GCONF GSETTINGS
NOTIFY ACL GNUTLS LIBXML2 FREETYPE LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS
LUCID X11

Important settings:
  value of $LC_COLLATE: C
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Summary

Minor modes in effect:
  timeclock-mode-line-display: t
  TeX-PDF-mode: t
  magit-wip-after-apply-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  diff-auto-refine-mode: t
  global-semantic-mru-bookmark-mode: t
  global-semanticdb-minor-mode: t
  global-semantic-decoration-mode: t
  global-semantic-idle-scheduler-mode: t
  global-semantic-idle-local-symbol-highlight-mode: t
  global-semantic-idle-summary-mode: t
  global-srecode-minor-mode: t
  global-ede-mode: t
  global-semantic-highlight-func-mode: t
  global-semantic-stickyfunc-mode: t
  semantic-mode: t
  global-git-gutter-mode: t
  global-auto-complete-mode: t
  global-cwarn-mode: t
  which-function-mode: t
  mv-shell-mode: t
  desktop-save-mode: t
  savehist-mode: t
  winner-mode: t
  Info-breadcrumbs-in-mode-line-mode: t
  shell-dirtrack-mode: t
  recentf-mode: t
  display-time-mode: t
  override-global-mode: t
  image-diredx-async-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t

Recent messages:
Auto-saving...done
Auto-saving...done
Quit
Mark set
previous-line: Beginning of buffer [4 times]
Mark set
C-x C-g is undefined
C-M-g is undefined

Load-path shadows:
/home/nix/lisp/defaults hides /usr/share/emacs/site-lisp/defaults
/home/nix/lisp/emacs/scroll-in-place hides /usr/share/emacs/site-lisp/scroll-in-place
/usr/share/emacs/site-lisp/emms/tq hides /usr/share/emacs/25.0.50/lisp/emacs-lisp/tq

Features:
(shadow emacsbug canlock gnus-html flow-fill mm-archive eieio-opt
mailalias misearch multi-isearch nnir gnus-picon gnus-cite qp gnus-async
gnus-bcklg gnus-salt gnus-dup gnus-ml disp-table semantic/tag-write
gnus-topic nndraft gnus-agent gnus-srvr gnus-score score-mode nnvirtual
gnus-msg utf-7 gnutls nnmh nnfolder gnus-cache bbdb-gnus bbdb-snarf
gnus-demon nntp gnus/mail gnus-registry registry eieio-compat
gnus/splits nnmairix nnml smtpmail gnus-art mm-uu mml2015 mm-view
mml-smime smime dig gnus/articles init/time-tracking timeclock
auto-edit-substitute gnus/sa background gnus-sum gnus-group gnus-undo
gnus/bbdb gnus/colourization tc mail-extr gnus-start gnus-cloud nnimap
nnmail mail-source utf7 netrc nnoo parse-time gnus-spec gnus-int
gnus-range gnus-win face-remap add-log shr-color color network-stream
nsm starttls url-http tls url-gw url-cache url-auth eww mm-url gnus
gnus-ems nnheader url-queue shr dom subr-x browse-url epa-file epa epg
semantic/tag-file semantic/bovine/c hideif semantic/bovine/c-by
semantic/lex-spp semantic/bovine/gcc semantic/bovine
semantic/analyze/refs semantic/analyze semantic/scope
semantic/analyze/fcn vc-filewise vc-rcs generic tex-buf semantic/texi
tex-info tex dbus xml texinfo semantic/imenu semantic/sb semantic/sort
semantic/db-file data-debug cedet-files semantic/wisent/python
semantic/decorate/include semantic/dep semantic/wisent/python-wy rx
python json ede/dired jka-compr magit-wip magit-core magit-process
with-editor server magit-popup magit-mode magit-git crm magit-section
magit-utils dash undo-tree diff vc vc-dispatcher vc-git diff-mode
flymake checkdoc hideshow elisp-slime-nav init/emacs init/prog-modes
filecache inversion semantic/db-global semantic/symref/global
semantic/symref semantic/db-find semantic/db-ref cedet-global
semantic/mru-bookmark semantic/db-mode semantic/db
semantic/decorate/mode semantic/idle srecode/mode semantic/senator
semantic/format semantic/ctxt semantic/wisent semantic/wisent/wisent
semantic/decorate pulse srecode/insert srecode/filters srecode/args
srecode/find srecode/map srecode/ctxt semantic/tag-ls semantic/find
srecode/compile srecode/dictionary srecode/table ede/speedbar ede/files
ede ede/detect ede/base ede/auto ede/source eieio-base eieio-speedbar
speedbar sb-image ezimage dframe eieio-custom srecode
semantic/util-modes semantic/util semantic semantic/tag semantic/lex
semantic/fw mode-local cedet git-gutter-fringe fringe-helper git-gutter
auto-complete-config auto-complete popup cwarn gtags filladapt cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
cc-defs which-func generic-x init/processes vlf-setup mv-shell
init/music emms-volume emms-volume-amixer emms-history emms-bookmarks
emms-metaplaylist-mode emms-browser sort emms-playlist-sort
emms-last-played emms-playing-time emms-stream-info emms-streams
emms-mode-line emms-playlist-limit emms-playlist-mode emms-player-mpd
emms-url url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util mailcap emms-cache emms-info later-do tq
emms-player-simple emms-source-playlist emms-source-file locate emms
emms-compat emms-auto init/messaging bbdb-expire bbdb-hooks silly-mail
sendmail message rfc822 mml mml-sec mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mail-utils gmm-utils
mailheader bbdb-autoloads init/state desktop frameset savehist saveplace
init/editing gpicker finder find-func lisp-mnt windmove framemove
miniedit winner scroll-in-place flash-paren init/faces nix-dark-theme
init/drew two-column icicles icicles-mode dired+ image-file dired-aux
mb-depth+ mb-depth icicles-cmd2 compile imenu completion icicles-cmd1
bbdb-com warnings mailabbrev bbdb timezone yow etags xref project
cookie1 icicles-mcmd filesets icicles-fn icicles-var icicles-opt ffap
url-parse url-vars fuzzy-match el-swank-fuzzy levenshtein fuzzy
cus-theme cus-edit cus-start cus-load hexrgb icicles-face icicles-mac
bookmark+ bookmark+-key derived dired-x crosshairs col-highlight vline
hl-line+ hl-line bookmark+-1 bookmark+-bmu bookmark+-lit bookmark
menu-bar+ linkd doremi apropos-fn+var help-fns+ help+ frame-fns avoid
info+ thingatpt+ thingatpt mouse3 naked apropos+ apropos init/appearance
tramp-cache tramp-sh tramp tramp-compat auth-source eieio byte-opt
eieio-core gnus-util mm-util help-fns mail-prsvr password-cache
tramp-loaddefs trampver shell pcomplete comint ansi-color ring recentf
tree-widget wid-edit printing ps-print ps-def lpr time-date
init/modelines time uptimes pp init/options xemacs-compat misc
gawd-lists use-package-external use-package diminish bytecomp
byte-compile cl-extra help-mode seq cconv bind-key easy-mmode finder-inf
tex-site iedit-autoloads image-dired+ image-dired format-spec dired info
yasnippet-bundle cl-macs cl-seq dropdown-list advice yasnippet edmacro
kmacro easymenu assoc cl gv cl-loaddefs pcase cl-lib package epg-config
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 2245900 1795918)
 (symbols 48 141793 238)
 (miscs 40 6910 9463)
 (strings 32 782235 1143702)
 (string-bytes 1 34552691)
 (vectors 16 100813)
 (vector-slots 8 3950744 1469730)
 (floats 8 2600 6311)
 (intervals 56 50025 16112)
 (buffers 976 145)
 (heap 1024 313480 457073))

bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Nix]

I reported:

> This happened after my first Emacs update for nearly a year (a bit of a
> 'big bang'), so it could have come from almost any change to nnimap
> since September last year. I'll do a bisection and hunt for it.

The first bad commit is

commit 242354a23acf214ad06d4e3e7e5f5580c8b21d4a (HEAD, refs/bisect/bad)
Author: Trevor Murphy <trevor.m.murphy@gmail.com>
Date:   Mon Jan 26 07:56:37 2015 +0000

    lisp/gnus/nnimap.el Allow using the Google X-GM-LABELS, if present

and indeed reverting this causes the devastating automatically-mark-
every-other-item-as-read behaviour to go away. I am *not* using gmail,
but Beehive, so clearly this is negatively affecting other imap servers,
when it should obviously have no effect on them.

I'm frankly surprised a bug this serious has persisted for half a year.
It makes me wonder if I'm the only person using trunk Emacs with non-
Google imap servers, or if this is something specific to the (rare)
Beehive server.

Now, if you'll excuse me, I have a huge inbox full of
wrongly-marked-as-read messages to sort through :(

-- 
NULL && (void)

bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Lars Ingebrigtsen]

Nix <nix@esperi.org.uk> writes:

> The first bad commit is
>
> commit 242354a23acf214ad06d4e3e7e5f5580c8b21d4a (HEAD, refs/bisect/bad)
> Author: Trevor Murphy <trevor.m.murphy@gmail.com>
> Date:   Mon Jan 26 07:56:37 2015 +0000
>
>     lisp/gnus/nnimap.el Allow using the Google X-GM-LABELS, if present
>
> and indeed reverting this causes the devastating automatically-mark-
> every-other-item-as-read behaviour to go away. I am *not* using gmail,
> but Beehive, so clearly this is negatively affecting other imap servers,
> when it should obviously have no effect on them.

Huh.  That patch should have no effect on any servers that say nil to
(nnimap-capability "X-GM-EXT-1"), unless I'm reading it wrong.

And nnimap is not behaving on this strange way for me, and I don't use
Gmail, so I don't quite understand this.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Lars Ingebrigtsen]

Nick Alcock <nix@esperi.org.uk> writes:

> This happened after my first Emacs update for nearly a year (a bit of a
> 'big bang'), so it could have come from almost any change to nnimap
> since September last year. I'll do a bisection and hunt for it. (It's
> not like I can do any more damage to my imap read marks than I already
> have! I wanted to get to 'inbox zero', but not this way...)

Are you still seeing this problem?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Nix]

On 7 Feb 2016, Lars Ingebrigtsen told this:

> Nick Alcock <nix@esperi.org.uk> writes:
>
>> This happened after my first Emacs update for nearly a year (a bit of a
>> 'big bang'), so it could have come from almost any change to nnimap
>> since September last year. I'll do a bisection and hunt for it. (It's
>> not like I can do any more damage to my imap read marks than I already
>> have! I wanted to get to 'inbox zero', but not this way...)
>
> Are you still seeing this problem?

It's hard to say -- I reverted the patch locally immediately. The
problem is that whenever this goes wrong, 50% of my unread email gets
marked as read, with no obvious way to reverse this. This is rather a
high price to pay for replicating a bug :)

I have some sacrificial folders I could give up to the cause -- I'll see
if I can make things go wrong with them in the next day or so.

-- 
NULL && (void)

bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Lars Ingebrigtsen]

Nix <nix@esperi.org.uk> writes:

> It's hard to say -- I reverted the patch locally immediately. The
> problem is that whenever this goes wrong, 50% of my unread email gets
> marked as read, with no obvious way to reverse this. This is rather a
> high price to pay for replicating a bug :)
>
> I have some sacrificial folders I could give up to the cause -- I'll see
> if I can make things go wrong with them in the next day or so.

I just don't understand how the following patch could have this
effect...  

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index ced5561..8e81abc 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -166,14 +166,21 @@ nnimap-buffer
   (nnimap-find-process-buffer nntp-server-buffer))
 
 (defun nnimap-header-parameters ()
-  (format "(UID RFC822.SIZE BODYSTRUCTURE %s)"
-	  (format
+  (let (params)
+    (push "UID" params)
+    (push "RFC822.SIZE" params)
+    (when (nnimap-capability "X-GM-EXT-1")
+      (push "X-GM-LABELS" params))

So it only adds that to the params if the server supports it...

+    (push "BODYSTRUCTURE" params)
+    (push (format
 	   (if (nnimap-ver4-p)
 	       "BODY.PEEK[HEADER.FIELDS %s]"
 	     "RFC822.HEADER.LINES %s")
 	   (append '(Subject From Date Message-Id
 			     References In-Reply-To Xref)
-		   nnmail-extra-headers))))
+		   nnmail-extra-headers))
+	  params)
+    (format "%s" (nreverse params))))

So on non-Gmail servers the result should be identical...
 
-  (let (article lines size string)
+  (let (article lines size string labels)
     (block nil
       (while (not (eobp))
 	(while (not (looking-at "\\* [0-9]+ FETCH"))
@@ -232,6 +239,9 @@ nnimap-transform-headers
 				      t)
 		   (match-string 1)))
 	(beginning-of-line)
+	(when (search-forward "X-GM-LABELS" (line-end-position) t)
+	  (setq labels (ignore-errors (read (current-buffer)))))
+	(beginning-of-line)

Again, it should have no effect on non-Gmail...

 	(when lines
 	  (insert (format "Lines: %s\n" lines)))
+	(when labels
+	  (insert (format "X-GM-LABELS: %s\n" labels)))
 	;; Most servers have a blank line after the headers, but
 	;; Davmail doesn't.
 	(unless (re-search-forward "^\r$\\|^)\r?$" nil t)

And ditto.  So it's a mystery to me why that patch should affect
anything unless you're using Gmail..

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Paul Eggert]

From: Lars Ingebrigtsen <larsi@gnus.org>
> +    (when (nnimap-capability "X-GM-EXT-1")
> +      (push "X-GM-LABELS" params))
>
> So it only adds that to the params if the server supports it...

Suppose the newly-introduced call to nnimap-capability throws an exception or 
signals an error? Could that introduce the symptoms in question?

(I don't really understand the code, just trying to help move this along...)

bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Nix]

On 7 Feb 2016, Lars Ingebrigtsen told this:

> Nick Alcock <nix@esperi.org.uk> writes:
>
>> This happened after my first Emacs update for nearly a year (a bit of a
>> 'big bang'), so it could have come from almost any change to nnimap
>> since September last year. I'll do a bisection and hunt for it. (It's
>> not like I can do any more damage to my imap read marks than I already
>> have! I wanted to get to 'inbox zero', but not this way...)
>
> Are you still seeing this problem?

Just retested with trunk. It still happens. Here's a smoking gun (on a
folder where I don't care about the read marks!). I marked no mails as
read in this time, just went into the group and out of it again.

Have a pile of debugging on the off-chance that you'll be able to figure
out the problem before I can even finish analyzing it :)

Enter group:

16:18:00 [stbeehive.oracle.com] 67 SELECT "Unrelated/Bugs"
16:18:01 [stbeehive.oracle.com] 68 SELECT "Unrelated/Bugs"
16:18:01 [stbeehive.oracle.com] 69 UID FETCH 4380,4910,5279,5788,6273,6790,7314,7610,7615,7619,7623,7629,7633,7637,7641,7645,7650,7653,7657,7661,7665,7669,7673,7677,7681,7685,7689,7695,7699,7703,7707,7711,7715,7720:7721,7725,7729,7733,7737,7741,7744,7749,7753,7757,7761,7765,7768:7769,7773,7777,7781:7782,7786,7790,7795,7802,7807,7812,7819,7824,7828,7832,7836,7841,7845,7849,7853,7857,7861,7865:7866,7869,7873,7877,7881,7885,7889,7894,7900,7904,7908,7911,7916,7920,7924,7928,7933,7937,7941,7945,7949,7954,7958,7962,7966,7971,7975,7979,7983,7987,7992,7996,8001,8005,8009,8013,8018,8022,8028,8032,8036,8040,8044,8048,8052,8059,8063,8067,8071,8075,8079,8085,8089,8093,8098,8102,8106,8110,8114,8118,8122,8126,8130,8134,8138,8142,8146,8150,8152,8156,8160,8164,8168,8172,8176,8180,8184:8185,8189,8194,8198,8202,8205,8212,8216,8220,8224,8228,8232,8237,8242,8248,8253:8254,8258,8262,8269,8275,8279,8283,8288,8294,8300,8304,8310,8315,8319,8323,8329,8333,8338,8342,8348,8352,8355,8360,8365,8367,8373,8377,8381,8386,8390,8395,8399,8404:8405,8409,8414,8419,8423,8427,8431,8436,8441,8445,8451,8455,8461:8462,8467,8471,8475,8479,8484,8488,8494,8498,8502,8506,8510,8514,8518,8522,8527,8531,8535,8540,8544,8548,8552,8556,8560,8564,8568,8572:8573,8577,8581,8584,8588,8591:8592,8596:8597,8601,8606,8610,8615,8619,8623,8628,8632,8636,8642,8646,8650,8655,8659,8669,8673,8677,8681,8685,8689,8694,8700,8705,8713,8717 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Gcc Newsgroups X-GM-LABELS)])

Exit group, having read not a single article:

16:18:26 [stbeehive.oracle.com] 70 SELECT "Unrelated/Bugs"
16:18:26 [stbeehive.oracle.com] 71 UID STORE 4910,5788,6790,7610,7619,7629,7637,7645,7653,7661,7669,7677,7685,7695,7703,7711,7720,7725,7733,7741,7749,7757,7765,7773,7781,7786,7795,7807,7819,7828,7836,7845,7853,7861,7869,7877,7885,7894,7904,7911,7920,7928,7937,7945,7954,7962,7971,7979,7987,7996,8005,8013,8022,8032,8040,8048,8059,8067,8075,8085,8093,8102,8110,8118,8126,8134,8142,8150,8156,8164,8172,8180,8189,8198,8205,8216,8224,8232,8242,8253,8258,8269,8279,8288,8300,8310,8319,8329,8338,8348,8355,8365,8373,8381,8390,8399,8409,8419,8427,8436,8445,8455,8467,8475,8484,8494,8502,8510,8518,8527,8535,8544,8552,8560,8568,8577,8584,8591,8596,8601,8610,8619,8628,8636,8646,8655,8669,8677,8685,8694,8705,8717 +FLAGS.SILENT (\Seen)

Note that it appears to believe the thing has X-GM-LABELS. This is
unfortunate, given that (nnimap-capabilities nnimap-object) returns
("IMAP4REV1" "IDLE" "AUTH=PLAIN"). No X-GM-EXT1.

The articles received, according to the " *nnimap ..." buffer (which I
had to extract from nnimap-process-buffers, I couldn't see it in the
buffer list, what's up with that?) all have initial headers that look
like this, and oh look X-GM-LABELS is there:

Chars: 3533
Lines: 39
X-GM-LABELS: )] {313}

and that latter header frankly looks like something very fubared that
Gnus has added itself. In the absence of the buggy patch, we see these
headers (content redacted):

Chars: 8196
Lines: 148
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1aOM3t-0004jM-O0@verbose.us.oracle.com>
To: [...]

In its presence, we see this:

Chars: 8196
Lines: 179
X-GM-LABELS: )] {278}
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1aOM3t-0004jM-O0@verbose.us.oracle.com>
To: [...]

)

Note the trailing ), extra X-GM-LABELS, and adjusted lines and length:
note also the unchanged Message-ID.

nnimap-header-parameters returns the exact same content before and after
the patch, so that can't be the problem. nnimap-transform-headers is
clearly fubared. If I add a message call into nnimap-transform-headers:

	(when (search-forward "X-GM-LABELS" (line-end-position) t)
	  (setq labels (ignore-errors (read (current-buffer))))
          (message "set labels to %s" labels))

what do I see but

set labels to )] {313}

Oops.

If I dump the line it's searching in, it becomes clear what's going on:

labels: * 4358 FETCH (UID 4380 RFC822.SIZE 3533 BODYSTRUCTURE ("TEXT" "PLAIN" NIL NIL NIL "8BIT" 1869 46 NIL ("INLINE" NIL) NIL NIL) BODY[HEADER.FIELDS ("Subject" "From" "Date" "Message-Id" "References" "In-Reply-To" "Xref" "X-Spam-Status" "To" "Cc" "Keywords" "Gcc" "Newsgroups" "X-GM-LABELS")] {313}

Youwhat? What's that even doing there? That's the header section: it
should have been erased!

Let's look at the *nnimap* buffer for that message right after the first
entry into the group with the buggy patch in place:

* 8196 FETCH (UID 8238 RFC822.SIZE 18125 BODYSTRUCTURE ("TEXT" "PLAIN" NIL NIL NIL "8BIT" 16480 378 NIL ("INLINE" NIL) NIL NIL) BODY[HEADER.FIELDS ("Subject" "From" "Date" "Message-Id" "References" "In-Reply-To" "Xref" "X-Spam-Status" "To" "Cc" "Keywords" "Gcc" "Newsgroups" "X-GM-LABELS")] {303}
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1aOM3t-0004jM-O0@verbose.us.oracle.com>
To: linuxrep_ww@oracle.com, linux_tools_us@oracle.com

)

Not erased. It's clear where the the Chars: 8196 above came from now,
and the X-GM-LABELS header is coming from the same place.

The nnimap buffer after transformation displays an alternating,
on-again, off-again character, which matches the "every other marked as
read" symptoms:

211 4380 Article retrieved.
Chars: 3533
Lines: 14
X-GM-LABELS: )] {313}
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1VjhEK-0003wo-Dk@verbose.us.oracle.com>
To: [...]

)
* 7655 FETCH (UID 7697 RFC822.SIZE 2045 BODYSTRUCTURE (
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1ZRnyM-0002bO-BJ@verbose.us.oracle.com>
To: linuxrep_ww@oracle.com, linux_tools_us@oracle.com
.
* 7656 FETCH (UID 7698 RFC822.SIZE 2492 BODYSTRUCTURE ("TEXT" "PLAIN" NIL NIL NIL "8BIT" 851 30 NIL ("INLINE" NIL) NIL NIL) BODY[HEADER.FIELDS ("Subject" "From" "Date" "Message-Id" "References" "In-Reply-To" "Xref" "X-Spam-Status" "To" "Cc" "Keywords" "Gcc" "Newsgroups" "X-GM-LABELS")] {302}
Subject: [...]
From: [...]
Date: [...]
Message-id: <E1ZRxL0-0004SK-8R@verbose.us.oracle.com>
To: linuxrep_ww@oracle.com, linux_tools_us@oracle.com

)

Note that the brackets in the FETCH are not balanced: the open bracket
after HEADER.FIELDS is closed, but never the BODY[] brackets nor the
FETCH (... list bracket.

-- 
NULL && (void)

Re: bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[joyperkins07]

On Monday, August 3, 2015 at 8:02:19 AM UTC-5, Nick Alcock wrote:
> The bug is fairly easily characterized, and easy to spot in a group with
> a great many consecutive unread messages: every time the group is
> refreshed, half the unread messages are marked as read, in an
> alternating pattern, so after one refresh the result runs (... unread,
> read, unread, read...): after more than one refresh the result is harder
> to see.
> 
> After the first instance of this problem, we also see valid messages
> apparently originating from nobody (their name appears as '(none)', and
> other similarly bizarre messups indicating something very wrong with
> IMAP communication. (Cc:ing Drew because the thing I was communicating
> with was an SSL channel to the very same Beehive mailserver he uses -- I
> know Drew doesn't use Gnus to post here, but maybe he sometimes looks at
> his work mailbox via Gnus, in which case he might appreciate a warning!)
> 
> This happened after my first Emacs update for nearly a year (a bit of a
> 'big bang'), so it could have come from almost any change to nnimap
> since September last year. I'll do a bisection and hunt for it. (It's
> not like I can do any more damage to my imap read marks than I already
> have! I wanted to get to 'inbox zero', but not this way...)
> 
> In GNU Emacs 25.0.50.1 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d scroll bars)
>  of 2015-08-01 on spindle
> Repository revision: 08d65696c947b09d4c32606f279c3c2594e99e16
> Windowing system distributor `The X.Org Foundation', version 11.0.11702000
> System LSB Version:	:core-4.1-amd64:core-4.1-noarch
> 
> Configured using:
>  `configure --without-pop --without-kerberos --without-hesiod
>  --with-x-toolkit=lucid --with-wide-int --with-file-notification=inotify
>  NO_FAST_MATH=t 'CFLAGS=-Wstrict-overflow=1 -g -gdwarf-4
>  -feliminate-unused-debug-types -fvar-tracking-assignments
>  -feliminate-dwarf2-dups -O2 -pipe -D__NO_STRING_INLINES
>  -D__NO_MATH_INLINES -D_LARGEFILE64_SOURCE'
>  'CPPFLAGS=-D__NO_STRING_INLINES -D__NO_MATH_INLINES
>  -D_LARGEFILE64_SOURCE' 'LDFLAGS=-Wl,--enable-new-dtags
>  -Wl,--hash-style=gnu -Wl,-O1 -Wl,-z,relro -pipe''
> 
> Configured features:
> XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS GCONF GSETTINGS
> NOTIFY ACL GNUTLS LIBXML2 FREETYPE LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS
> LUCID X11
> 
> Important settings:
>   value of $LC_COLLATE: C
>   value of $LANG: en_GB.UTF-8
>   locale-coding-system: utf-8-unix
> 
> Major mode: Summary
> 
> Minor modes in effect:
>   timeclock-mode-line-display: t
>   TeX-PDF-mode: t
>   magit-wip-after-apply-mode: t
>   global-undo-tree-mode: t
>   undo-tree-mode: t
>   diff-auto-refine-mode: t
>   global-semantic-mru-bookmark-mode: t
>   global-semanticdb-minor-mode: t
>   global-semantic-decoration-mode: t
>   global-semantic-idle-scheduler-mode: t
>   global-semantic-idle-local-symbol-highlight-mode: t
>   global-semantic-idle-summary-mode: t
>   global-srecode-minor-mode: t
>   global-ede-mode: t
>   global-semantic-highlight-func-mode: t
>   global-semantic-stickyfunc-mode: t
>   semantic-mode: t
>   global-git-gutter-mode: t
>   global-auto-complete-mode: t
>   global-cwarn-mode: t
>   which-function-mode: t
>   mv-shell-mode: t
>   desktop-save-mode: t
>   savehist-mode: t
>   winner-mode: t
>   Info-breadcrumbs-in-mode-line-mode: t
>   shell-dirtrack-mode: t
>   recentf-mode: t
>   display-time-mode: t
>   override-global-mode: t
>   image-diredx-async-mode: t
>   tooltip-mode: t
>   global-eldoc-mode: t
>   electric-indent-mode: t
>   mouse-wheel-mode: t
>   menu-bar-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   blink-cursor-mode: t
>   auto-composition-mode: t
>   auto-encryption-mode: t
>   auto-compression-mode: t
>   buffer-read-only: t
>   column-number-mode: t
>   line-number-mode: t
> 
> Recent messages:
> Auto-saving...done
> Auto-saving...done
> Quit
> Mark set
> previous-line: Beginning of buffer [4 times]
> Mark set
> C-x C-g is undefined
> C-M-g is undefined
> 
> Load-path shadows:
> /home/nix/lisp/defaults hides /usr/share/emacs/site-lisp/defaults
> /home/nix/lisp/emacs/scroll-in-place hides /usr/share/emacs/site-lisp/scroll-in-place
> /usr/share/emacs/site-lisp/emms/tq hides /usr/share/emacs/25.0.50/lisp/emacs-lisp/tq
> 
> Features:
> (shadow emacsbug canlock gnus-html flow-fill mm-archive eieio-opt
> mailalias misearch multi-isearch nnir gnus-picon gnus-cite qp gnus-async
> gnus-bcklg gnus-salt gnus-dup gnus-ml disp-table semantic/tag-write
> gnus-topic nndraft gnus-agent gnus-srvr gnus-score score-mode nnvirtual
> gnus-msg utf-7 gnutls nnmh nnfolder gnus-cache bbdb-gnus bbdb-snarf
> gnus-demon nntp gnus/mail gnus-registry registry eieio-compat
> gnus/splits nnmairix nnml smtpmail gnus-art mm-uu mml2015 mm-view
> mml-smime smime dig gnus/articles init/time-tracking timeclock
> auto-edit-substitute gnus/sa background gnus-sum gnus-group gnus-undo
> gnus/bbdb gnus/colourization tc mail-extr gnus-start gnus-cloud nnimap
> nnmail mail-source utf7 netrc nnoo parse-time gnus-spec gnus-int
> gnus-range gnus-win face-remap add-log shr-color color network-stream
> nsm starttls url-http tls url-gw url-cache url-auth eww mm-url gnus
> gnus-ems nnheader url-queue shr dom subr-x browse-url epa-file epa epg
> semantic/tag-file semantic/bovine/c hideif semantic/bovine/c-by
> semantic/lex-spp semantic/bovine/gcc semantic/bovine
> semantic/analyze/refs semantic/analyze semantic/scope
> semantic/analyze/fcn vc-filewise vc-rcs generic tex-buf semantic/texi
> tex-info tex dbus xml texinfo semantic/imenu semantic/sb semantic/sort
> semantic/db-file data-debug cedet-files semantic/wisent/python
> semantic/decorate/include semantic/dep semantic/wisent/python-wy rx
> python json ede/dired jka-compr magit-wip magit-core magit-process
> with-editor server magit-popup magit-mode magit-git crm magit-section
> magit-utils dash undo-tree diff vc vc-dispatcher vc-git diff-mode
> flymake checkdoc hideshow elisp-slime-nav init/emacs init/prog-modes
> filecache inversion semantic/db-global semantic/symref/global
> semantic/symref semantic/db-find semantic/db-ref cedet-global
> semantic/mru-bookmark semantic/db-mode semantic/db
> semantic/decorate/mode semantic/idle srecode/mode semantic/senator
> semantic/format semantic/ctxt semantic/wisent semantic/wisent/wisent
> semantic/decorate pulse srecode/insert srecode/filters srecode/args
> srecode/find srecode/map srecode/ctxt semantic/tag-ls semantic/find
> srecode/compile srecode/dictionary srecode/table ede/speedbar ede/files
> ede ede/detect ede/base ede/auto ede/source eieio-base eieio-speedbar
> speedbar sb-image ezimage dframe eieio-custom srecode
> semantic/util-modes semantic/util semantic semantic/tag semantic/lex
> semantic/fw mode-local cedet git-gutter-fringe fringe-helper git-gutter
> auto-complete-config auto-complete popup cwarn gtags filladapt cc-mode
> cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
> cc-defs which-func generic-x init/processes vlf-setup mv-shell
> init/music emms-volume emms-volume-amixer emms-history emms-bookmarks
> emms-metaplaylist-mode emms-browser sort emms-playlist-sort
> emms-last-played emms-playing-time emms-stream-info emms-streams
> emms-mode-line emms-playlist-limit emms-playlist-mode emms-player-mpd
> emms-url url url-proxy url-privacy url-expand url-methods url-history
> url-cookie url-domsuf url-util mailcap emms-cache emms-info later-do tq
> emms-player-simple emms-source-playlist emms-source-file locate emms
> emms-compat emms-auto init/messaging bbdb-expire bbdb-hooks silly-mail
> sendmail message rfc822 mml mml-sec mm-decode mm-bodies mm-encode
> mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mail-utils gmm-utils
> mailheader bbdb-autoloads init/state desktop frameset savehist saveplace
> init/editing gpicker finder find-func lisp-mnt windmove framemove
> miniedit winner scroll-in-place flash-paren init/faces nix-dark-theme
> init/drew two-column icicles icicles-mode dired+ image-file dired-aux
> mb-depth+ mb-depth icicles-cmd2 compile imenu completion icicles-cmd1
> bbdb-com warnings mailabbrev bbdb timezone yow etags xref project
> cookie1 icicles-mcmd filesets icicles-fn icicles-var icicles-opt ffap
> url-parse url-vars fuzzy-match el-swank-fuzzy levenshtein fuzzy
> cus-theme cus-edit cus-start cus-load hexrgb icicles-face icicles-mac
> bookmark+ bookmark+-key derived dired-x crosshairs col-highlight vline
> hl-line+ hl-line bookmark+-1 bookmark+-bmu bookmark+-lit bookmark
> menu-bar+ linkd doremi apropos-fn+var help-fns+ help+ frame-fns avoid
> info+ thingatpt+ thingatpt mouse3 naked apropos+ apropos init/appearance
> tramp-cache tramp-sh tramp tramp-compat auth-source eieio byte-opt
> eieio-core gnus-util mm-util help-fns mail-prsvr password-cache
> tramp-loaddefs trampver shell pcomplete comint ansi-color ring recentf
> tree-widget wid-edit printing ps-print ps-def lpr time-date
> init/modelines time uptimes pp init/options xemacs-compat misc
> gawd-lists use-package-external use-package diminish bytecomp
> byte-compile cl-extra help-mode seq cconv bind-key easy-mmode finder-inf
> tex-site iedit-autoloads image-dired+ image-dired format-spec dired info
> yasnippet-bundle cl-macs cl-seq dropdown-list advice yasnippet edmacro
> kmacro easymenu assoc cl gv cl-loaddefs pcase cl-lib package epg-config
> mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
> lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd fontset
> image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
> prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
> mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
> georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
> korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
> european ethiopic indian cyrillic chinese charscript case-table epa-hook
> jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
> loaddefs button faces cus-face macroexp files text-properties overlay
> sha1 md5 base64 format env code-pages mule custom widget
> hashtable-print-readable backquote dbusbind inotify dynamic-setting
> system-font-setting font-render-setting x-toolkit x multi-tty
> make-network-process emacs)
> 
> Memory information:
> ((conses 16 2245900 1795918)
>  (symbols 48 141793 238)
>  (miscs 40 6910 9463)
>  (strings 32 782235 1143702)
>  (string-bytes 1 34552691)
>  (vectors 16 100813)
>  (vector-slots 8 3950744 1469730)
>  (floats 8 2600 6311)
>  (intervals 56 50025 16112)
>  (buffers 976 145)
>  (heap 1024 313480 457073))

bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Nix <nix@esperi.org.uk> writes:
>
>> The first bad commit is
>>
>> commit 242354a23acf214ad06d4e3e7e5f5580c8b21d4a (HEAD, refs/bisect/bad)
>> Author: Trevor Murphy <trevor.m.murphy@gmail.com>
>> Date:   Mon Jan 26 07:56:37 2015 +0000
>>
>>     lisp/gnus/nnimap.el Allow using the Google X-GM-LABELS, if present
>>
>> and indeed reverting this causes the devastating automatically-mark-
>> every-other-item-as-read behaviour to go away. I am *not* using gmail,
>> but Beehive, so clearly this is negatively affecting other imap servers,
>> when it should obviously have no effect on them.
>
> Huh.  That patch should have no effect on any servers that say nil to
> (nnimap-capability "X-GM-EXT-1"), unless I'm reading it wrong.
>
> And nnimap is not behaving on this strange way for me, and I don't use
> Gmail, so I don't quite understand this.

Are you still seeing this?  Nobody else has reported anything similar (I
think), so I suspect it's something particular with your setup or server...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21182: 25.0.50; gnus: every other unread message is marked as read on each nnimap group refresh

[Nix]

On 26 Jan 2017, Lars Ingebrigtsen spake thusly:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Nix <nix@esperi.org.uk> writes:
>>
>>> The first bad commit is
>>>
>>> commit 242354a23acf214ad06d4e3e7e5f5580c8b21d4a (HEAD, refs/bisect/bad)
>>> Author: Trevor Murphy <trevor.m.murphy@gmail.com>
>>> Date:   Mon Jan 26 07:56:37 2015 +0000
>>>
>>>     lisp/gnus/nnimap.el Allow using the Google X-GM-LABELS, if present
>>>
>>> and indeed reverting this causes the devastating automatically-mark-
>>> every-other-item-as-read behaviour to go away. I am *not* using gmail,
>>> but Beehive, so clearly this is negatively affecting other imap servers,
>>> when it should obviously have no effect on them.
>>
>> Huh.  That patch should have no effect on any servers that say nil to
>> (nnimap-capability "X-GM-EXT-1"), unless I'm reading it wrong.
>>
>> And nnimap is not behaving on this strange way for me, and I don't use
>> Gmail, so I don't quite understand this.
>
> Are you still seeing this?  Nobody else has reported anything similar (I
> think), so I suspect it's something particular with your setup or server...

I haven't retried for a few months: I'm running with that patch backed
out. I'll check again, but I think it's quite unlikely that the problem
has gone away. It was still present when I last checked, and nothing has
changed in this part of Gnus at all since then, at least not on the
Emacs 25 branch where I'm living.

I did a bunch more tracking down in the bug back in June last year: see
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21182#30>.

I would be utterly, cosmically unsurprised to find that the problem was
specific to the format of the IMAP sent back by the crawling horror
which is Beehive. (I could see if I could convince Drew Adams to set up
Gnus to talk to the Oracle external IMAP server, but I suspect that if I
tell him that this is to try to replicate a bug that makes every other
mail get marked as read he'll tell me to bugger off, he likes his read
states just the way they are!)

-- 
NULL && (void)