bug#17845: 24.2; nnimap performs 'intial sync' on all mailboxes every time 'M-x gnus' is run

bug#17845: 24.2; nnimap performs 'intial sync' on all mailboxes every time 'M-x gnus' is run

[Duncan Bayne]

I'm using nnimap, configured to use the Fastmail IMAP servers.  Every
time I do 'M-x gnus', nnimap performs an 'initial sync' of each mailbox,
downloading many MiB of data.

In addition to being very slow, this is probably irritating the folks at
Fastmail no end.

My gnus configuration is as follows:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(require 'gnus)

(setq user-mail-address "duncan@bayne.id.au")
(setq user-full-name "Duncan Bayne")

(setq gnus-select-method
      '(nnimap "duncan@bayne.id.au"
               (nnimap-address "mail.messagingengine.com")
               (nnimap-server-port 993)
               (nnimap-stream ssl)
               (nnimap-list-pattern ("INBOX" "INBOX.*"))))

(setq message-send-mail-function 'smtpmail-send-it
      smtpmail-starttls-credentials '(("mail.messagingengine.com" 587 nil nil))
      smtpmail-auth-credentials '(("mail.messagingengine.com" 587 "duncan@bayne.id.au" nil))
      smtpmail-default-smtp-server "mail.messagingengine.com"
      smtpmail-smtp-server "mail.messagingengine.com"
      smtpmail-smtp-service 587
      gnus-ignored-newsgroups "^to\\.\\|^[0-9. ]+\\(\\|$\\)\\|^[\"]\"[#'()]")

(defun my-message-mode-setup ()
  (setq fill-column 72)
  (turn-on-auto-fill))
(add-hook 'message-mode-hook 'my-message-mode-setup)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

In GNU Emacs 24.2.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.20)
 of 2014-02-27 on duncan-laptop
Windowing system distributor `The X.Org Foundation', version 11.0.11405000
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_AU.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Group

Minor modes in effect:
  gnus-undo-mode: t
  global-diff-hl-mode: t
  diff-auto-refine-mode: t
  global-auto-revert-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  global-hl-line-mode: t
  global-linum-mode: t
  linum-mode: t
  delete-selection-mode: t
  cua-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
M-x g n u s <return> n M-x r e p o <tab> r <tab> <
return>

Recent messages:
nnimap read 3065k from mail.messagingengine.com (initial sync of 30 groups; please wait)
nnimap read 3163k from mail.messagingengine.com (initial sync of 30 groups; please wait)
nnimap read 3331k from mail.messagingengine.com (initial sync of 30 groups; please wait)
nnimap read 3482k from mail.messagingengine.com (initial sync of 30 groups; please wait)
nnimap read 3622k from mail.messagingengine.com (initial sync of 30 groups; please wait)
nnimap read 3642k from mail.messagingengine.com (initial sync of 30 groups; please wait)
nnimap read 3761k from mail.messagingengine.com (initial sync of 30 groups; please wait)
Checking new news...done
Auto-saving...
Making completion list...

Load-path shadows:
~/.emacs.d/ruby-mode/ruby-mode hides /usr/local/share/emacs/24.2/lisp/progmodes/ruby-mode
~/.emacs.d/emms/tq hides /usr/local/share/emacs/24.2/lisp/emacs-lisp/tq

Features:
(shadow sort mail-extr emacsbug sendmail help-mode view nndraft nnmh
nnfolder utf-7 network-stream auth-source eieio starttls nnimap
parse-time tls utf7 netrc gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 epg-config mm-view mml-smime
smime password-cache dig mailcap nntp gnus-cache gnus-sum nnoo
gnus-group gnus-undo nnmail mail-source gnus-start gnus-spec gnus-int
gnus-range message format-spec rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
gmm-utils mailheader gnus-win uniquify diff-hl face-remap vc-hg vc-git
cl-lib vc-dir ewoc vc ediff-merg ediff-diff ediff-wind ediff-help
ediff-util ediff-mult ediff-init ediff vc-dispatcher diff-mode geiser
emotion-iconous-mode xml jabber-autoloads gnus gnus-ems nnheader
gnus-util mail-utils mm-util mail-prsvr wid-edit ws-trim tomatinho
autorevert find-file-in-project ido undo-tree slime pp hyperspec
browse-url multi-term warnings advice advice-preload term disp-table
ehelp electric jka-compr-ccrypt jka-compr hl-line linum delsel cua-base
php-mode etags speedbar sb-image ezimage dframe assoc yaml-mode
feature-mode cucumber-mode sass-mode coffee-mode comint ansi-color ring
js2-mode cc-langs haml-mode help-fns js byte-opt bytecomp byte-compile
cconv macroexp json imenu newcomment cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs markdown-mode
derived edmacro kmacro thingatpt noutline outline easy-mmode easymenu
css-mode ruby-mode regexp-opt cl duncans_emacs time-date tooltip
ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd
fontset image fringe lisp-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer loaddefs button faces cus-face files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty emacs)

bug#17845: 24.2; nnimap performs 'intial sync' on all mailboxes every time 'M-x gnus' is run

[David Engster]

Duncan Bayne writes:
> I'm using nnimap, configured to use the Fastmail IMAP servers.  Every
> time I do 'M-x gnus', nnimap performs an 'initial sync' of each mailbox,
> downloading many MiB of data.
>
> In addition to being very slow, this is probably irritating the folks at
> Fastmail no end.

This was fixed around July last year. Updating either to the latest Gnus
release (http://git.gnus.org/cgit/gnus.git/snapshot/gnus-m0-11.zip) or
to the latest Emacs pretest (24.3.91) should fix your problem. If not,
please reopen this bug.

-David

bug#17845: Re-opening; can repro on GNU Emacs 24.3.91.1

[Duncan Bayne]

I've upgraded to GNU Emacs 24.3.91.1, built from source.  The problem
persists; every time I start Emacs and do 'M-x gnus', it performs an
'initial sync' of all my IMAP folders.

Details from report-emacs-bug:

In GNU Emacs 24.3.91.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.17)
 of 2014-06-26 on duncan-greensync
Windowing system distributor `The X.Org Foundation', version
11.0.11303000
System Description:     Linux Mint 15 Olivia

Important settings:
  value of $LANG: en_AU.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  gnus-undo-mode: t
  global-diff-hl-mode: t
  diff-auto-refine-mode: t
  global-auto-revert-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  global-hl-line-mode: t
  global-linum-mode: t
  linum-mode: t
  delete-selection-mode: t
  cua-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
M-x g n u s <return> y M-x r e p o r <tab> <return
>

Recent messages:
Checking new news...
nnimap read 0k from mail.messagingengine.com (initial sync of 22 groups;
please wait) [5 times]
nnimap read 28k from mail.messagingengine.com (initial sync of 22
groups; please wait)
nnimap read 73k from mail.messagingengine.com (initial sync of 22
groups; please wait)
nnimap read 125k from mail.messagingengine.com (initial sync of 22
groups; please wait)
nnimap read 203k from mail.messagingengine.com (initial sync of 22
groups; please wait)
nnimap read 337k from mail.messagingengine.com (initial sync of 22
groups; please wait)
nnimap read 528k from mail.messagingengine.com (initial sync of 22
groups; please wait)
Reading active file via nndraft...done
Checking new news...done

Load-path shadows:
~/.emacs.d/ruby-mode/ruby-mode hides
/usr/local/share/emacs/24.3.91/lisp/progmodes/ruby-mode
~/.emacs.d/emms/tq hides
/usr/local/share/emacs/24.3.91/lisp/emacs-lisp/tq
/usr/local/share/emacs/24.3.91/lisp/emacs-lisp/cl-lib hides
~/.emacs.d/cl-lib/cl-lib

Features:
(shadow sort mail-extr emacsbug sendmail nndraft nnmh nnfolder utf-7
gnutls network-stream auth-source eieio eieio-core starttls nnimap
parse-time tls utf7 netrc gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 epg-config mm-view mml-smime
smime password-cache dig mailcap nntp gnus-cache gnus-sum nnoo
gnus-group gnus-undo nnmail mail-source gnus-start gnus-spec gnus-int
gnus-range message format-spec rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
gmm-utils mailheader gnus-win diff-hl face-remap vc-hg vc-git vc-dir
ewoc vc vc-dispatcher diff-mode geiser emotion-iconous-mode xml
jabber-autoloads gnus gnus-ems nnheader gnus-util mail-utils mm-util
mail-prsvr wid-edit ws-trim tomatinho autorevert filenotify
find-file-in-project warnings ido undo-tree slime pp hyperspec
browse-url multi-term term disp-table ehelp jka-compr-ccrypt jka-compr
hl-line linum delsel cua-base php-mode etags speedbar sb-image ezimage
dframe yaml-mode feature-mode cucumber-mode sass-mode coffee-mode comint
ansi-color ring js2-mode cc-langs haml-mode js advice help-fns byte-opt
bytecomp byte-compile cconv json imenu cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
markdown-mode derived edmacro kmacro thingatpt noutline outline
easy-mmode easymenu css-mode smie ruby-mode cl-macs cl gv cl-loaddefs
cl-lib duncans_emacs time-date tooltip electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process gfilenotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty emacs)

Memory information:
((conses 16 422701 21099)
 (symbols 48 41562 1)
 (miscs 40 114 212)
 (strings 32 71710 10626)
 (string-bytes 1 2301646)
 (vectors 16 27298)
 (vector-slots 8 581006 8205)
 (floats 8 294 249)
 (intervals 56 293 0)
 (buffers 960 23)
 (heap 1024 28463 2674))

bug#17845: Still happens with Emacs 24.4

[Rainer Orth]

I'm also seeing this issue on Solaris 11/x86 (i386-pc-solaris2.11): on
my workstation, it occured for the first time when switching from Emacs
24.3 to 24.4.  With a 100 Mbit/s connection to the IMAP server,
gnus/nnimap is barely usable.

The issue isn't new, though: on my Solaris 11/x86 laptop, it occured for
the first time when switching from Emacs 23.3 to 24.1, and in all
versions up to and including 24.4.  With slower network connectivity,
the Emacs 24 versions are completely unusable due to this bug, so I have
to stay at 23.3 there.

Any suggestions on how to further investigate this?  This bug is most
annoying.

Thanks.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

bug#17845: Re-opening; can repro on GNU Emacs 24.3.91.1

[Duncan Bayne]

I'm going to try to repro this tonight, with the latest Emacs and gnus,
and see if I can track down what's causing it.  I'm moving more and more
of my workflow into Emacs (I'm now an avid org-mode user) and this is
the 'last piece of the puzzle' for me.

bug#17845: Re-opening; can repro on GNU Emacs 24.3.91.1

[Lars Ingebrigtsen]

Duncan Bayne <duncan@bayne.id.au> writes:

> I've upgraded to GNU Emacs 24.3.91.1, built from source.  The problem
> persists; every time I start Emacs and do 'M-x gnus', it performs an
> 'initial sync' of all my IMAP folders.

If Gnus is doing the "initial sync" every time, then that means that
Gnus is not storing the fill IMAP data in the ~/.newsrc.eld file.

What does the `G E' data on one of these nnimap groups look like?
(`G E' is a group mode command.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#17845: Re-opening; can repro on GNU Emacs 24.3.91.1

[Duncan Bayne]

> If Gnus is doing the "initial sync" every time, then that means that
> Gnus is not storing the fill IMAP data in the ~/.newsrc.eld file.
 
That sounds reasonable, as the file ~/.newsrc.eld doesn't exist.

> What does the `G E' data on one of these nnimap groups look like?
> (`G E' is a group mode command.)

From running the above on INBOX:

;;; Editing the group info for `INBOX'.
;; Type `C-c C-c' after you've finished editing.

("INBOX" 3
 ((1 . 22064))
 ((reply 22015)
  (unexist
   (1 . 19759)
   (19761 . 19832)
   (19834 . 20409)
   (20411 . 20681)
   (20683 . 21194)
   (21197 . 21970)
   (21972 . 22014)
   (22016 . 22019)
   (22021 . 22027)
   (22029 . 22050)
   (22054 . 22058)
   (22060 . 22062)))
 nil
 ((modseq . "175357")
  (uidvalidity . "1366435887")
  (active 19760 . 22066)
  (permanent-flags %Answered %Flagged %Draft %Deleted %Seen
  $X-ME-Annot-2 $IsMailingList $IsNotification $HasAttachment $HasTD
  $NotJunk $Forwarded $Junk $IsTrusted %*)))

-- 
Duncan Bayne
ph: +61 420817082 | web: http://duncan-bayne.github.com/ | skype:
duncan_bayne

I usually check my mail every 24 - 48 hours.  If there's something
urgent going on, please send me an SMS or call me.

bug#17845: Re-opening; can repro on GNU Emacs 24.3.91.1

[Lars Ingebrigtsen]

Duncan Bayne <duncan@bayne.id.au> writes:

>> If Gnus is doing the "initial sync" every time, then that means that
>> Gnus is not storing the fill IMAP data in the ~/.newsrc.eld file.
>
> That sounds reasonable, as the file ~/.newsrc.eld doesn't exist.

If that file doesn't exist, then that's your problem.  Are you exiting
Gnus without saving anything?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#17845: Re-opening; can repro on GNU Emacs 24.3.91.1

[Duncan Bayne]

On Sat, Feb 14, 2015, at 08:07 AM, Duncan Bayne wrote:
> 
> To my embarassment, I wasn't pressing 's' to save that file; I'd just
> assumed that it'd be hooked to save on exit :-(
> 
> Once I do that, it works as expected.

-- 
Duncan Bayne
ph: +61 420817082 | web: http://duncan-bayne.github.com/ | skype:
duncan_bayne

I usually check my mail every 24 - 48 hours.  If there's something
urgent going on, please send me an SMS or call me.