bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Feng Shu]

I use gnus to connect imap.163.com server.

  (setq gnus-secondary-select-methods
        '((nnimap "imap.163.com"
                  (nnimap-inbox "INBOX")
                  (nnimap-split-methods default)
                  (nnimap-expunge t)
                  (nnimap-stream ssl)
                  (nnimap-fetch-partial-articles t))))

It connect success and have the below group:

   *      0: nnimap+imap.163.com:INBOX


But If no new mail, emacs will hang until run out memory when I into this group,
If have new mail, works.


process buffer is like below:

  * 10 EXISTS
  * 0 RECENT
  * OK [UIDVALIDITY 8358120] UIDs valid
  * FLAGS (\Answered \Seen \Deleted \Draft \Flagged)
  * OK [PERMANENTFLAGS (\Answered \Seen \Deleted \Draft \Flagged)] Limited
  169 OK [READ-WRITE] SELECT completed


*imap log*

  17:12:57 [imap.163.com] (inhibited)
  17:12:57 [imap.163.com] 208 CAPABILITY
  17:12:57 [imap.163.com] 209 LIST "" "*" 
  17:12:58 [imap.163.com] 217 LIST "" "*"
  17:12:58 [imap.163.com] 218 SELECT "INBOX"
  17:12:58 [imap.163.com] 219 UID FETCH 1:* FLAGS
  17:12:58 [imap.163.com] 220 EXAMINE "mail.misc"
  17:12:58 [imap.163.com] 221 UID FETCH 1320004027:* FLAGS
  17:12:58 [imap.163.com] 222 EXAMINE "INBOX"
  17:12:58 [imap.163.com] 223 UID FETCH 1320753282:* FLAGS
  17:13:02 [imap.163.com] 224 SELECT "mail.misc"



In GNU Emacs 28.0.60 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0)
 of 2021-11-16 built on debian
Repository revision: e852822f3db469c985bf022651f184d6ff2c518a
Repository branch: emacs-28
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Configured using:
 'configure --prefix=/home/feng/emacs/emacs28-gtk-20211116
 --with-native-compilation --with-xwidgets --with-json'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM XWIDGETS GTK3 ZLIB

Important settings:
  value of $LANG: zh_CN.UTF-8
  value of $XMODIFIERS: @im=fcitx
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  global-undo-tree-mode: t
  projectile-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  emms-playing-time-display-mode: t
  emms-playing-time-mode: t
  yas-global-mode: t
  company-posframe-mode: t
  global-company-mode: t
  org-super-agenda-mode: t
  shell-dirtrack-mode: t
  pyim-isearch-mode: t
  vertico-mode: t
  savehist-mode: t
  global-tab-line-mode: t
  global-so-long-mode: t
  electric-pair-mode: t
  recentf-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  context-menu-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/feng/projects/emacs-packages/pyim/pyim-cregexp hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-cregexp
/home/feng/projects/emacs-packages/pyim/pyim-liberime hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-liberime
/home/feng/projects/emacs-packages/pyim/pyim-candidates hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-candidates
/home/feng/projects/emacs-packages/pyim/pyim-pinyin hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-pinyin
/home/feng/projects/emacs-packages/pyim/pyim-punctuation hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-punctuation
/home/feng/projects/emacs-packages/pyim/pyim-imobjs hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-imobjs
/home/feng/projects/emacs-packages/pyim/pyim-dcache hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-dcache
/home/feng/projects/emacs-packages/pyim/pyim-autoselector hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-autoselector
/home/feng/projects/emacs-packages/pyim/pyim-scheme hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-scheme
/home/feng/projects/emacs-packages/pyim/pyim-codes hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-codes
/home/feng/projects/emacs-packages/pyim/pyim-probe hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-probe
/home/feng/projects/emacs-packages/pyim/pyim-magic hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-magic
/home/feng/projects/emacs-packages/pyim/pyim-dict hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-dict
/home/feng/projects/emacs-packages/pyim/pyim-hacks hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-hacks
/home/feng/projects/emacs-packages/pyim/pyim-pymap hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-pymap
/home/feng/projects/emacs-packages/pyim/pyim-indicator hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-indicator
/home/feng/projects/emacs-packages/pyim/pyim-page hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-page
/home/feng/projects/emacs-packages/pyim/pyim-dhashcache hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-dhashcache
/home/feng/projects/emacs-packages/pyim/pyim-preview hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-preview
/home/feng/projects/emacs-packages/pyim/pyim hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim
/home/feng/projects/emacs-packages/pyim/pyim-entered hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-entered
/home/feng/projects/emacs-packages/pyim/pyim-cstring hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-cstring
/home/feng/projects/emacs-packages/pyim/pyim-outcome hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-outcome
/home/feng/projects/emacs-packages/pyim/pyim-common hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-common
/home/feng/projects/emacs-packages/pyim/pyim-process hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-process
/home/feng/projects/emacs-packages/pyim/pyim-dregcache hides /home/feng/.emacs.d/elpa-28/pyim-20211116.22/pyim-dregcache
/home/feng/.emacs.d/elpa-28/transient-20211105.100/transient hides /home/feng/emacs/emacs28-gtk-20211116/share/emacs/28.0.60/lisp/transient

Features:
(shadow mail-extr emacsbug sendmail misearch multi-isearch jka-compr
gnus-async pyim-dhashcache async mule-util gnus-ml hl-line nndraft nnmh
utf-7 gnutls network-stream nsm nnml cursor-sensor eh-gnus-common
gnus-select-account eww xdg url-queue mm-url gnus-demon gnus-agent
gnus-srvr gnus-score score-mode nnvirtual gnus-msg nntp gnus-cite
gnus-art mm-uu mml2015 mm-view mml-smime smime dig gnus-cache gnus-sum
shr kinsoku svg dom gnus-group gnus-undo gnus-start gnus-dbus dbus
gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int
gnus-range gnus-win gnus nnheader wjw eh-gnus emacs-helper eh-misc
undo-tree xwidget image-mode exif projectile lisp-mnt grep ibuf-ext
ibuffer ibuffer-loaddefs magit-bookmark magit-submodule magit-obsolete
magit-popup magit-blame magit-stash magit-reflog magit-bisect magit-push
magit-pull magit-fetch magit-clone magit-remote magit-commit
magit-sequence magit-notes magit-worktree magit-tag magit-merge
magit-branch magit-reset magit-files magit-refs magit-status magit
magit-repos magit-apply magit-wip magit-log which-func imenu magit-diff
smerge-mode diff diff-mode git-commit log-edit message rmc puny rfc822
mml mml-sec epa gnus-util rmail rmail-loaddefs mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr mailabbrev mail-utils gmm-utils mailheader pcvs-util add-log
magit-core magit-autorevert magit-margin magit-transient magit-process
with-editor server magit-mode transient magit-git magit-section
magit-utils crm gitpatch aggressive-indent eh-emms emms-librefm-stream
xml emms-librefm-scrobbler emms-playlist-limit emms-i18n emms-history
emms-score emms-stream-info emms-metaplaylist-mode emms-bookmarks
emms-cue emms-mode-line-icon emms-player-xine emms-player-mpd tq
emms-lyrics emms-url emms-streams emms-show-all emms-tag-editor
emms-tag-tracktag emms-mark emms-mode-line emms-info-tinytag
emms-info-metaflac emms-info-opusinfo emms-info-ogginfo
emms-info-mp3info emms-player-vlc emms-player-mpv emms-playing-time
emms-player-mplayer emms-player-simple emms-browser sort
emms-playlist-sort emms-last-played emms-volume emms-volume-sndioctl
emms-volume-mixerctl emms-volume-pulse emms-volume-amixer
emms-playlist-mode emms-source-playlist emms-source-file locate
emms-cache emms-info-native bindat emms-info-exiftool emms-info
emms-later-do emms-setup emms emms-compat epg rfc6068 epg-config el2org
cnfonts cnfonts-ui cus-edit cus-load thingatpt yasnippet vterm
face-remap compile color term disp-table ehelp vterm-module term/xterm
xterm eh-complete company-posframe company-keywords company-etags etags
fileloop xref project company-gtags company-template
company-dabbrev-code company-dabbrev company-files company-capf company
eh-org org-super-agenda ts ht s org-habit ob-python python tramp-sh
tramp tramp-loaddefs trampver tramp-integration files-x tramp-compat
shell parse-time ls-lisp ob-org ob-plantuml ox-extra ox-md ox-org ox-odt
rng-loc rng-uri rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse
nxml-ns nxml-enc xmltok nxml-util ox-latex ox-icalendar org-agenda
ox-html table ox-ascii ox-publish ox org-element avl-tree generator
org-protocol autorevert filenotify org-attach org-id org-refile org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src
ob-comint org-pcomplete pcomplete comint ansi-color org-list org-faces
org-entities noutline outline org-version ob-emacs-lisp ob-core ob-eval
org-table oc-basic bibtex iso8601 time-date org-keys oc org-loaddefs
find-func eh-functions eh-theme modus-vivendi-theme modus-operandi-theme
modus-themes eh-basic cal-china-x cal-china lunar solar cal-dst holidays
hol-loaddefs cal-menu calendar cal-loaddefs pyim-liberime liberime
liberime-core pyim pyim-hacks pyim-probe pyim-cregexp easy-mmode pcase
xr pyim-process pyim-cstring pyim-autoselector pyim-punctuation
pyim-outcome pyim-indicator pyim-preview pyim-magic pyim-candidates
pyim-codes pyim-imobjs pyim-pinyin pyim-pymap pyim-dcache pyim-dict
derived pyim-page pyim-entered popup posframe pyim-scheme pyim-common
dired-ranger ring dired-hacks-utils dash wdired dired dired-loaddefs vc
vc-dispatcher consult-vertico consult orderless vertico savehist
tab-line switch-window switch-window-mvborder switch-window-asciiart
quail so-long elec-pair bookmark text-property-search pp recentf
tree-widget wid-edit elpa-mirror finder-inf ol org-compat advice
org-macs format-spec edmacro kmacro comp comp-cstr warnings rx cl-extra
help-mode info package browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util mailcap
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib china-util iso-transl
tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads
xwidget-internal dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting cairo move-toolbar gtk x-toolkit
x multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 914774 81591)
 (symbols 48 51877 119)
 (strings 32 275864 8282)
 (string-bytes 1 7292453)
 (vectors 16 89807)
 (vector-slots 8 13555358 110222)
 (floats 8 1197 421)
 (intervals 56 1964 1082)
 (buffers 992 27))

-- 

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Lars Ingebrigtsen]

"Feng Shu" <tumashu@163.com> writes:

> But If no new mail, emacs will hang until run out memory when I into
> this group, If have new mail, works.

Try (setq debug-on-quit t) and then `C-g' when Emacs hangs, and post the
resulting backtrace.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Feng Shu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "Feng Shu" <tumashu@163.com> writes:
>
>> But If no new mail, emacs will hang until run out memory when I into
>> this group, If have new mail, works.
>
> Try (setq debug-on-quit t) and then `C-g' when Emacs hangs, and post the
> resulting backtrace.

Debugger entered--Lisp error: (quit)
  gnus-uncompress-range((1 . 1320004225))
  gnus-articles-to-read("nnimap+imap.163.com:mail.misc" t)
  gnus-select-newsgroup("nnimap+imap.163.com:mail.misc" t nil)
  gnus-summary-read-group-1("nnimap+imap.163.com:mail.misc" t t nil nil nil)
  gnus-summary-read-group("nnimap+imap.163.com:mail.misc" t t nil nil nil nil)
  gnus-group-read-group(nil t)
  gnus-group-select-group(nil)
  funcall-interactively(gnus-group-select-group nil)
  command-execute(gnus-group-select-group)


-- 

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Lars Ingebrigtsen]

"Feng Shu" <tumashu@163.com> writes:

> Debugger entered--Lisp error: (quit)
>   gnus-uncompress-range((1 . 1320004225))
>   gnus-articles-to-read("nnimap+imap.163.com:mail.misc" t)

Gnus doesn't deal well with newsgroups that have message numbers in the
billions by default.  Does setting gnus-newsgroup-maximum-articles help
here?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Feng Shu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "Feng Shu" <tumashu@163.com> writes:
>
>> Debugger entered--Lisp error: (quit)
>>   gnus-uncompress-range((1 . 1320004225))
>>   gnus-articles-to-read("nnimap+imap.163.com:mail.misc" t)
>
> Gnus doesn't deal well with newsgroups that have message numbers in the
> billions by default.  Does setting gnus-newsgroup-maximum-articles help
> here?

When set gnus-newsgroup-maximum-articles to 100000, it works.

I only have <10000 mails, just email's UID are very large numbers.


-- 

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Lars Ingebrigtsen]

"Feng Shu" <tumashu@163.com> writes:

> I only have <10000 mails, just email's UID are very large numbers.

Yup.  The issue is that Gnus will sometimes convert these ranges into
lists, and when the UIDs are very large, that blows up.  The real fix
here is to change Gnus to never do that, but always work on ranges.
It's a rather daunting task, though, since so many of Gnus' innards work
on lists of article numbers, not ranges.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Feng Shu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "Feng Shu" <tumashu@163.com> writes:
>
>> I only have <10000 mails, just email's UID are very large numbers.
>
> Yup.  The issue is that Gnus will sometimes convert these ranges into
> lists, and when the UIDs are very large, that blows up.  The real fix
> here is to change Gnus to never do that, but always work on ranges.
> It's a rather daunting task, though, since so many of Gnus' innards work
> on lists of article numbers, not ranges.

Maybe we can let user choice a max number when faced this situation instead of hang emacs.

-- 

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Lars Ingebrigtsen]

"Feng Shu" <tumashu@163.com> writes:

> Maybe we can let user choice a max number when faced this situation
> instead of hang emacs.

Well, people have different computers, so finding a reasonable limit
isn't trivial.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Feng Shu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "Feng Shu" <tumashu@163.com> writes:
>
>> Maybe we can let user choice a max number when faced this situation
>> instead of hang emacs.
>
> Well, people have different computers, so finding a reasonable limit
> isn't trivial.  

Do we really open >100000 mails a time? :-)

-- 

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Lars Ingebrigtsen]

"Feng Shu" <tumashu@163.com> writes:

> Do we really open >100000 mails a time? :-)

I do that sometimes.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#51887: 28.0.60; Gnus use imap.163.com will hang emacs and ran out of memory

[Feng Shu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> "Feng Shu" <tumashu@163.com> writes:
>
>> Do we really open >100000 mails a time? :-)

牛... cool ...

>
> I do that sometimes.

--