bug#31044: 25.3; Memory use explodes splitting email into gnus nnfolders

[Alexandre Oliva <oliva@gnu.org>]

I have some 5 GB of active gnus nnfolder files.  Emacs memory use, in a
fresh -nw remote session, skyrocketed to over 55GB while splitting some
150MB of email.  This happened to me today.

However, this has not been unusual.  Smaller batches of email have often
used up over 20GB of RAM; my poor x200, limited to 8GB of RAM, hasn't
been able to split my mail any more for months; I can't promise this
started with 25.3, it may have been before that, but some older version
of Emacs had little trouble splitting similar aomunts of email onto
pretty much the same set of folders with 8GB, and even 4GB of RAM,
though the latter was already pushing it a longer while ago.

I'm clueless as to how to debug this, or even to identify the source of
such outrageous memory consumption.  All I know is that memory use grows
while gnus is splitting email, keeping multiple unsaved folders open
until the splitting is complete and it goes about saving them all, at
which point memory use goes back down.

While splitting is ongoing, it doesn't look like the working set is that
large, and, given enough swap space, it will complete, slowed down more
by the swapping out of inactive pages than by actually bringing stuff
back in.

Pointers about debugging Emacs memory use while it's active would be
appreciated (interrupting mail splitting in the middle is not such a
great idea).

I'm using a Fedora build of GNU Emacs, if that matters, but I could
build custom versions if that would help.  I can do some Emacs Lisp too,
though not much.

Thanks in advance for any advice,




In GNU Emacs 25.3.1 (x86_64-redhat-linux-gnu, GTK+ Version 3.22.19)
 of 2017-09-14 built on buildvm-31.phx2.fedoraproject.org
Windowing system distributor 'Fedora Project', version 11.0.11906000
System Description:	Generic release 27 (Generic)

Configured using:
 'configure --build=x86_64-redhat-linux-gnu
 --host=x86_64-redhat-linux-gnu --program-prefix=
 --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
 --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
 --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png
 --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=gtk3
 --with-gpm=no --with-xwidgets --with-modules
 build_alias=x86_64-redhat-linux-gnu host_alias=x86_64-redhat-linux-gnu
 'CFLAGS=-DMAIL_USE_LOCKF -O2 -g -pipe -Wall -Werror=format-security
 -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
 --param=ssp-buffer-size=4 -grecord-gcc-switches
 -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic'
 LDFLAGS=-Wl,-z,relro
 PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS GCONF GSETTINGS NOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 MODULES XWIDGETS

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  gnus-topic-mode: t
  cursor-sensor-mode: t
  gnus-undo-mode: t
  delete-selection-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Deleting article 17615 in redhat.tools.team...
Deleting articles...done
Saving file /home/aoliva/.../mail/Mail/nnfolder/redhat/tools/team...
Wrote /home/aoliva/.../mail/Mail/nnfolder/redhat/tools/team
Expiring articles...done
Saving /home/aoliva/.newsrc.eld...
Saving file /home/aoliva/.newsrc.eld...
Wrote /home/aoliva/.newsrc.eld
Saving /home/aoliva/.newsrc.eld...done
Making completion list... [2 times]

Load-path shadows:
/home/aoliva/src/elsp/site-start hides /usr/share/emacs/site-lisp/site-start
/usr/share/emacs/site-lisp/lilypond-init hides /usr/share/emacs/site-lisp/site-start.d/lilypond-init

Features:
(shadow warnings emacsbug diff-mode flow-fill tramp-cmds tramp-cache
tramp-sh tramp tramp-compat tramp-loaddefs trampver ucs-normalize shell
pcomplete comint ring custom advice misearch multi-isearch gnus-dup
sgml-mode shr-color color shr seq dom subr-x browse-url network-stream
nsm starttls mailalias smtpmail sendmail supercite easy-mmode regi nnir
mail-extr qp sort ansi-color gnus-cite mm-archive gnus-async gnus-bcklg
gnus-ml disp-table gnus-topic cursor-sensor nndraft nnrss xml mm-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util url-parse auth-source cl-seq eieio byte-opt bytecomp
byte-compile cl-extra cconv eieio-core cl-macs gv url-vars nnml nnfolder
nnmh gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg
gnus-art mm-uu mml2015 mm-view mml-smime smime dig mailcap gnus-cache
gnus-demon nntp gnus-mlspl gnus-sum gnus-group gnus-undo gnus-start
gnus-cloud nnimap nnmail mail-source tls gnutls utf7 netrc nnoo
parse-time gnus-spec gnus-int gnus-range message idna dired format-spec
rfc822 mml mml-sec password-cache epg epg-config mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
gmm-utils mailheader gnus-win gnus gnus-ems nnheader gnus-util
mail-utils mm-util help-fns help-mode easymenu mail-prsvr wid-edit
cl-loaddefs pcase cl-lib delsel cus-start cus-load flyspell ispell
iso-transl jka-compr dot-emacs preview-latex tex-site auto-loads
time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting xwidget-internal move-toolbar
gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 536826 110508)
 (symbols 48 37634 64)
 (miscs 40 330 1684)
 (strings 32 70356 15142)
 (string-bytes 1 2305445)
 (vectors 16 56895)
 (vector-slots 8 2174791 108123)
 (floats 8 1059 549)
 (intervals 56 6371 615)
 (buffers 976 65))

-- 
Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
You must be the change you wish to see in the world. -- Gandhi
Be Free! -- http://FSFLA.org/   FSF Latin America board member
Free Software Evangelist|Red Hat Brasil GNU Toolchain Engineer