From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Alexandre Oliva Newsgroups: gmane.emacs.bugs Subject: bug#31044: 25.3; Memory use explodes splitting email into gnus nnfolders Date: Tue, 03 Apr 2018 05:39:25 -0300 Message-ID: NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: blaine.gmane.org 1522764319 3684 195.159.176.226 (3 Apr 2018 14:05:19 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 3 Apr 2018 14:05:19 +0000 (UTC) To: 31044@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Tue Apr 03 16:05:15 2018 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3MYl-0000qs-24 for geb-bug-gnu-emacs@m.gmane.org; Tue, 03 Apr 2018 16:05:15 +0200 Original-Received: from localhost ([::1]:37214 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f3Mao-0006zU-ON for geb-bug-gnu-emacs@m.gmane.org; Tue, 03 Apr 2018 10:07:22 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:36615) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f3MaX-0006wX-SY for bug-gnu-emacs@gnu.org; Tue, 03 Apr 2018 10:07:07 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f3MaU-0007Ph-KR for bug-gnu-emacs@gnu.org; Tue, 03 Apr 2018 10:07:05 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:57467) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1f3MaU-0007PB-Fa for bug-gnu-emacs@gnu.org; Tue, 03 Apr 2018 10:07:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1f3MaU-0003J0-39 for bug-gnu-emacs@gnu.org; Tue, 03 Apr 2018 10:07:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Alexandre Oliva Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 03 Apr 2018 14:07:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 31044 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.152276439912666 (code B ref -1); Tue, 03 Apr 2018 14:07:01 +0000 Original-Received: (at submit) by debbugs.gnu.org; 3 Apr 2018 14:06:39 +0000 Original-Received: from localhost ([127.0.0.1]:37129 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3Ma7-0003IC-0S for submit@debbugs.gnu.org; Tue, 03 Apr 2018 10:06:39 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:44602) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f3HTo-0008IL-BQ for submit@debbugs.gnu.org; Tue, 03 Apr 2018 04:39:48 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f3HTf-0004Ej-T8 for submit@debbugs.gnu.org; Tue, 03 Apr 2018 04:39:43 -0400 Original-Received: from lists.gnu.org ([2001:4830:134:3::11]:49872) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f3HTf-0004Ed-QB for submit@debbugs.gnu.org; Tue, 03 Apr 2018 04:39:39 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:35553) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f3HTe-0006uq-0f for bug-gnu-emacs@gnu.org; Tue, 03 Apr 2018 04:39:39 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f3HTc-0004CR-IS for bug-gnu-emacs@gnu.org; Tue, 03 Apr 2018 04:39:38 -0400 Original-Received: from linux-libre.fsfla.org ([208.118.235.54]:40224) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1f3HTc-0004CL-Dp for bug-gnu-emacs@gnu.org; Tue, 03 Apr 2018 04:39:36 -0400 Original-Received: from free.home (home.lxoliva.fsfla.org [172.31.160.164]) by linux-libre.fsfla.org (8.14.4/8.14.4/Debian-4.1ubuntu1) with ESMTP id w338dWMS018198 for ; Tue, 3 Apr 2018 08:39:34 GMT Original-Received: from livre (livre.home [172.31.160.2]) by free.home (8.15.2/8.15.2) with ESMTP id w338dPMU160630; Tue, 3 Apr 2018 05:39:26 -0300 X-Scanned-By: MIMEDefang 2.84 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Mailman-Approved-At: Tue, 03 Apr 2018 10:06:38 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:144849 Archived-At: I have some 5 GB of active gnus nnfolder files. Emacs memory use, in a fresh -nw remote session, skyrocketed to over 55GB while splitting some 150MB of email. This happened to me today. However, this has not been unusual. Smaller batches of email have often used up over 20GB of RAM; my poor x200, limited to 8GB of RAM, hasn't been able to split my mail any more for months; I can't promise this started with 25.3, it may have been before that, but some older version of Emacs had little trouble splitting similar aomunts of email onto pretty much the same set of folders with 8GB, and even 4GB of RAM, though the latter was already pushing it a longer while ago. I'm clueless as to how to debug this, or even to identify the source of such outrageous memory consumption. All I know is that memory use grows while gnus is splitting email, keeping multiple unsaved folders open until the splitting is complete and it goes about saving them all, at which point memory use goes back down. While splitting is ongoing, it doesn't look like the working set is that large, and, given enough swap space, it will complete, slowed down more by the swapping out of inactive pages than by actually bringing stuff back in. Pointers about debugging Emacs memory use while it's active would be appreciated (interrupting mail splitting in the middle is not such a great idea). I'm using a Fedora build of GNU Emacs, if that matters, but I could build custom versions if that would help. I can do some Emacs Lisp too, though not much. Thanks in advance for any advice, In GNU Emacs 25.3.1 (x86_64-redhat-linux-gnu, GTK+ Version 3.22.19) of 2017-09-14 built on buildvm-31.phx2.fedoraproject.org Windowing system distributor 'Fedora Project', version 11.0.11906000 System Description: Generic release 27 (Generic) Configured using: 'configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=gtk3 --with-gpm=no --with-xwidgets --with-modules build_alias=x86_64-redhat-linux-gnu host_alias=x86_64-redhat-linux-gnu 'CFLAGS=-DMAIL_USE_LOCKF -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' LDFLAGS=-Wl,-z,relro PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig' Configured features: XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS GCONF GSETTINGS NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 MODULES XWIDGETS Important settings: value of $LANG: en_US.UTF-8 value of $XMODIFIERS: @im=none locale-coding-system: utf-8-unix Major mode: Group Minor modes in effect: diff-auto-refine-mode: t shell-dirtrack-mode: t gnus-topic-mode: t cursor-sensor-mode: t gnus-undo-mode: t delete-selection-mode: t tooltip-mode: t global-eldoc-mode: t electric-indent-mode: t mouse-wheel-mode: t tool-bar-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t buffer-read-only: t line-number-mode: t transient-mark-mode: t Recent messages: Deleting article 17615 in redhat.tools.team... Deleting articles...done Saving file /home/aoliva/.../mail/Mail/nnfolder/redhat/tools/team... Wrote /home/aoliva/.../mail/Mail/nnfolder/redhat/tools/team Expiring articles...done Saving /home/aoliva/.newsrc.eld... Saving file /home/aoliva/.newsrc.eld... Wrote /home/aoliva/.newsrc.eld Saving /home/aoliva/.newsrc.eld...done Making completion list... [2 times] Load-path shadows: /home/aoliva/src/elsp/site-start hides /usr/share/emacs/site-lisp/site-start /usr/share/emacs/site-lisp/lilypond-init hides /usr/share/emacs/site-lisp/site-start.d/lilypond-init Features: (shadow warnings emacsbug diff-mode flow-fill tramp-cmds tramp-cache tramp-sh tramp tramp-compat tramp-loaddefs trampver ucs-normalize shell pcomplete comint ring custom advice misearch multi-isearch gnus-dup sgml-mode shr-color color shr seq dom subr-x browse-url network-stream nsm starttls mailalias smtpmail sendmail supercite easy-mmode regi nnir mail-extr qp sort ansi-color gnus-cite mm-archive gnus-async gnus-bcklg gnus-ml disp-table gnus-topic cursor-sensor nndraft nnrss xml mm-url url url-proxy url-privacy url-expand url-methods url-history url-cookie url-domsuf url-util url-parse auth-source cl-seq eieio byte-opt bytecomp byte-compile cl-extra cconv eieio-core cl-macs gv url-vars nnml nnfolder nnmh gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime dig mailcap gnus-cache gnus-demon nntp gnus-mlspl gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source tls gnutls utf7 netrc nnoo parse-time gnus-spec gnus-int gnus-range message idna dired format-spec rfc822 mml mml-sec password-cache epg epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev gmm-utils mailheader gnus-win gnus gnus-ems nnheader gnus-util mail-utils mm-util help-fns help-mode easymenu mail-prsvr wid-edit cl-loaddefs pcase cl-lib delsel cus-start cus-load flyspell ispell iso-transl jka-compr dot-emacs preview-latex tex-site auto-loads time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces cus-face macroexp files text-properties overlay sha1 md5 base64 format env code-pages mule widget hashtable-print-readable backquote dbusbind inotify dynamic-setting system-font-setting font-render-setting xwidget-internal move-toolbar gtk x-toolkit x multi-tty make-network-process emacs) Memory information: ((conses 16 536826 110508) (symbols 48 37634 64) (miscs 40 330 1684) (strings 32 70356 15142) (string-bytes 1 2305445) (vectors 16 56895) (vector-slots 8 2174791 108123) (floats 8 1059 549) (intervals 56 6371 615) (buffers 976 65)) -- Alexandre Oliva, freedom fighter http://FSFLA.org/~lxoliva/ You must be the change you wish to see in the world. -- Gandhi Be Free! -- http://FSFLA.org/ FSF Latin America board member Free Software Evangelist|Red Hat Brasil GNU Toolchain Engineer