bug#27498: 25.2; Memory leak in M-x shell

bug#27498: 25.2; Memory leak in M-x shell

[Ricardo Wurmus]

Emacs leaks memory as lines are printed in a shell buffer.  Here is a
recipe:

    emacs -Q
    M-x shell
    yes "This should not leak, but it does."

Observe the memory usage grow in a tool like “top”.  Once you decide
that Emacs has eaten enough memory, kill the shell buffer.  Note that
memory is not released.

This is a problem for me because I often compile software in shell
buffers (via “guix build”).  The compilation processes often print out
very large amounts of text and Emacs quickly eats up all memory on my
machine, freezing it.  I can only recover from this by killing Emacs
(which is terrible, because I live in Emacs).

This has not been a problem in Emacs 24.x; I’m also pretty confident
that this was no problem in version 25.1.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net




In GNU Emacs 25.2.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.22.12)
 of 2017-04-24 built on localhost
Windowing system distributor 'The X.Org Foundation', version 11.0.11903000
Configured using:
 'configure
 CONFIG_SHELL=/gnu/store/k7029k5va68lkapbzcycdzj7m5bjb4b8-bash-4.4.12/bin/bash
 SHELL=/gnu/store/k7029k5va68lkapbzcycdzj7m5bjb4b8-bash-4.4.12/bin/bash
 --prefix=/gnu/store/q3kvsn12x99fpiw08a2jgw9h6biky907-emacs-25.2
 --enable-fast-install'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS NOTIFY ACL GNUTLS
LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: mu4e:compose

Minor modes in effect:
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  mml-mode: t
  erc-track-mode: t
  erc-track-minor-mode: t
  erc-ring-mode: t
  erc-networks-mode: t
  erc-netsplit-mode: t
  erc-menu-mode: t
  erc-list-mode: t
  erc-pcomplete-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-autojoin-mode: t
  erc-autoaway-mode: t
  erc-smiley-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  typo-mode: t
  TeX-PDF-mode: t
  guide-key-mode: t
  diff-auto-refine-mode: t
  pdf-occur-global-minor-mode: t
  savehist-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  delete-selection-mode: t
  desktop-save-mode: t
  display-time-mode: t
  projectile-global-mode: t
  projectile-mode: t
  ido-ubiquitous-mode: t
  ido-vertical-mode: t
  global-paren-face-mode: t
  global-guix-prettify-mode: t
  guix-prettify-mode: t
  shell-switcher-mode: t
  shell-dirtrack-mode: t
  show-paren-mode: t
  save-place-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  global-prettify-symbols-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  auto-fill-function: message-do-auto-fill
  transient-mark-mode: t
  abbrev-mode: t

Recent messages:
Mark saved where search started [2 times]
Contacting host: debbugs.gnu.org:80
Opening nndoc server on /tmp/gnus-temp-group-266625pe-ephemeral...done
Mark set
Mark saved where search started
Mark set
Quit [3 times]
(New file)
Mark set
(Inserted HYPHEN-MINUS; type - for other options)
Quit

Load-path shadows: […]

Features:
(shadow emacsbug gnus-cite gnus-async gnus-bcklg gnus-agent gnus-srvr
gnus-score score-mode nnvirtual nntp gnus-ml gnus-msg nndoc gnus-cache
url-handlers gnus-dup mm-archive debbugs-gnu debbugs soap-client
warnings rng-xsd rng-dt rng-util xsd-regexp woman tabify man vc
vc-dispatcher tramp-cache epa-mail qp mail-extr sort rect shr-color
epa-file latexenc autoconf autoconf-mode ffap find-dired network-stream
nsm starttls url-http url-gw url-cache url-auth mu4e-contrib org-mu4e
mu4e mu4e-speedbar speedbar sb-image ezimage dframe mu4e-main
mu4e-context mu4e-view epa derived mu4e-headers mu4e-compose mu4e-draft
mu4e-actions rfc2368 smtpmail mu4e-mark mu4e-message html2text mu4e-proc
mu4e-utils mu4e-lists mu4e-vars hl-line mu4e-meta hippie-exp
magit-version magit-obsolete magit-blame magit-stash magit-bisect
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-branch magit-files magit-refs magit-status magit magit-repos
magit-apply magit-wip magit-log magit-diff smerge-mode magit-core
magit-autorevert magit-process magit-margin magit-mode magit-git
magit-section magit-popup git-commit magit-utils log-edit pcvs-util
add-log with-editor async-bytecomp async tramp-sh server pcmpl-unix
guix-build-log smiley gnus-art mm-uu mml2015 mm-view mml-smime smime dig
gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail
mail-source tls gnutls utf7 netrc nnoo parse-time gnus-spec gnus-int
gnus-range message sendmail rfc822 mml mml-sec epg mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
gmm-utils mailheader gnus-win znc erc-track erc-match erc-ring
erc-networks erc-netsplit erc-menu erc-list erc-pcomplete erc-button
erc-fill erc-stamp erc-join erc-autoaway erc-goodies erc erc-backend
erc-compat cl misearch multi-isearch markdown-mode preview prv-emacs
font-latex tex-mode eww mm-url gnus gnus-ems nnheader mail-utils
url-queue url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util url-parse url-vars mailcap shr dom
browse-url conf-mode typo face-remap sh-script smie executable org-rmail
org-mhe org-irc org-info org-gnus org-docview doc-view subr-x org-bibtex
bibtex org-bbdb org-w3m org-element avl-tree ob-lilypond org-bullets
org-indent org org-macro org-footnote org-pcomplete org-list org-faces
org-entities noutline outline org-version ob-C cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs ob-sh
ob-scheme ob-R ob-emacs-lisp ob ob-tangle org-src ob-ref ob-lob ob-table
ob-keys ob-exp ob-comint ob-core ob-eval org-compat org-macs
org-loaddefs cal-menu calendar cal-loaddefs tex-buf latex tex-ispell
tex-style tex-info tex dbus xml crm texinfo autorevert filenotify
guide-key s popwin vc-git diff-mode easy-mmode guix-devel guix-misc
guix-ui guix-ui-messages bui bui-list bui-info bui-entry bui-core
bui-history bui-button guix-read guix-help-vars guix-repl guix-profiles
guix-external guix-config guix-build-config guix-geiser geiser-mode
geiser-xref geiser-compile geiser-debug guix-guile geiser-guile
geiser-repl geiser-image geiser-company geiser-doc geiser-menu
geiser-autodoc info-look geiser-edit geiser-completion geiser-eval
geiser-connection geiser-syntax geiser-impl geiser-custom geiser-log
geiser-base geiser-popup view paredit company-oddmuse company-keywords
company-etags etags xref project company-gtags company-dabbrev-code
company-dabbrev company-files company-capf company-cmake company-xcode
company-clang company-semantic company-eclim company-template
company-css company-nxml company-bbdb color company scheme linum
pdf-occur tablist tablist-filter semantic/wisent/comp semantic/wisent
semantic/wisent/wisent semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw mode-local find-func cedet dired
pdf-isearch let-alist pdf-misc imenu pdf-tools pdf-view bookmark pp
jka-compr pdf-cache pdf-info tq pdf-util image-mode savehist undo-tree
diff delsel page-break-lines fill-column-indicator re-builder desktop
frameset smart-mode-line-respectful-theme smart-mode-line time
rich-minority lilypond-mode rot13 disp-table projectile grep compile
ibuf-ext ibuffer thingatpt smex ido-ubiquitous ido-completing-read+
cus-edit cus-start cus-load wid-edit ido-vertical-mode paren-face
guix-prettify guix-utils bui-utils guix shell-switcher rswitcher tramp
tramp-compat auth-source eieio eieio-core gnus-util mm-util help-fns
mail-prsvr password-cache tramp-loaddefs trampver ucs-normalize shell
pcomplete comint ansi-color ring format-spec info solarized-light-theme
solarized dash better-defaults finder-inf package epg-config
znc-autoloads ido-completing-read+-autoloads advice dash-autoloads
shell-switcher-autoloads js2-mode-autoloads clojure-mode-autoloads
bui-autoloads page-break-lines-autoloads tagedit-autoloads
perspective-autoloads org-bullets-autoloads
emms-mode-line-cycle-autoloads visual-fill-column-autoloads
debbugs-autoloads ivy-autoloads projectile-autoloads
emms-player-mpv-autoloads rich-minority-autoloads with-editor-autoloads
skewer-mode-autoloads expand-region-autoloads solarized-theme-autoloads
ido-ubiquitous-autoloads s-autoloads paren-face-autoloads
web-mode-autoloads smart-mode-line-autoloads undo-tree-autoloads
epl-autoloads pdf-tools-autoloads typo-autoloads ace-window-autoloads
magit-popup-autoloads multiple-cursors-autoloads markdown-mode-autoloads
fill-column-indicator-autoloads auctex-autoloads tex-site
god-mode-autoloads hydra-autoloads pkg-info-autoloads
simple-httpd-autoloads smex-autoloads org-autoloads avy-autoloads
lispy-autoloads yasnippet-autoloads better-defaults-autoloads paren
edmacro kmacro saveplace ido seq byte-opt bytecomp byte-compile cconv
iedit-autoloads async-autoloads company-autoloads mu4e-autoloads
emms-auto emacs-paredit-autoloads geiser-install geiser wget-autoloads
guix-autoloads cl-extra help-mode easymenu cl-seq guix-emacs cl-macs gv
rx cl-loaddefs pcase cl-lib time-date mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 5498967 500617)
 (symbols 48 81951 0)
 (miscs 40 104299 4766)
 (strings 32 302649 135536)
 (string-bytes 1 12956536)
 (vectors 16 105509)
 (vector-slots 8 2321803 110802)
 (floats 8 3961 1252)
 (intervals 56 409060 3231)
 (buffers 976 129))

bug#27498: 25.2; Memory leak in M-x shell

[npostavs]

forcemerge 26952 27498
quit

Ricardo Wurmus <rekado@elephly.net> writes:

> Emacs leaks memory as lines are printed in a shell buffer.  Here is a
> recipe:
>
>     emacs -Q
>     M-x shell
>     yes "This should not leak, but it does."
>
> Observe the memory usage grow in a tool like “top”.  Once you decide
> that Emacs has eaten enough memory, kill the shell buffer.  Note that
> memory is not released.
>
> This is a problem for me because I often compile software in shell
> buffers (via “guix build”).  The compilation processes often print out
> very large amounts of text and Emacs quickly eats up all memory on my
> machine, freezing it.  I can only recover from this by killing Emacs
> (which is terrible, because I live in Emacs).

This is Bug#26952, it's due to using the custom "GNU version of malloc"
without the "relocating allocater" (there were several bugs with the
relocating allocater in 25.1).  To fix you can configure with
REL_ALLOC=yes (this should be okay in 25.2), or use Emacs' master branch
where the standard system allocater can be used once again.

bug#27498: 25.2; Memory leak in M-x shell

[Ricardo Wurmus]

npostavs@users.sourceforge.net writes:

[…]

> This is Bug#26952, it's due to using the custom "GNU version of malloc"
> without the "relocating allocater" (there were several bugs with the
> relocating allocater in 25.1).  To fix you can configure with
> REL_ALLOC=yes (this should be okay in 25.2), or use Emacs' master branch
> where the standard system allocater can be used once again.

Thank you for this information.  I’ll give this a try.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#27498: 25.2; Memory leak in M-x shell

[Ricardo Wurmus]

npostavs@users.sourceforge.net writes:

> This is Bug#26952, it's due to using the custom "GNU version of malloc"
> without the "relocating allocater" (there were several bugs with the
> relocating allocater in 25.1).  To fix you can configure with
> REL_ALLOC=yes (this should be okay in 25.2), or use Emacs' master branch
> where the standard system allocater can be used once again.

I have tried this and while the memory consumption still grows as the
buffer fills up (which I suppose is expected), memory is released as
soon as the buffer is killed.

Thank you for the hint!

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#27498: 25.2; Memory leak in M-x shell

[Francesco Potortì]

>Emacs leaks memory as lines are printed in a shell buffer.  Here is a
>recipe:
>
>    emacs -Q
>    M-x shell
>    yes "This should not leak, but it does."
>
>Observe the memory usage grow in a tool like “top”.  Once you decide
>that Emacs has eaten enough memory, kill the shell buffer.  Note that
>memory is not released.

This may be a duplicate of #26952