bug#38629: 25.2; garbage-collect doesn't reclaim large *compilation*

bug#38629: 25.2; garbage-collect doesn't reclaim large *compilation*

[Peter Ludemann]

[-- Attachment #1: Type: text/plain, Size: 7084 bytes --]

I ran a large compilation (to the *compilation* buffer) (232,701 lines,
52M).
While running this, the memory usage increased as follows (output from ps
auwwwxxx):

Sun Dec 15 10:52:27 PST 2019
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
peter     1496  0.0  0.0   4524   828 pts/1    S    10:50   0:00
emacsclient -c
peter    31443  6.4  0.5 417868 90012 ?        Ssl  10:49   0:10 emacs
--daemon

Sun Dec 15 11:42:41 PST 2019
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
peter     1496  0.0  0.0   4524    64 pts/1    S    10:50   0:00
emacsclient -c
peter    31443  5.6 51.9 9277928 8466908 ?     Ssl  10:49   3:00 emacs
--daemon

I deleted the *compilation* window and ran garbage-collect; memory usage
decreased slightly, but not nearly to the original memory usage.
It appears that the *compilation* window doesn't get garbage collected.

Sun Dec 15 11:45:08 PST 2019
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
peter     1496  0.0  0.0   4524    64 pts/1    S    10:50   0:00
emacsclient -c
peter    31443  5.5 49.6 8913748 8103072 ?     Ssl  10:49   3:03 emacs
--daemon

Running emacs daemon, started with the command
GDK=emacs emacs --daemon

In GNU Emacs 25.2.2 (x86_64-pc-linux-gnu, GTK+ Version 3.22.21)
 of 2017-09-22, modified by Debian built on lgw01-amd64-050
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description: Ubuntu 18.04.3 LTS

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs25:/etc/emacs:/usr/local/share/emacs/25.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.2/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --build x86_64-linux-gnu
 --prefix=/usr --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs25:/etc/emacs:/usr/local/share/emacs/25.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.2/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-x=yes --with-x-toolkit=gtk3
 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs25-jYekUr/emacs25-25.2+1=.
-fstack-protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LC_MONETARY: en_CA.UTF-8
  value of $LC_NUMERIC: en_CA.UTF-8
  value of $LC_TIME: en_CA.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Shell

Minor modes in effect:
  shell-dirtrack-mode: t
  diff-auto-refine-mode: t
  global-auto-revert-mode: t
  show-paren-mode: t
  display-time-mode: t
  savehist-mode: t
  desktop-save-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Error during redisplay: (jit-lock-function 53777230) signaled (quit)
Quit
Mark set
Quit
Mark set
Quit [2 times]
Saving file /tmp/compilation-2...
Wrote /tmp/compilation-2
Quit [4 times]
Type C-x 1 to delete the help window.
Quit [5 times]

Load-path shadows:
/usr/share/emacs/25.2/site-lisp/debian-startup hides
/usr/share/emacs/site-lisp/debian-startup
/usr/share/emacs25/site-lisp/cmake-data/cmake-mode hides
/usr/share/emacs/site-lisp/cmake-mode
/usr/share/emacs/site-lisp/rst hides
/usr/share/emacs/25.2/lisp/textmodes/rst
~/emacs/prolog hides /usr/share/emacs/25.2/lisp/progmodes/prolog
/usr/share/emacs25/site-lisp/latex-cjk-thai/thai-word hides
/usr/share/emacs/25.2/lisp/language/thai-word

Features:
(shadow sort mail-extr emacsbug message rfc822 mml mml-sec epg mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mail-utils eieio-opt speedbar
sb-image ezimage dframe find-func css-mode misearch multi-isearch
minibuffer-complete-cycle server tempo erlang perl-mode asm-mode
cmake-mode conf-mode jka-compr add-log rst derived haskell-mode
haskell-cabal haskell-utils haskell-font-lock haskell-indentation
haskell-string haskell-sort-imports haskell-lexeme rx
haskell-align-imports haskell-compat haskell-complete-module
haskell-ghc-support flymake dabbrev haskell-customize go-mode url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap find-file ffap etags xref project make-mode
sh-script smie executable markdown-mode color url-parse url-vars
noutline outline tar-mode python tramp-sh tramp tramp-compat auth-source
cl-seq eieio eieio-core cl-macs gnus-util mm-util help-fns mail-prsvr
password-cache tramp-loaddefs trampver ucs-normalize format-spec
smerge-mode prolog align shell pcomplete dired vc-git diff-mode js
advice sgml-mode json map imenu thingatpt cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs finder-inf
go-mode-autoloads info package epg-config seq byte-opt gv bytecomp
byte-compile cl-extra help-mode easymenu cconv autorevert filenotify
grep compile comint ansi-color ring cus-start cus-load time-date paren
time savehist desktop frameset cl-loaddefs pcase cl-lib erlang-start
emacs-goodies-el emacs-goodies-custom emacs-goodies-loaddefs easy-mmode
devhelp mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 637441 50039)
 (symbols 48 38954 0)
 (miscs 40 1139 1456)
 (strings 32 101762 17154)
 (string-bytes 1 3031599)
 (vectors 16 64532)
 (vector-slots 8 1858924 188630)
 (floats 8 527 642)
 (intervals 56 29869 3931)
 (buffers 976 493))

[-- Attachment #2: Type: text/html, Size: 8104 bytes --]

bug#38629: Loading/killing the compilation output doesn't lose memory

[Peter Ludemann]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

If I save the *compilation* buffer and restart emacs, memory usage goes up
modestly when I load the file (52MB; it is automatically fontified by emacs
when I load it):

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
peter    19037  0.6  0.7 441776 115476 ?       Ssl  11:59   2:49 emacs
--daemon
peter    19037  0.6  1.0 495832 169776 ?       Ssl  11:59   2:50 emacs
--daemon

and memory usage returns to about the previous value when I kill the buffer
(I don't need to run 'garbage-collect).

So, the problems seems to be somewhere in the "compile" command:
(a) it uses a *lot* more memory than needed to display the result
(b) it doesn't free that memory when compilation ends or when the
compilation buffer is killed

My "compilation" is actually compile plus 8200 tests, speeding them up by
using GNU-parallel.

[-- Attachment #2: Type: text/html, Size: 1290 bytes --]

bug#38629: 25.2; garbage-collect doesn't reclaim large *compilation*

[Noam Postavsky]

Peter Ludemann <peter.ludemann@gmail.com> writes:

> I ran a large compilation (to the *compilation* buffer) (232,701 lines,
> 52M).

> In GNU Emacs 25.2.2 (x86_64-pc-linux-gnu, GTK+ Version 3.22.21)
>  of 2017-09-22, modified by Debian built on lgw01-amd64-050
> Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
> System Description: Ubuntu 18.04.3 LTS

I think this is a variant of Bug#26952 - "repeated buffer insertion
(e.g. yank-rectangle) consumes excessive memory (4GB+ for 90MB of
text)".  I recommend upgrading to version 26.

bug#38629: 25.2; garbage-collect doesn't reclaim large *compilation*

[Peter Ludemann]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

It seems that emacs25 and emacs26 can't be in the same Ubuntu 18.04
installation (using ppa:kelleyk/emacs), so it'll take me a bit of work to
verify that the problem is fixed in emacs26.


On Mon, 16 Dec 2019 at 06:24, Noam Postavsky <npostavs@gmail.com> wrote:

> Peter Ludemann <peter.ludemann@gmail.com> writes:
>
> > I ran a large compilation (to the *compilation* buffer) (232,701 lines,
> > 52M).
>
> > In GNU Emacs 25.2.2 (x86_64-pc-linux-gnu, GTK+ Version 3.22.21)
> >  of 2017-09-22, modified by Debian built on lgw01-amd64-050
> > Windowing system distributor 'The X.Org Foundation', version
> 11.0.11906000
> > System Description: Ubuntu 18.04.3 LTS
>
> I think this is a variant of Bug#26952 - "repeated buffer insertion
> (e.g. yank-rectangle) consumes excessive memory (4GB+ for 90MB of
> text)".  I recommend upgrading to version 26.
>
>
>

[-- Attachment #2: Type: text/html, Size: 1440 bytes --]

bug#38629: 25.2; garbage-collect doesn't reclaim large *compilation*

[Peter Ludemann]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

Upgrading to Emacs 26.3 seems to have fixed the memory problem but now
there's another problem ... emacs becomes incredibly sluggish (almost
unuseable) -- top(1) shows 70-100% CPU utilization even when the
compilation step isn't outputting anything. (I use GNU parallel on the
tests, with options to preserve the output order, so output happens at
intervals.)

The compilation command is "find ... | sort | nice parallel -j 8
--keep-order --group -L80" (on a 4 CPU machine).
I tried reducing the number of parallel jobs to 3, but that didn't help.

Should I open a new bug for this? If so, what information can I collect, to
help determine the cause?

On Mon, 16 Dec 2019 at 06:24, Noam Postavsky <npostavs@gmail.com> wrote:

> Peter Ludemann <peter.ludemann@gmail.com> writes:
>
> > I ran a large compilation (to the *compilation* buffer) (232,701 lines,
> > 52M).
>
> > In GNU Emacs 25.2.2 (x86_64-pc-linux-gnu, GTK+ Version 3.22.21)
> >  of 2017-09-22, modified by Debian built on lgw01-amd64-050
> > Windowing system distributor 'The X.Org Foundation', version
> 11.0.11906000
> > System Description: Ubuntu 18.04.3 LTS
>
> I think this is a variant of Bug#26952 - "repeated buffer insertion
> (e.g. yank-rectangle) consumes excessive memory (4GB+ for 90MB of
> text)".  I recommend upgrading to version 26.
>
>
>

[-- Attachment #2: Type: text/html, Size: 2207 bytes --]

bug#38629: 25.2; garbage-collect doesn't reclaim large *compilation*

[Peter Ludemann]

[-- Attachment #1: Type: text/plain, Size: 238 bytes --]

The sluggishness is not related to the *compilation* window, so we can
close this bug (as fixed by emacs 26.3) and I'll open a new bug report.

Thank-you for your help! (And thank-you to Kevin Kelley for setting up the
PPA for emacs26.3)

[-- Attachment #2: Type: text/html, Size: 552 bytes --]

bug#38629: 25.2; garbage-collect doesn't reclaim large *compilation*

[Noam Postavsky]

unarchive 26952
forcemerge 26952 38629
quit

Peter Ludemann <peter.ludemann@gmail.com> writes:

> The sluggishness is not related to the *compilation* window, so we can
> close this bug (as fixed by emacs 26.3) and I'll open a new bug report.

Thanks for checking.