bug#43406: Emacs 27.1 memory consumption grows indefinitely

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Ludovic Courtès]

Hello,

Since I upgraded to Emacs 27.1, its memory consumption grows without
bounds, to the point that it gets OOM-killed after just a couple of
hours (meaning that it’s used most of the 16G of RAM of my laptop!).

Does that ring a bell to anyone?  I have 50+ ^emacs- packages in my
profile; any clues on what could be the cause before diving further?

--8<---------------cut here---------------start------------->8---
$ guix package -I ^emacs$
emacs   27.1    out     /gnu/store/ipgb6s3phz3n740xlcdzaxnbgvncdh11-emacs-27.1
$ guix describe
Generacio 157   Sep 10 2020 08:49:21    (nuna)
  guix 7090159
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 7090159c23d6345992ab976d71fefeb1583cfcdf
$ uname -rm
5.8.8-gnu x86_64
--8<---------------cut here---------------end--------------->8---

Ludo’.

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> Since I upgraded to Emacs 27.1, its memory consumption grows without
> bounds, to the point that it gets OOM-killed after just a couple of
> hours (meaning that it’s used most of the 16G of RAM of my laptop!).
>
> Does that ring a bell to anyone?  I have 50+ ^emacs- packages in my
> profile; any clues on what could be the cause before diving further?

Here's another data point: although I haven't paid attention to the
memory consumption of Emacs 27.1, I've used it successfully for at least
10 hours at a time (probably more) on my X200 with only 4G of RAM.  I
use 'emacs-no-x-toolkit' with only a few emacs packages:

  emacs-no-x-toolkit
  emacs-paredit
  emacs-magit
  emacs-geiser
  emacs-org-caldav
  emacs-nov-el
  notmuch

I haven't actually used 'emacs-org-caldav' or 'emacs-nov-el' recently.
Paredit, Geiser, Magit, and Notmuch's Emacs mail client are the only
out-of-tree Emacs packages that I actively use.

     Mark  (boring Emacs user :)

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Since I upgraded to Emacs 27.1, its memory consumption grows without
> bounds, to the point that it gets OOM-killed after just a couple of
> hours (meaning that it’s used most of the 16G of RAM of my laptop!).
[…]
> $ guix package -I ^emacs$
> emacs   27.1    out     /gnu/store/ipgb6s3phz3n740xlcdzaxnbgvncdh11-emacs-27.1

I have not observed this with /gnu/store/0v2r0d9gwbks7gmg5rygjb9mpb8i1bhl-emacs-27.1

(emacs-uptime) says "5 days, 11 hours, 38 minutes, 57 seconds"

-- 
Ricardo

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[zimoun]

> > emacs   27.1    out     /gnu/store/ipgb6s3phz3n740xlcdzaxnbgvncdh11-emacs-27.1

With the exact store item, M-x emacs-uptime says "3 days, 3 hours, 4
minutes, 7 seconds".
My kernel is 4.19.0-10 from Debian.

I use these packages:
            "diminish"
            "yasnippet"
            "smex"
            "counsel"
            "ivy"
            "ivy-rich"
            "magit"
            "debbugs"
            "ag"
            "org"
            "org-contrib"
            "flycheck"
            "auctex"
            "auto-dictionary-mode"
            "ox-pandoc"
            "pandoc-mode"
            "htmlize"
            "org-re-reveal"
            "typo"
            "pdf-tools"
            "paredit"
            "geiser"
            "guix"
            ;; "ess"
            "julia-mode"
            "pyvenv"
            "haskell-mode"
            "tuareg"
            "rust-mode"
            "lua-mode"
            "markdown-mode"
            "google-c-style"
            "graphviz-dot-mode"
            "skewer-mode"
            "ws-butler"
            "fill-column-indicator"
            "page-break-lines"
            "info-plus"
            "keyfreq"
in addition to the ones in core, in case it helps.

Cheers,
simon

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> Since I upgraded to Emacs 27.1, its memory consumption grows without
> bounds, to the point that it gets OOM-killed after just a couple of
> hours (meaning that it’s used most of the 16G of RAM of my laptop!).
>
> Does that ring a bell to anyone?  I have 50+ ^emacs- packages in my
> profile; any clues on what could be the cause before diving further?
>
> $ guix package -I ^emacs$
> emacs   27.1    out     /gnu/store/ipgb6s3phz3n740xlcdzaxnbgvncdh11-emacs-27.1
> $ guix describe
> Generacio 157   Sep 10 2020 08:49:21    (nuna)
>   guix 7090159
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     branch: master
>     commit: 7090159c23d6345992ab976d71fefeb1583cfcdf

Emacs is about to crash :-), so let me share this finding from
M-x profiler-report (mem) on a sample of a few minutes:

--8<---------------cut here---------------start------------->8---
- timer-event-handler                                     805,671,418  87%
 - apply                                                  804,987,874  87%
  + helm-ff--cache-mode-refresh                           802,103,698  86%
  + emms-playing-time-display                               1,349,461   0%
  + battery-update-handler                                  1,109,005   0%
  + #<compiled 0x1ff1153b5547>                                221,094   0%
  + #<compiled 0x1ff11537c089>                                 49,632   0%
  + blink-cursor-start                                         30,368   0%
  + auto-revert-buffers                                        29,480   0%
  + highlight-symbol-temp-highlight                            22,076   0%
  + mouse-avoidance-fancy                                      20,440   0%
  + display-time-event-handler                                 18,383   0%
  + show-paren-function                                        13,464   0%
  + erc-server-send-ping                                        7,500   0%
    jit-lock-force-redisplay                                    4,136   0%
  + blink-cursor-timer-function                                 4,136   0%
  + nnimap-keepalive                                            2,400   0%
  + gnus-demon-run-callback                                     1,545   0%
  + isearch-lazy-highlight-start                                1,056   0%
 + timer-activate                                             306,016   0%
 + timer-inc-time                                             191,264   0%
 + timer-activate-when-idle                                    92,928   0%
+ command-execute                                         100,936,674  10%
+ erc-server-filter-function                                5,178,272   0%
+ redisplay_internal (C function)                           4,988,306   0%
--8<---------------cut here---------------end--------------->8---

It’s Helm eating memory at each timer tick.  That’s with:

--8<---------------cut here---------------start------------->8---
emacs-helm-wikipedia	0.0.0-1.126f044	out	/gnu/store/iy08rcx4hbravm9k3qlinc4wvklp3pqi-emacs-helm-wikipedia-0.0.0-1.126f044
emacs-helm-swoop	3.0.0	out	/gnu/store/qnr6asnk0b360ilx8cnrp35kwrh4fsf7-emacs-helm-swoop-3.0.0
emacs-helm-shell-history	0.1-1.110d3c3	out	/gnu/store/ljgfa72mw8jx6br9khl7idcmrw4i33hc-emacs-helm-shell-history-0.1-1.110d3c3
emacs-helm-make	0.1.0-1.feae8df	out	/gnu/store/lnizl47zxlj03waficpy7fvfrx1mdqn7-emacs-helm-make-0.1.0-1.feae8df
emacs-helm-ls-git	1.9.1-1.76654c7	out	/gnu/store/897fkkh4bzpvbd06kf7vxr53hgkk0vfs-emacs-helm-ls-git-1.9.1-1.76654c7
emacs-helm-gtags	1.5.7	out	/gnu/store/lhibi05hvvn3zglcx2ii44yw723pfyja-emacs-helm-gtags-1.5.7
emacs-helm	3.6.5	out	/gnu/store/ka9lph0hpzaky0sa52zf09469apkhb68-emacs-helm-3.6.5
emacs-helm-system-packages	1.10.1-1.6572340	out	/gnu/store/lvkan5dsbcdkvl3y1fjmsgb423nzmmqc-emacs-helm-system-packages-1.10.1-1.6572340
emacs-helm-emms	1.3-3.37e5aa0	out	/gnu/store/g06lnxvyqnp9d2yqgfanp69350f7m2yp-emacs-helm-emms-1.3-3.37e5aa0
--8<---------------cut here---------------end--------------->8---

Cc’ing a couple of people who I think use Helm and hopefully have
valuable advice to share.  :-)

Ludo’.

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Ludovic Courtès]

More precisely:

--8<---------------cut here---------------start------------->8---
- timer-event-handler                                     805,671,418  87%
 - apply                                                  804,987,874  87%
  - helm-ff--cache-mode-refresh                           802,103,698  86%
   - helm-ff--cache-mode-refresh-1                        801,321,290  86%
    - helm-ff-refresh-cache                               801,237,818  86%
     - maphash                                            801,237,818  86%
      + #<compiled 0xec7961>                              801,237,818  86%
      helm-ff--cache-mode-max-idle-time                         6,384   0%
   + run-with-idle-timer                                      651,464   0%
--8<---------------cut here---------------end--------------->8---

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 233 bytes --]

I believe believe helm-ff cache came with Helm 3.6.4.

I find it slow (and not so useful?) so I've disabled it with:

(setq helm-ff-keep-cached-candidates nil)

Hope that helps!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Ludovic Courtès]

Hi!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> I believe believe helm-ff cache came with Helm 3.6.4.
>
> I find it slow (and not so useful?) so I've disabled it with:
>
> (setq helm-ff-keep-cached-candidates nil)

Michael Rohleder <mike@rohleder.de> skribis:

> What helped me was disabling it with:
> (setq helm-ff-keep-cached-candidates nil)

I definitely helps, thanks!

I seem to have something else leaking still, but this time it’s
primarily Gnus and/or EIEIO:

--8<---------------cut here---------------start------------->8---
- command-execute                                       1,117,368,135  88%
 - call-interactively                                   1,117,276,295  88%
  - funcall-interactively                               1,116,838,121  88%
   - gnus-group-save-newsrc                               460,241,156  36%
    - gnus-save-newsrc-file                               460,241,156  36%
     - gnus-run-hooks                                     429,283,350  34%
      - apply                                             429,283,350  34%
       - run-hooks                                        429,283,350  34%
        - gnus-registry-save                              429,283,350  34%
         - eieio-persistent-save                          427,938,700  33%
          - apply                                         427,938,700  33%
           + #<compiled 0x6005bf1>                        427,938,700  33%
         + gnus-registry--munge-group-names                 1,340,064   0%
         + gnus-message                                         3,530   0%
     + gnus-gnus-to-quick-newsrc-format                    20,126,054   1%
     + save-buffer                                          9,613,098   0%
     + gnus-agent-save-local                                  595,652   0%
     + gnus-gnus-to-newsrc-format                             577,848   0%
     + gnus-group-set-mode-line                                 6,458   0%
     + gnus-message                                             3,736   0%
     + gnus-dribble-delete-file                                 3,352   0%
       gnus-get-buffer-create                                   2,138   0%
     + gnus-prune-buffers                                       1,056   0%
   - gnus-topic-read-group                                208,616,775  16%
    - gnus-group-read-group                               208,616,775  16%
     - gnus-summary-read-group                            208,616,775  16%
      - gnus-summary-read-group-1                         208,616,775  16%
       - gnus-run-hooks                                   163,821,469  13%
        - apply                                           163,821,469  13%
         - run-hooks                                      163,821,469  13%
          - spam-find-spam                                162,365,376  12%
           - mapcar                                       162,328,432  12%
            + #<compiled 0x86338dd>                       162,328,432  12%
           + gnus-parameter-spam-autodetect                    35,888   0%
           + gnus-parameter-spam-autodetect-methods             1,056   0%
          + gnus-apply-kill-file                            1,456,093   0%
       + gnus-select-newsgroup                             31,456,212   2%
--8<---------------cut here---------------end--------------->8---

Any cache to turn off?  :-)

Ludo’.

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 132 bytes --]

I don't use Gnus, sorry.
Can you reproduce on a vanilla configuration (emacs -Q)?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Michael Rohleder]

[-- Attachment #1: Type: text/plain, Size: 281 bytes --]

Hi Ludovic,

I also had trouble with that helm caching. I had the impression that it
was because of newer helm versions not emacs versions.

What helped me was disabling it with:
(setq helm-ff-keep-cached-candidates nil)

-- 
Alles fließt, nichts bleibt.
     Heraklit

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Eli Zaretskii via web]

Please don't use "memory profile" as any indication of the Emacs memory usage: it is entirely unrelated to memory.

The Emacs profiler only profiles CPU usage.  It samples the PC register using a special signal (SIGPROF), which is the usual method of sampling CPU in profilers.  For those systems which don't have SIGPROF, it can fall back on sampling the CPU whenever Emacs calls one of the memory allocation functions -- this is called "mem" profiling.

So the "memory" profiles shown here don't show memory usage.

bug#43406: Emacs 27.1 memory consumption grows indefinitely

[Ludovic Courtès]

Hi Eli,

Eli Zaretskii via web <issues.guix.gnu.org@elephly.net> skribis:

> Please don't use "memory profile" as any indication of the Emacs memory usage: it is entirely unrelated to memory.
>
> The Emacs profiler only profiles CPU usage.  It samples the PC register using a special signal (SIGPROF), which is the usual method of sampling CPU in profilers.  For those systems which don't have SIGPROF, it can fall back on sampling the CPU whenever Emacs calls one of the memory allocation functions -- this is called "mem" profiling.
>
> So the "memory" profiles shown here don't show memory usage.

Understood, that’s the same thing as Guile’s ‘gcprof’.  :-)

Thanks for your feedback,
Ludo’.