bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Jose A. Ortega Ruiz]

It seems to be easy to make emacs (under X) to consume more and more
RAM, which is never released, by making it display images.  A extreme
(in my experience) case is animated GIFs, try:

  - emacs -Q
  - M-x eww RET https://xenodium.com/emacs-zones-to-lift-you-up/
  - RAM consumption grows to ~600Mb
  - R (redisplay page): RAM grows to ~1100Mb
  - R (redisplay page): RAM grows to ~1752Mb
  - R (redisplay page): RAM grows to ~2222Mb
  - rinse and repeat: RAM never goes down
  - (image-cache-size) reports a modest 82Mb
  - Kill buffer:  high RAM consumption is still at its maximum, even
    after (image-cache-size) goes to 0

My impression is that this bad behaviour is not limited to animated
gifts, although for regular images i don't have a solid recipe: what i
observe is that running long sessions in term mode for days of continous
use in an xterm, RAM tops at about 0.5Mb, while running similar sessions
under X (using the same packages and doing the same kind of things
inside of emacs), RAM will steadily increase.  The only difference
between the two scenarios i can think of is that in X i sometimes
display images, mainly in eww and, sometimes, inside HTML email messages
rendered via shr.  I also observe that while it's pretty common for
emacs in an x term (either run directly or connected to a demon) to
release memory (i.e., to have its RSS decrease), that almost never
happens when running under X: there RAM almost never goes down, no
matter what.


In GNU Emacs 29.0.50 (build 16, x86_64-pc-linux-gnu, cairo version 1.16.0)
 of 2022-07-14 built on rivendell
Repository revision: 9a888323c60c60fb37f471ef03f0bcdff91cb850
Repository branch: master
System Description: Debian GNU/Linux bookworm/sid

Configured using:
 'configure --prefix=/usr/local/stow/emacs --with-x-toolkit=no
 --with-imagemagick'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
IMAGEMAGICK JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2
M17N_FLT MODULES NOTIFY INOTIFY OLDXMENU PDUMPER PNG RSVG SECCOMP SOUND
SQLITE3 THREADS TIFF WEBP X11 XDBE XIM XINPUT2 XPM ZLIB

Important settings:
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  telega-root-auto-fill-mode: t
  telega-active-locations-mode: t
  telega-patrons-mode: t
  telega-mode-line-mode: t
  diff-hl-margin-mode: t
  global-diff-hl-mode: t
  eshell-vterm-mode: t
  eshell-syntax-highlighting-global-mode: t
  pdf-occur-global-minor-mode: t
  winner-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  global-auto-revert-mode: t
  shell-dirtrack-mode: t
  vertico-mode: t
  global-company-mode: t
  company-mode: t
  marginalia-mode: t
  persistent-scratch-autosave-mode: t
  global-so-long-mode: t
  pulsar-global-mode: t
  pulsar-mode: t
  display-battery-mode: t
  jao-minibuffer-mode: t
  minibuffer-electric-default-mode: t
  minibuffer-depth-indicate-mode: t
  xclip-mode: t
  repeat-mode: t
  savehist-mode: t
  recentf-mode: t
  save-place-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  column-number-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/jao/lib/elisp/org-static-blog/org-static-blog hides /home/jao/.emacs.d/elpa.29/org-static-blog-20220508.1410/org-static-blog
/home/jao/etc/emacs/site/custom hides /usr/local/stow/emacs/share/emacs/29.0.50/lisp/custom
/home/jao/.emacs.d/elpa.29/transient-20220527.2213/transient hides /usr/local/stow/emacs/share/emacs/29.0.50/lisp/transient

Features:
(shadow mailalias bbdb-message vertico-directory tramp-cmds sort
gnus-cite mm-archive mail-extr textsec uni-scripts idna-mapping
ucs-normalize uni-confusable textsec-check gnus-async gnus-bcklg
gnus-dup qp gnus-ml gnus-topic nnml bbdb-gnus network-stream bbdb-mua
gnus-icalendar icalendar ol-gnus nnselect org-capture gnus-delay
gnus-draft gnus-agent gnus-srvr gnus-score score-mode nnvirtual
gnus-cache gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime dig
gnus-sum nndraft nnmh gnus-demon nntp gnus-group gnus-undo gnus-start
gnus-dbus gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec
gnus-int gnus-range gnus-win org-duration org-appear org-agenda
org-refile cal-iso mule-util cal-move bigml bml-logs bml bml-misc
bml-whizzml bml-clojure bml-clj-tests bml-python bml-skels bml-utils
multisession sqlite whizzml-skeletons whizzml-mode lice sieve sieve-mode
sieve-manage sasl sasl-anonymous sasl-login sasl-plain jao-mpc
jao-random-album jao-lyrics jao-mpris telega-obsolete telega
telega-tdlib-events telega-webpage visual-fill-column telega-match
telega-root telega-info telega-chat telega-modes telega-company
telega-user telega-notifications telega-voip telega-msg telega-tme
telega-sticker telega-i18n telega-vvnote bindat telega-ffplay
telega-sort telega-filter telega-ins telega-folders telega-inline
telega-util telega-media telega-tdlib rainbow-identifiers dired-aux
telega-server telega-core cursor-sensor telega-customize emacsbug
jao-mullvad bluetooth enwc enwc-backend json-mode json-snatcher js
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs yaml-mode virtualenvwrapper gud ediprolog pie
haskell-doc inf-haskell haskell-decl-scan haskell haskell-completions
haskell-load haskell-commands highlight-uses-mode haskell-modules
haskell-sandbox haskell-navigate-imports haskell-repl haskell-svg
haskell-collapse hideshow haskell-debug haskell-interactive-mode
haskell-presentation-mode haskell-compile haskell-hoogle haskell-process
haskell-session haskell-mode haskell-cabal haskell-utils
haskell-font-lock haskell-indentation haskell-string
haskell-sort-imports haskell-lexeme haskell-align-imports
haskell-complete-module haskell-ghc-support flymake-proc flymake
warnings dabbrev haskell-customize geiser-guile info-look geiser-repl
geiser-compile geiser-debug geiser-image geiser-capf geiser-doc
geiser-menu geiser-edit etags fileloop xref project geiser-completion
geiser-autodoc geiser-eval geiser-connection geiser-syntax scheme
geiser-impl help-fns radix-tree geiser-log geiser-popup view
geiser-custom geiser-base geiser idris-mode idris-commands
idris-hole-list idris-ipkg-mode idris-tree-info idris-warnings-tree
idris-info idris-repl idris-highlight-input idris-prover inferior-idris
idris-warnings idris-log idris-events idris-simple-indent idris-syntax
idris-common-utils idris-settings idris-keys idris-core idris-compat
prop-menu package-lint finder lisp-mnt edit-list outline-minor-faces
git-modes gitignore-mode gitconfig-mode conf-mode gitattributes-mode
git-link git-timemachine diff-hl-margin diff-hl-dired diff-hl log-view
vc-dir ewoc vc jao-eshell-here eshell-autojump em-dirs esh-var eshell-up
git-ps1-mode eshell-vterm em-term eshell-syntax-highlighting em-alias
vterm tramp tramp-loaddefs trampver tramp-integration files-x
tramp-compat ls-lisp face-remap term ehelp vterm-module term/xterm xterm
jao-custom-email bbdb-anniv bbdb-com bbdb bbdb-site timezone randomsig
nov esxml-query saveplace-pdf-view pdf-occur ibuf-ext ibuffer
ibuffer-loaddefs tablist tablist-filter semantic/wisent/comp
semantic/wisent semantic/wisent/wisent semantic/util-modes semantic/util
semantic semantic/tag semantic/lex semantic/fw mode-local cedet
pdf-isearch pdf-misc socks elpher jao-custom-eww ol-eww jao-eww-session
eww xdg url-queue mm-url gnus nnheader range markdown-toc
jao-custom-blog htmlize jao-custom-org jao-org-links jao-doc-view
doc-view pdf-tools pdf-view pdf-cache pdf-info tq pdf-util pdf-macs
image-mode exif ol-info ol-eshell esh-mode eshell esh-cmd esh-ext
esh-opt esh-proc esh-io esh-arg esh-module esh-groups esh-util
jao-org-notes ob-shell ob-scheme ob-python python ob-org ob-ocaml
ob-makefile ob-haskell ob-gnuplot ob-clojure ob-calc calc-store
calc-trail ob-prolog prolog smie align org-tempo tempo ox-texinfo
ox-latex ox-html table ox-ascii ox-publish ox org-fragtog org-element
avl-tree generator jao-afio winner consult-recoll embark-consult
consult-vertico consult compat-28 magit-bookmark bookmark jao-recoll org
ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote
org-src ob-comint org-pcomplete org-list org-faces org-entities
org-version ob-emacs-lisp ob-core ob-eval org-table oc-basic bibtex ol
org-keys oc org-compat org-macs org-loaddefs find-func
jao-custom-completion embark-vc s code-review code-review-actions
code-review-comment code-review-section code-review-bitbucket
code-review-faces shr pixel-fill kinsoku url-file url-dired svg dom
emojify apropos tar-mode arc-mode archive-mode ht code-review-gitlab
code-review-utils code-review-parse-hunk code-review-github
code-review-db uuidgen calc-misc calc-ext calc calc-loaddefs rect
calc-macs a code-review-interfaces deferred forge-list forge-commands
forge-semi forge-bitbucket buck forge-gogs gogs forge-gitea gtea
forge-gitlab glab forge-github ghub-graphql treepy gsexp ghub let-alist
gnutls forge-notify forge-revnote forge-pullreq forge-issue forge-topic
yaml parse-time iso8601 bug-reference forge-post markdown-mode
edit-indirect noutline outline forge-repo forge forge-core forge-db
closql emacsql-sqlite advice emacsql emacsql-compiler url-http url-auth
url-gw nsm magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func magit-diff smerge-mode diff git-commit log-edit message
sendmail yank-media puny rfc822 mml mml-sec gnus-util time-date
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader
pcvs-util add-log magit-core magit-autorevert autorevert filenotify
magit-margin magit-transient magit-process with-editor shell pcomplete
magit-mode magit-git magit-base magit-section crm dash compat-27
compat-26 embark ffap thingatpt vertico company-keywords company-dabbrev
company-files company-capf company pcase marginalia orderless imenu
jao-skel-latex jao-skel-haskell jao-compilation jao-skel-lisp
jao-skel-geiser jao-skel skeleton autoinsert wgrep grep compile
text-property-search comint ring jka-compr dired-x dired dired-loaddefs
persistent-scratch so-long cal-china lunar solar cal-dst cal-bahai
cal-islam cal-hebrew holidays holiday-loaddefs vc-git diff-mode
vc-dispatcher appt diary-lib diary-loaddefs cal-menu calendar
cal-loaddefs pulsar pulse color tmr jao-tracking tracking shorten
jao-notify alert log4e notifications gntp battery jao-mode-line
jao-minibuffer minibuf-eldef mb-depth xclip diminish
jao-light-term-theme jao-themes ansi-color disp-table server pinentry
epa-file epa derived epg rfc6068 epg-config transient format-spec compat
cus-edit pp cus-load repeat edmacro kmacro jao-shell jao-sleep dbus xml
savehist recentf tree-widget wid-edit saveplace jao-gnus-private
gnu-elpa-keyring-update cl-extra help-mode use-package
use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core finder-inf
tex-site bbdb-autoloads cider-autoloads clojure-mode-autoloads
company-autoloads diff-hl-autoloads edit-indirect-autoloads
elpher-autoloads embark-consult-autoloads consult-autoloads
embark-vc-autoloads embark-autoloads code-review-autoloads
forge-autoloads ghub-autoloads git-link-autoloads idris-mode-autoloads
json-mode-autoloads rx link-hint-autoloads magit-autoloads
git-commit-autoloads magit-section-autoloads marginalia-autoloads
markdown-mode-autoloads org-appear-autoloads org-fragtog-autoloads
outline-minor-faces-autoloads paredit-autoloads pdf-tools-autoloads
persistent-scratch-autoloads pulsar-autoloads racket-mode-autoloads
request-autoloads switch-window-autoloads telega-autoloads tmr-autoloads
vertico-autoloads dash-autoloads vterm-autoloads with-editor-autoloads
info compat-autoloads xclip-autoloads yaml-autoloads package browse-url
url url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x
map byte-opt gv bytecomp byte-compile cconv url-vars cl-loaddefs cl-lib
rmc iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo xinput2 x
multi-tty make-network-process emacs)

Memory information:
((conses 16 1050346 86454)
 (symbols 48 89440 112)
 (strings 32 323218 19301)
 (string-bytes 1 9679236)
 (vectors 16 148160)
 (vector-slots 8 2070697 64427)
 (floats 8 1357 901)
 (intervals 56 4457 7319)
 (buffers 992 29))

-- 
The greatest of faults, I should say, is to be conscious of none.
 -Thomas Carlyle, writer (1795-1881)

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Eli Zaretskii]

> From: "Jose A. Ortega Ruiz" <mail@jao.io>
> Date: Thu, 14 Jul 2022 02:35:46 +0100
> 
> 
> It seems to be easy to make emacs (under X) to consume more and more
> RAM, which is never released, by making it display images.  A extreme
> (in my experience) case is animated GIFs, try:
> 
>   - emacs -Q
>   - M-x eww RET https://xenodium.com/emacs-zones-to-lift-you-up/
>   - RAM consumption grows to ~600Mb
>   - R (redisplay page): RAM grows to ~1100Mb
>   - R (redisplay page): RAM grows to ~1752Mb
>   - R (redisplay page): RAM grows to ~2222Mb
>   - rinse and repeat: RAM never goes down
>   - (image-cache-size) reports a modest 82Mb
>   - Kill buffer:  high RAM consumption is still at its maximum, even
>     after (image-cache-size) goes to 0

Run some utility that displays the memory-map of an application, and
you will see that most of that memory is free for use.  Emacs just
didn't release it to the OS, but kept it in the memory pages allocated
to the process, for future allocations.

The strategy for releasing memory to the OS is in glibc, not under our
control.  Last time we talked with glibc developers, they maintained
that this is the expected and correct behavior.

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Eli Zaretskii]

> Cc: 56546@debbugs.gnu.org
> Date: Thu, 14 Jul 2022 08:45:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: "Jose A. Ortega Ruiz" <mail@jao.io>
> > Date: Thu, 14 Jul 2022 02:35:46 +0100
> > 
> > 
> > It seems to be easy to make emacs (under X) to consume more and more
> > RAM, which is never released, by making it display images.  A extreme
> > (in my experience) case is animated GIFs, try:
> > 
> >   - emacs -Q
> >   - M-x eww RET https://xenodium.com/emacs-zones-to-lift-you-up/
> >   - RAM consumption grows to ~600Mb
> >   - R (redisplay page): RAM grows to ~1100Mb
> >   - R (redisplay page): RAM grows to ~1752Mb
> >   - R (redisplay page): RAM grows to ~2222Mb
> >   - rinse and repeat: RAM never goes down
> >   - (image-cache-size) reports a modest 82Mb
> >   - Kill buffer:  high RAM consumption is still at its maximum, even
> >     after (image-cache-size) goes to 0
> 
> Run some utility that displays the memory-map of an application, and
> you will see that most of that memory is free for use.  Emacs just
> didn't release it to the OS, but kept it in the memory pages allocated
> to the process, for future allocations.
> 
> The strategy for releasing memory to the OS is in glibc, not under our
> control.  Last time we talked with glibc developers, they maintained
> that this is the expected and correct behavior.

Btw, did you try

  M-: (clear-image-cache) RET

followed by "M-x garbage-collect RET"?

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Btw, did you try
>
>   M-: (clear-image-cache) RET

The problem is the animation cache, not the image cache, I think.  And I
see that I forgot to make clear-image-cache clear the animation cache,
which it should.

And also -- the animation cache should be pruned regularly, but it's not
only pruned when doing animated images.  I think it should also be
pruned from...  gc?  Or is there a different place it'd make sense to
prune that cache from?

Finally, image-cache-size should also report the animation cache size, I
think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mail@jao.io,  56546@debbugs.gnu.org
> Date: Thu, 14 Jul 2022 10:53:12 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Btw, did you try
> >
> >   M-: (clear-image-cache) RET
> 
> The problem is the animation cache, not the image cache, I think.  And I
> see that I forgot to make clear-image-cache clear the animation cache,
> which it should.

When we remove an image from the cache, its animations should also be
removed, I agree.

> And also -- the animation cache should be pruned regularly, but it's not
> only pruned when doing animated images.  I think it should also be
> pruned from...  gc?  Or is there a different place it'd make sense to
> prune that cache from?

GC could be problematic.  Imagine someone who has a frame displayed at
all times animating something -- you don't want to clear those caches,
right?

I think we should perhaps track which images are actually displayed,
and when was the last time they were displayed, and evict them from
the cache after enough time has passed since the image was last
displayed.

> Finally, image-cache-size should also report the animation cache size, I
> think.

That'd help, yes.

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> When we remove an image from the cache, its animations should also be
> removed, I agree.

The image cache is for displayed images, so an animated image will
usually have dozens of image cache entries, so it doesn't work that way.

> GC could be problematic.  Imagine someone who has a frame displayed at
> all times animating something -- you don't want to clear those caches,
> right?

No, but the cache should be pruned of old entries.  (Which is what the
pruning function does.)

>> Finally, image-cache-size should also report the animation cache size, I
>> think.
>
> That'd help, yes.

Looking at that now, that's easier said than done, because the animation
caches are opaque -- we call out to support libraries, and it doesn't
look like all of them have a way to say how much memory they use.  (I'm
looking at the webp functions right now...)

But we can at least include the size of the main data blob (e.g., the
.webp data itself); we know the size of that for all the anim caches.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> But we can at least include the size of the main data blob (e.g., the
> .webp data itself); we know the size of that for all the anim caches.

No, not even that -- the cache is pretty opaque in general.  Hm...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mail@jao.io,  56546@debbugs.gnu.org
> Date: Thu, 14 Jul 2022 12:01:16 +0200
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > But we can at least include the size of the main data blob (e.g., the
> > .webp data itself); we know the size of that for all the anim caches.
> 
> No, not even that -- the cache is pretty opaque in general.  Hm...

Don't we allocate memory for the cached stuff?  If so, we could count
the bytes there, and record them in the cache itself.

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Don't we allocate memory for the cached stuff?  If so, we could count
> the bytes there, and record them in the cache itself.

No, we just call the gif/webp functions, and they allocate stuff on
their own.  For webp, we do know the size of the main data blob, but not
for GIF -- we just call DGifOpenFileName+DGifSlurp, and looking over the
documentation, there doesn't seem to be any way to get it to cough up
how much data it allocated.

In the non-file case, we know the size of the data blob, and I guess we
could just see how big the GIF file is in the DGifOpenFileName case, and
assume that it allocates memory for the entire file.  (Which may or may
not be true -- perhaps it uses mmap into the file instead, or
whatever...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Jose A. Ortega Ruiz]

On Thu, Jul 14 2022, Eli Zaretskii wrote:

>> From: "Jose A. Ortega Ruiz" <mail@jao.io>
>> Date: Thu, 14 Jul 2022 02:35:46 +0100
>> 
>> 
>> It seems to be easy to make emacs (under X) to consume more and more
>> RAM, which is never released, by making it display images.  A extreme
>> (in my experience) case is animated GIFs, try:
>> 
>>   - emacs -Q
>>   - M-x eww RET https://xenodium.com/emacs-zones-to-lift-you-up/
>>   - RAM consumption grows to ~600Mb
>>   - R (redisplay page): RAM grows to ~1100Mb
>>   - R (redisplay page): RAM grows to ~1752Mb
>>   - R (redisplay page): RAM grows to ~2222Mb
>>   - rinse and repeat: RAM never goes down
>>   - (image-cache-size) reports a modest 82Mb
>>   - Kill buffer:  high RAM consumption is still at its maximum, even
>>     after (image-cache-size) goes to 0
>
> Run some utility that displays the memory-map of an application, and
> you will see that most of that memory is free for use.  Emacs just
> didn't release it to the OS, but kept it in the memory pages allocated
> to the process, for future allocations.

then, why is the process not reusing that memory the second, third,
etc. times i reload the exact same page, with the exact same images in
the same buffer? if the first allocation reserved that memory and the
process has it, after killing the buffer and opening the page again, it
has plenty of free for use memory at its disposal, yet it allocates a
fresh new block of 600Mb every time.  with that behaviour, i just have
to open 10 times the same page to run out of RAM in my computer.  not
even firefox is that hungry.

> The strategy for releasing memory to the OS is in glibc, not under our
> control.  Last time we talked with glibc developers, they maintained
> that this is the expected and correct behavior.

well, all i can say is that none of the programs i use in my linux
distribution (debian unstable), which are compiled against the same
libraries, has shown such a dramatic increase of RAM consumption in the
last months.  so they seem to have optimized their allocation behaviour
very narrowly against the way i use emacs! ;)

perhaps i'll try to compile emacs using older gcc versions (up to version
27, emacs was really slim RAM-wise for me).

thanks,
jao

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Lars Ingebrigtsen]

"Jose A. Ortega Ruiz" <mail@jao.io> writes:

>   - emacs -Q
>   - M-x eww RET https://xenodium.com/emacs-zones-to-lift-you-up/
>   - RAM consumption grows to ~600Mb
>   - R (redisplay page): RAM grows to ~1100Mb
>   - R (redisplay page): RAM grows to ~1752Mb
>   - R (redisplay page): RAM grows to ~2222Mb
>   - rinse and repeat: RAM never goes down
>   - (image-cache-size) reports a modest 82Mb
>   - Kill buffer:  high RAM consumption is still at its maximum, even
>     after (image-cache-size) goes to 0

I think this should all be fixed on the current trunk now -- can you
check?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Eli Zaretskii]

> Cc: 56546@debbugs.gnu.org
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 14 Jul 2022 18:59:20 +0200
> 
> "Jose A. Ortega Ruiz" <mail@jao.io> writes:
> 
> >   - emacs -Q
> >   - M-x eww RET https://xenodium.com/emacs-zones-to-lift-you-up/
> >   - RAM consumption grows to ~600Mb
> >   - R (redisplay page): RAM grows to ~1100Mb
> >   - R (redisplay page): RAM grows to ~1752Mb
> >   - R (redisplay page): RAM grows to ~2222Mb
> >   - rinse and repeat: RAM never goes down
> >   - (image-cache-size) reports a modest 82Mb
> >   - Kill buffer:  high RAM consumption is still at its maximum, even
> >     after (image-cache-size) goes to 0
> 
> I think this should all be fixed on the current trunk now -- can you
> check?

FWIW, on my system the pattern of the memory footprint didn't change
after this.  I still see the footprint going up after each "M-x eww",
and killing the buffer and invoking clear-image-cache causes Emacs to
go back to somewhat higher than its original size.

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Glenn Morris]

564f6c breaks --without-all --without-x builds.
Ref eg https://hydra.nixos.org/build/183877918

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> 564f6c breaks --without-all --without-x builds.
> Ref eg https://hydra.nixos.org/build/183877918

Yup.  Now fixed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Jose A. Ortega Ruiz]

On Thu, Jul 14 2022, Lars Ingebrigtsen wrote:

> "Jose A. Ortega Ruiz" <mail@jao.io> writes:
>
>>   - emacs -Q
>>   - M-x eww RET https://xenodium.com/emacs-zones-to-lift-you-up/
>>   - RAM consumption grows to ~600Mb
>>   - R (redisplay page): RAM grows to ~1100Mb
>>   - R (redisplay page): RAM grows to ~1752Mb
>>   - R (redisplay page): RAM grows to ~2222Mb
>>   - rinse and repeat: RAM never goes down
>>   - (image-cache-size) reports a modest 82Mb
>>   - Kill buffer:  high RAM consumption is still at its maximum, even
>>     after (image-cache-size) goes to 0
>
> I think this should all be fixed on the current trunk now -- can you
> check?

Yes, it essentially does.  Only the first jump to ~630Mb happens, and
then memory stays there after repeated reloads.  If i do it often (or
quickly, i think) enough i can make it occasionally jump to the ~1200Mb
mark, but for instance cleaning the image cache makes it go back to
~600Mb (so there are clearly situations where big chunks of memory do
get released back to the system, thankfully).

So i think you did fix the problem, thanks a lot Lars!  

While we're here, i was inspecting the image-cache-size during the
reloads of the page, and i saw every reload increases that size by about
30Mb (then it will eventually go down as old images are
released)... given that, the big initial jump of ~500Mb in RAM
consumption surprises me a little, but i see reasons why it can be
completely normal for some initial allocation (and how it might be
staying there because of libgcc policies).  I was also naively expecting
that, given that i'm reloading the same page with the "same" images,
they'd be found in the cache the second and subsequent times, but, since
the cache size increases, i see that they're not considered the "same"
by the caching mechanism; i can again see reasons for that behaviour (i
am sure the cache is not judging image equality using "image urls"),
just mentioning it in case it's unexpected or there's an easy way of
reusing the cached images).

Anyway, thanks a lot for bearing with me and solving the problem: this
bug is all but fixed now from my point of view.

Cheers,
jao
-- 
That's a high price to pay for a theoretically inelegant misfeature
that's seldom used correctly in portable code.
  -Will Clinger, r6rs-discuss mailing list

bug#56546: 29.0.50; unbounded RAM comsumption when displaying images

[Lars Ingebrigtsen]

"Jose A. Ortega Ruiz" <mail@jao.io> writes:

> I was also naively expecting
> that, given that i'm reloading the same page with the "same" images,
> they'd be found in the cache the second and subsequent times, but, since
> the cache size increases, i see that they're not considered the "same"
> by the caching mechanism; i can again see reasons for that behaviour (i
> am sure the cache is not judging image equality using "image urls"),
> just mentioning it in case it's unexpected or there's an easy way of
> reusing the cached images).

From Emacs' point of view, these aren't "the same", because they're
not taken (directly) from files, I think.

> Anyway, thanks a lot for bearing with me and solving the problem: this
> bug is all but fixed now from my point of view.

Thanks for checking; I'm closing this bug report, then.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no