bug#61763: 30.0.50; Image Cache Size growth

bug#61763: 30.0.50; Image Cache Size growth

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi,

My image cache size grows really quick when browsing some images from
Emacs.  My recipe:

        emacs -Q
        M-: (clear-image-cache)
        Then, I open a image in a directory with other images.
        I hit 'n' 5 times (to open next 5 images).
        Each image is a JPG of about 1.3MiB.
        Then, M-x memory-report says:
           366 MiB  Total Image Cache Size

With such growth, Emacs quickly tells me that the memory is exhausted.


In GNU Emacs 30.0.50 (build 1, x86_64-unknown-openbsd7.2, cairo version
 1.17.8) of 2023-02-24 built on computer
Repository revision: 078cff71abc7125558ed492e894aa7d1b487d9bd
Repository branch: mgi/image-dired-fix
Windowing system distributor 'The X.Org Foundation', version 11.0.12101006
System Description: OpenBSD computer 7.2 GENERIC.MP#1052 amd64

Configured using:
 'configure --prefix=/home/manuel/emacs --bindir=/home/manuel/bin
 --with-x-toolkit=no --without-sound --without-compress-install
 CPPFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBXML2 MODULES NOTIFY KQUEUE OLDXMENU PDUMPER PNG RSVG
SQLITE3 THREADS TIFF TREE_SITTER WEBP X11 XDBE XIM XINPUT2 XPM ZLIB

Important settings:
  value of $LC_ALL: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Special

Minor modes in effect:
  display-time-mode: t
  display-battery-mode: t
  server-mode: t
  shell-dirtrack-mode: t
  repeat-mode: t
  desktop-save-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  button-mode: t

Load-path shadows:
/home/manuel/.emacs.d/elpa/ef-themes-0.10.0/theme-loaddefs hides /home/manuel/emacs/share/emacs/30.0.50/lisp/theme-loaddefs
/home/manuel/.emacs.d/elpa/transient-0.3.7/transient hides /home/manuel/emacs/share/emacs/30.0.50/lisp/transient

Features:
(shadow sort mail-extr emacsbug image-file image-converter dabbrev pulse
memory-report org-indent reveal sh-script executable texinfo
texinfo-loaddefs org-element org-persist org-id org-refile avl-tree
oc-basic ol-eww ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect
ol-docview ol-bibtex bibtex ol-bbdb ol-w3m ol-doi org-link-doi doc-view
jka-compr vc-hg conf-mode css-mode treesit smie sgml-mode facemenu eww
xdg url-queue mm-url make-mode pov-mode imenu vc-cvs vc-rcs log-view
pcvs-util vc-dir ewoc image-mode exif paredit edmacro autorevert
filenotify vc-git diff-mode vc bug-reference gnus-dired time battery
exwm-randr xcb-randr exwm-config ido exwm exwm-input xcb-keysyms xcb-xkb
exwm-manage exwm-floating xcb-cursor xcb-render exwm-layout
exwm-workspace exwm-core xcb-ewmh xcb-icccm xcb xcb-xproto xcb-types
xcb-debug kmacro server modus-operandi-theme modus-themes ytdious mingus
libmpdee reporter edebug debug backtrace transmission color calc-bin
calc-ext calc calc-loaddefs rect calc-macs w3m-load supercite regi
ebdb-message ebdb-gnus gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime
smime gnutls dig gnus-sum shr pixel-fill kinsoku url-file svg dom
gnus-group gnus-undo gnus-start gnus-dbus gnus-cloud nnimap nnmail
mail-source utf7 nnoo gnus-spec gnus-int gnus-range message sendmail
yank-media puny rfc822 mml mml-sec epa epg rfc6068 epg-config mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums
gmm-utils mailheader gnus-win gnus nnheader gnus-util mail-utils range
mm-util mail-prsvr ebdb-mua ebdb-com crm ebdb-format ebdb mailabbrev
eieio-opt cl-extra help-mode speedbar ezimage dframe eieio-base pcase
timezone org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro
org-src ob-comint org-pcomplete org-list org-footnote org-faces
org-entities ob-emacs-lisp ob-core ob-eval org-cycle org-table ol
org-fold org-fold-core org-keys oc org-loaddefs find-func org-version
org-compat org-macs visual-basic-mode cl web-mode derived disp-table
erlang-start smart-tabs-mode skeleton cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs slime-asdf grep
slime-tramp tramp rx tramp-loaddefs trampver tramp-integration cus-edit
cus-load wid-edit files-x tramp-compat shell pcomplete parse-time
iso8601 time-date ls-lisp format-spec slime-fancy slime-indentation
slime-cl-indent cl-indent slime-trace-dialog slime-fontifying-fu
slime-package-fu slime-references slime-compiler-notes-tree advice
slime-scratch slime-presentations bridge slime-macrostep macrostep
slime-mdot-fu slime-enclosing-context slime-fuzzy slime-fancy-trace
slime-fancy-inspector slime-c-p-c slime-editing-commands slime-autodoc
slime-repl slime-parse slime apropos compile text-property-search etags
fileloop generator xref project arc-mode archive-mode noutline outline
icons pp comint ansi-osc ansi-color ring hyperspec thingatpt
slime-autoloads view mule-util cal-china lunar solar cal-dst cal-bahai
cal-islam cal-hebrew holidays holiday-loaddefs vc-dispatcher vc-svn appt
diary-lib diary-loaddefs cal-menu calendar cal-loaddefs dired-aux
dired-x dired dired-loaddefs notifications dbus xml repeat easy-mmode
desktop frameset osm-autoloads rust-mode-autoloads compat-autoloads
ebdb-autoloads magit-autoloads debbugs-autoloads git-commit-autoloads
magit-section-autoloads ef-themes-autoloads with-editor-autoloads
paredit-autoloads dash-autoloads ytdious-autoloads
transmission-autoloads transient-autoloads exwm-autoloads
hyperbole-autoloads detached-autoloads info package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x
map byte-opt gv bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind kqueue lcms2 dynamic-setting system-font-setting
font-render-setting cairo xinput2 x multi-tty make-network-process
emacs)

Memory information:
((conses 16 725653 31835)
 (symbols 48 54284 4)
 (strings 32 174900 22384)
 (string-bytes 1 5521428)
 (vectors 16 101351)
 (vector-slots 8 2135369 211753)
 (floats 8 1008 384)
 (intervals 56 28286 106)
 (buffers 984 120))

-- 
Manuel Giraud

bug#61763: 30.0.50; Image Cache Size growth

[Eli Zaretskii]

> Date: Fri, 24 Feb 2023 18:00:00 +0100
> From:  Manuel Giraud via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> My image cache size grows really quick when browsing some images from
> Emacs.  My recipe:
> 
>         emacs -Q
>         M-: (clear-image-cache)
>         Then, I open a image in a directory with other images.
>         I hit 'n' 5 times (to open next 5 images).
>         Each image is a JPG of about 1.3MiB.
>         Then, M-x memory-report says:
>            366 MiB  Total Image Cache Size
> 
> With such growth, Emacs quickly tells me that the memory is exhausted.

Did you try to lower the value of image-cache-eviction-delay?

bug#61763: 30.0.50; Image Cache Size growth

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

> Did you try to lower the value of image-cache-eviction-delay?

No, it is 300.  But isn't there something wrong with this growth rate?
Can you reproduce (if you have time)?
-- 
Manuel Giraud

bug#61763: 30.0.50; Image Cache Size growth

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: 61763@debbugs.gnu.org
> Date: Fri, 24 Feb 2023 20:41:46 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> [...]
> 
> > Did you try to lower the value of image-cache-eviction-delay?
> 
> No, it is 300.  But isn't there something wrong with this growth rate?

Why do you think it's wrong?

> Can you reproduce (if you have time)?

Not sure what I need to reproduce.  I've visited 6 images of 2 MiB
each and Memory Report says I have 581 MiB in my image cache.
Meanwhile the memory footprint of the Emacs process is 350 MiB.  Is
this what you wanted to see?

bug#61763: 30.0.50; Image Cache Size growth

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

>> Can you reproduce (if you have time)?
>
> Not sure what I need to reproduce.  I've visited 6 images of 2 MiB
> each and Memory Report says I have 581 MiB in my image cache.
> Meanwhile the memory footprint of the Emacs process is 350 MiB.  Is
> this what you wanted to see?

Ok.  So in the same ballpark.  I thought there was something wrong on my
setup.

Why I think this growth is wrong is because it seems high.  And if I'm
browsing some images in Emacs, I'm getting a "Memory exhausted (restart
Emacs)" message after only 20 or 30 images.
-- 
Manuel Giraud

bug#61763: 30.0.50; Image Cache Size growth

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: 61763@debbugs.gnu.org
> Date: Fri, 24 Feb 2023 21:27:41 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> [...]
> 
> >> Can you reproduce (if you have time)?
> >
> > Not sure what I need to reproduce.  I've visited 6 images of 2 MiB
> > each and Memory Report says I have 581 MiB in my image cache.
> > Meanwhile the memory footprint of the Emacs process is 350 MiB.  Is
> > this what you wanted to see?
> 
> Ok.  So in the same ballpark.  I thought there was something wrong on my
> setup.
> 
> Why I think this growth is wrong is because it seems high.  And if I'm
> browsing some images in Emacs, I'm getting a "Memory exhausted (restart
> Emacs)" message after only 20 or 30 images.

JPEG compression is very good, it routinely compresses images with
ratios of 10:1 to 20:1.  If I use djpeg to convert the 2 MiB images I
used into BMP, I get 36 MiB BMP files -- that's a 1:18 expansion
ratio.  And Emacs converts each image to a pixmap for display, which
is basically similar to what I did.  Multiply that by 20 or 30, and
you get the numbers you see, I think.

The solution is to enlarge the VM for your machine (by enlarging swap,
for example).  If you don't keep those images displayed in windows,
lowering image-cache-eviction-delay might also help.

bug#61763: 30.0.50; Image Cache Size growth

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

> JPEG compression is very good, it routinely compresses images with
> ratios of 10:1 to 20:1.  If I use djpeg to convert the 2 MiB images I
> used into BMP, I get 36 MiB BMP files -- that's a 1:18 expansion
> ratio.  And Emacs converts each image to a pixmap for display, which
> is basically similar to what I did.  Multiply that by 20 or 30, and
> you get the numbers you see, I think.

Yes, one of the images I'm using is 3264 by 2448 pixels times 3 octets
(RGB, I guess) and we get about 23MiB... so those numbers seems ok.

> The solution is to enlarge the VM for your machine (by enlarging swap,
> for example).  If you don't keep those images displayed in windows,
> lowering image-cache-eviction-delay might also help.

In image.c line 2079, there is already a mecanism to automatically lower
this delay if the cache has grown large (so I think we're covered here).

But OTOH, at line 3010, we can see that this cache will grow no matter
what.  Maybe we should have parameter (maybe a custom) that limit this
growth up to a certain point and then start uncaching older images.
WDYT?
-- 
Manuel Giraud

bug#61763: 30.0.50; Image Cache Size growth

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: 61763@debbugs.gnu.org
> Date: Fri, 24 Feb 2023 22:32:49 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> [...]
> 
> > JPEG compression is very good, it routinely compresses images with
> > ratios of 10:1 to 20:1.  If I use djpeg to convert the 2 MiB images I
> > used into BMP, I get 36 MiB BMP files -- that's a 1:18 expansion
> > ratio.  And Emacs converts each image to a pixmap for display, which
> > is basically similar to what I did.  Multiply that by 20 or 30, and
> > you get the numbers you see, I think.
> 
> Yes, one of the images I'm using is 3264 by 2448 pixels times 3 octets
> (RGB, I guess) and we get about 23MiB... so those numbers seems ok.
> 
> > The solution is to enlarge the VM for your machine (by enlarging swap,
> > for example).  If you don't keep those images displayed in windows,
> > lowering image-cache-eviction-delay might also help.
> 
> In image.c line 2079, there is already a mecanism to automatically lower
> this delay if the cache has grown large (so I think we're covered here).

But that happens when the number of cached images is above 40, and you
are saying you have problems with memory before that.

Also, the limit of 40 images is _per_frame_, so if you show images in
separate frames, the automatic lowering will happen later, if at all.

> But OTOH, at line 3010, we can see that this cache will grow no matter
> what.

That's not what happens at line 3010.  There, we enlarge the number of
available slots in the cache, but not the number of cached images.
The memory taken by the cache itself is very small compared to the
memory used by the images.

> Maybe we should have parameter (maybe a custom) that limit this
> growth up to a certain point and then start uncaching older images.
> WDYT?

We cannot uncache an image that is being displayed in some window.
Doing so would immediately cause a crash on the next redisplay cycle.
So lowering the eviction delay is the mechanism you should use,
assuming it helps in your use case.  Did you try that, and if you did,
did it help?  For how much time do you display each image before you
discard its buffer or delete its window?

bug#61763: 30.0.50; Image Cache Size growth

[Manuel Giraud via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

>> Maybe we should have parameter (maybe a custom) that limit this
>> growth up to a certain point and then start uncaching older images.
>> WDYT?
>
> We cannot uncache an image that is being displayed in some window.
> Doing so would immediately cause a crash on the next redisplay cycle.
> So lowering the eviction delay is the mechanism you should use,
> assuming it helps in your use case.  Did you try that, and if you did,
> did it help?  For how much time do you display each image before you
> discard its buffer or delete its window?

I've try with image-cache-eviction-delay at 30 seconds and I never
exceed 700MiB as Image Cache and never hit a "Memory exhausted".  So
this is solved for this use case.  Thanks for your patience Eli!
-- 
Manuel Giraud

bug#61763: 30.0.50; Image Cache Size growth

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: 61763@debbugs.gnu.org
> Date: Sat, 25 Feb 2023 13:19:20 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> [...]
> 
> >> Maybe we should have parameter (maybe a custom) that limit this
> >> growth up to a certain point and then start uncaching older images.
> >> WDYT?
> >
> > We cannot uncache an image that is being displayed in some window.
> > Doing so would immediately cause a crash on the next redisplay cycle.
> > So lowering the eviction delay is the mechanism you should use,
> > assuming it helps in your use case.  Did you try that, and if you did,
> > did it help?  For how much time do you display each image before you
> > discard its buffer or delete its window?
> 
> I've try with image-cache-eviction-delay at 30 seconds and I never
> exceed 700MiB as Image Cache and never hit a "Memory exhausted".  So
> this is solved for this use case.  Thanks for your patience Eli!

Thanks, I'm therefore closing this bug.