bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

[Ihor Radchenko <yantar92@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

Thanks for the links. I got to know more about memory management now.
Now, it is clear why the memory is consumption is so high when I open a
bunch of images at the same time. However, it still does not explain my
observations during usual workflow.

For my real usage, I open pdfs (or images) one by one most of the time
and kill the buffers periodically.
Then, if the memory is freed upon closing a pdf buffer, I expect it to
be reused for opening a new buffer. Even though memory consumption is
expected to grow in this case, it should be in order of the largest
image I open (times maximum number of image buffers open at the same
time). But it is not what I see.

I did a small test by modifying my earlier lisp code to open and close
the same image list sequentially:

#+begin_src emacs-lisp
(dolist (file (directory-files "~/Tosort/pictures&photos/" 'full ".*jpg"))
  (find-file file)
  (mapc #'kill-buffer (seq-filter (apply-partially #'string-match ".+.jpg$") (mapcar #'buffer-name (buffer-list)))))
#+end_src

The resulting memory usage graph is attached.

What we can see is that the memory is indeed growing (as expected).
Moreover, the memory consumption does not increase as much as if we open
all the images together. However, the final heap size appears to be over
400Mb (from smaps), which is almost half of what was observed with all
the images open at the same time. Since the largest .jpg file I have in
the folder is just around 5.5Mb, 400Mb sounds strange for me. Googling
on memory consumption issues, I found that there might be some memory
fragmentation problem happening [1].

P.S. Were there any attempts to implement garbage collection for emacs
in C code? I found an article [2] showing that using an actual GC may speed
up an application in comparison with malloc/free approach. 


[1] https://stackoverflow.com/a/9069474/9196985
[2] https://www.linuxjournal.com/article/6679

Regards,
Ihor


[-- Attachment #2: images-seq.png --]
[-- Type: image/png, Size: 22275 bytes --]

[-- Attachment #3: Type: text/plain, Size: 626 bytes --]



Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Date: Sat, 23 Nov 2019 22:37:44 +0800
>> 
>> If I open a large (>250) photos using the code below, the memory usage
>> increases permanently even after I close all the image buffers and wait
>> for some time. 
>
> For some background on this, see
>
>   https://unix.stackexchange.com/questions/53447/does-free-unmap-the-memory-of-a-process
>   https://stackoverflow.com/questions/1421491/does-calling-free-or-delete-ever-release-memory-back-to-the-system
>
> I think what you see is just normal behavior of memory allocation in
> glibc.