bug#38345: 27.0.50; Permanent increase in memory consumption after opening images (or pdfs)

[Ihor Radchenko <yantar92@gmail.com>]

> Feel free to ask questions about the details you don't understand.

I tried to figure out why the memory consumption is increasing with less
frequent cache clearing.
As I understand, there may be two general reasons of this:
1. Increasing of memory used by the image cache data structure
2. Imperfect implementation of xfree or img->type->free, which are
called by free_image

xfree and img->type->free seem to rely on system libraries and are
unlikely to have such a noticeable memory leaks. 

For the image cache data structure, I can only see one place where it
may request extra memory allocation. It is when the number of images in
the cache exceeds the size of c->images array (= c->size, which is 50 by
default). I observed memory consumption increase even when the frequency
of cache clearing in my test was <50, which makes it unclear for me
where the extra memory consumption is coming from.

> I don't think replacing the system malloc on GNU/Linux systems (let
> alone on others) is an idea we'd like to pursue.  You may have more
> luck playing with mallopt calls in init_alloc_once_for_pdumper
> (assuming your build defines DOUG_LEA_MALLOC).

Do I understand correctly that you refer to emacs compiled with
alternative Doug Lea's implementation of malloc?

Actually, I tried to find a way to compile emacs with alternative
variants of malloc, but I did not find how to do it. 

P.S. I am running emacs with jemalloc for a few days and the overall
impression is that emacs became a lot more responsive. Previously, I
had a slow overtime degradation of delay between commands as emacs
process ran for a long time, up to the point when I can type a whole
sentence until emacs finally displays it on screen. Now, I do not see so
much degradation. 

Best,
Ihor


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: juri@linkov.net, 38345@debbugs.gnu.org
>> Date: Mon, 02 Dec 2019 16:04:46 +0800
>> 
>> > The doc string assumes "normal" operation, it doesn't assume some Lisp
>> > command runs for a long time, thus preventing redisplay.
>> 
>> Sounds counter-intuitive. Elisp commands can run for a long time
>> (pointing at org-agenda). 
>
> Commands that run for a long time make Emacs unresponsive, and
> therefore are a misfeature or a bug.
>
> That said, a command that runs a long time is unlikely to create many
> images, destroying each one before creating the next one, because
> what would be the point of that?
>
> IOW, yours is a scenario that is infrequent at best, IMO.
>
>> I tried to look at the code, but I have to admit that I don't understand
>> the reason. Probably because of my poor C. 
>
> Feel free to ask questions about the details you don't understand.
>
>> In any case, I tried different approach trying to reduce memory
>> fragmentation. Instead of using the default malloc/free, I used jemalloc
>> through LD_PRELOAD. For periodic cache clearing (every 2,10,50, and 100
>> opened/killed images), there was no noticeable difference in the memory
>> consumption. However, when I tried to open a bunch of images all
>> together, killed them, and cleared the cache (as in my initial bug
>> report), the jemalloc instance of emacs showed much lower memory
>> consumption (~240Mb vs 1200Mb). 
>
> I don't think replacing the system malloc on GNU/Linux systems (let
> alone on others) is an idea we'd like to pursue.  You may have more
> luck playing with mallopt calls in init_alloc_once_for_pdumper
> (assuming your build defines DOUG_LEA_MALLOC).