Memory usage report

Memory usage report

[Lars Ingebrigtsen]

A reoccurring question people have is "where is Emacs using memory now,
then?"

This can't be reported accurately, really, but...  does Emacs already
have a command that tries to visualise this for users?  There's

(garbage-collect)
=>
((conses 16 6833300 570188)
 (symbols 48 43826 42)
 (strings 32 599007 230739)
 (string-bytes 1 57524636)
 (vectors 16 77428)
 (vector-slots 8 2510954 524038)
 (floats 8 6607 6143)
 (intervals 56 1505503 18066)
 (buffers 992 207))

but that leaves a lot out, and isn't very user friendly.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 17 Sep 2020 21:09:22 +0200
> 
> A reoccurring question people have is "where is Emacs using memory now,
> then?"
> 
> This can't be reported accurately, really, but...  does Emacs already
> have a command that tries to visualise this for users?  There's
> 
> (garbage-collect)

This doesn't include memory we allocate for stuff that isn't Lisp
objects.

Re: Memory usage report

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> This can't be reported accurately, really, but...  does Emacs already
>> have a command that tries to visualise this for users?  There's
>> 
>> (garbage-collect)
>
> This doesn't include memory we allocate for stuff that isn't Lisp
> objects.

Indeed.  A useful memory usage report would need more data, but it's a
starting point...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Stefan Monnier]

> This can't be reported accurately, really, but...  does Emacs already
> have a command that tries to visualise this for users?  There's
>
> (garbage-collect)
> =>
> ((conses 16 6833300 570188)
>  (symbols 48 43826 42)
>  (strings 32 599007 230739)
>  (string-bytes 1 57524636)
>  (vectors 16 77428)
>  (vector-slots 8 2510954 524038)
>  (floats 8 6607 6143)
>  (intervals 56 1505503 18066)
>  (buffers 992 207))
>
> but that leaves a lot out,

Yes, that's a very serious shortcoming.

>  and isn't very user friendly.

GNU ELPA has `memory-usage` which makes this more user-friendly (and
adds a bit more information, tho far from enough in many cases).


        Stefan

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 17 Sep 2020 21:28:07 +0200
> 
> >> (garbage-collect)
> >
> > This doesn't include memory we allocate for stuff that isn't Lisp
> > objects.
> 
> Indeed.  A useful memory usage report would need more data, but it's a
> starting point...

Not that I'd object to formatting this in better ways (especially
since a package to do that is already on ELPA), but from experience,
memory related problems are very rarely in GC-related areas.  So for
diagnosing memory leaks, the GC report is not very useful.  See, for
example, the currently on-going discussion on the bug list, where
someone with a 5GB RSS of the Emacs process posted the output of
garbage-collect, which shows fewer objects and less memory taken by
them than in my 13-day old Emacs session whose total VM size is just
0.5GB.

Re: Memory usage report

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Not that I'd object to formatting this in better ways (especially
> since a package to do that is already on ELPA), but from experience,
> memory related problems are very rarely in GC-related areas.  So for
> diagnosing memory leaks, the GC report is not very useful.

Even in the presence of a C-level genuine leak, it's useful -- because
it shows you that the memory is indeed not taken by Lisp-level objects.

But for this command to be useful in general, I think we'll have to
expose more data from the C layer.  What caches and stuff do we have on
the C layer that can take a significant amount of memory?  The image
and font caches?  Uhm...  Anything more?

I'd also like the display to list, say, the ten "largest variables".
This will be inaccurate, of course, since structures may share members,
but could perhaps be useful.  And I'd also like to do the same with
buffer-local variables, in case a lot of data is hiding there (for
instance, the eww-history, which caches old rendered versions of web
pages, and may be large).

The memory-usage package does some of this, I see, but not a lot...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 18 Sep 2020 12:51:52 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Not that I'd object to formatting this in better ways (especially
> > since a package to do that is already on ELPA), but from experience,
> > memory related problems are very rarely in GC-related areas.  So for
> > diagnosing memory leaks, the GC report is not very useful.
> 
> Even in the presence of a C-level genuine leak, it's useful -- because
> it shows you that the memory is indeed not taken by Lisp-level objects.

Yes, but that's a negative evidence, so it doesn't give any hints
regarding where to look for the problem.

> But for this command to be useful in general, I think we'll have to
> expose more data from the C layer.  What caches and stuff do we have on
> the C layer that can take a significant amount of memory?  The image
> and font caches?  Uhm...  Anything more?

We have a legion of them.

The problem with reporting that memory is that we'd need to monitor
calls that free memory as well, and "forget" the chunks that have been
freed.  At which point we will probably realize that there are
memory-debugging libraries out there, and it's probably easier to
build Emacs with one of them instead of rolling out our own.

> I'd also like the display to list, say, the ten "largest variables".

Which variables did you have in mind in this context?  Can you show an
example?

> And I'd also like to do the same with buffer-local variables, in
> case a lot of data is hiding there (for instance, the eww-history,
> which caches old rendered versions of web pages, and may be large).

But those are part of the GC report, aren't they?

Re: Memory usage report

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes, but that's a negative evidence, so it doesn't give any hints
> regarding where to look for the problem.

It's negative evidence, but that's useful, too, sometimes.

>> But for this command to be useful in general, I think we'll have to
>> expose more data from the C layer.  What caches and stuff do we have on
>> the C layer that can take a significant amount of memory?  The image
>> and font caches?  Uhm...  Anything more?
>
> We have a legion of them.

That can have significant memory?

> The problem with reporting that memory is that we'd need to monitor
> calls that free memory as well, and "forget" the chunks that have been
> freed.  At which point we will probably realize that there are
> memory-debugging libraries out there, and it's probably easier to
> build Emacs with one of them instead of rolling out our own.

I don't think we have to go that far to be useful.  Reporting that the
image cache takes foo GB will help somebody.

>> I'd also like the display to list, say, the ten "largest variables".
>
> Which variables did you have in mind in this context?  Can you show an
> example?

It'd just traverse all the variables and compute the "largest" ones.

>> And I'd also like to do the same with buffer-local variables, in
>> case a lot of data is hiding there (for instance, the eww-history,
>> which caches old rendered versions of web pages, and may be large).
>
> But those are part of the GC report, aren't they?

Yes.  It's a more detailed look at what's holding on to the memory.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 18 Sep 2020 13:47:48 +0200
> 
> >> But for this command to be useful in general, I think we'll have to
> >> expose more data from the C layer.  What caches and stuff do we have on
> >> the C layer that can take a significant amount of memory?  The image
> >> and font caches?  Uhm...  Anything more?
> >
> > We have a legion of them.
> 
> That can have significant memory?

Some of them could, yes.

> I don't think we have to go that far to be useful.  Reporting that the
> image cache takes foo GB will help somebody.

Why would an image cache take that much? we flush it from time to
time, and I have difficulty believing that all those users reporting
large memory footprints load thousands of images every hour of every
day.

> > Which variables did you have in mind in this context?  Can you show an
> > example?
> 
> It'd just traverse all the variables and compute the "largest" ones.

Again, they are included in the GC report.  So what do you expect to
see?

> >> And I'd also like to do the same with buffer-local variables, in
> >> case a lot of data is hiding there (for instance, the eww-history,
> >> which caches old rendered versions of web pages, and may be large).
> >
> > But those are part of the GC report, aren't they?
> 
> Yes.  It's a more detailed look at what's holding on to the memory.

But in the cases where people see the memory footprint increasing, the
GC report doesn't show any increase in memory use, and the sum total
of what it reports is way smaller than several GB.  So, while a more
detailed report might be nice to have, I don't see how it would help
to diagnose "leaks" of the kind we are discussing now on the bug list.

Re: Memory usage report

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> That can have significant memory?
>
> Some of them could, yes.

Do you know which ones?

>> I don't think we have to go that far to be useful.  Reporting that the
>> image cache takes foo GB will help somebody.
>
> Why would an image cache take that much? we flush it from time to
> time, and I have difficulty believing that all those users reporting
> large memory footprints load thousands of images every hour of every
> day.

It is unlikely that "all those users" use that many images, but some
may.  (As you no doubt remember, calling `image-size' in a loop will
kill Emacs with an out of memory fault.)

>> > Which variables did you have in mind in this context?  Can you show an
>> > example?
>> 
>> It'd just traverse all the variables and compute the "largest" ones.
>
> Again, they are included in the GC report.  So what do you expect to
> see?

I expect to see what variables take "all that size"?  I'm not sure what
you're asking here.

> So, while a more detailed report might be nice to have, I don't see
> how it would help to diagnose "leaks" of the kind we are discussing
> now on the bug list.

I am not talking about any bug report in particular.  If I were, that is
where I would have posted this.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 18 Sep 2020 14:59:24 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> That can have significant memory?
> >
> > Some of them could, yes.
> 
> Do you know which ones?

bidi_cache, for one.

> > Why would an image cache take that much? we flush it from time to
> > time, and I have difficulty believing that all those users reporting
> > large memory footprints load thousands of images every hour of every
> > day.
> 
> It is unlikely that "all those users" use that many images, but some
> may.  (As you no doubt remember, calling `image-size' in a loop will
> kill Emacs with an out of memory fault.)

That loop is highly unlikely to happen in practice, certainly not
happen constantly in a long-lived session.

> >> > Which variables did you have in mind in this context?  Can you show an
> >> > example?
> >> 
> >> It'd just traverse all the variables and compute the "largest" ones.
> >
> > Again, they are included in the GC report.  So what do you expect to
> > see?
> 
> I expect to see what variables take "all that size"?  I'm not sure what
> you're asking here.

I guess I don't understand the utility of knowing, e.g., which Lisp
string in the current session is the longest one.  What would you do
with such information?

> > So, while a more detailed report might be nice to have, I don't see
> > how it would help to diagnose "leaks" of the kind we are discussing
> > now on the bug list.
> 
> I am not talking about any bug report in particular.  If I were, that is
> where I would have posted this.

Then I guess I don't understand the purpose of the features you'd like
to have.  Can you explain?

Re: Memory usage report

[Ihor Radchenko]

> I guess I don't understand the utility of knowing, e.g., which Lisp
> string in the current session is the longest one.  What would you do
> with such information?

Knowing a buffer with largest total memory occupied by buffer-local
variables might be useful.


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 18 Sep 2020 14:59:24 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> That can have significant memory?
>> >
>> > Some of them could, yes.
>> 
>> Do you know which ones?
>
> bidi_cache, for one.
>
>> > Why would an image cache take that much? we flush it from time to
>> > time, and I have difficulty believing that all those users reporting
>> > large memory footprints load thousands of images every hour of every
>> > day.
>> 
>> It is unlikely that "all those users" use that many images, but some
>> may.  (As you no doubt remember, calling `image-size' in a loop will
>> kill Emacs with an out of memory fault.)
>
> That loop is highly unlikely to happen in practice, certainly not
> happen constantly in a long-lived session.
>
>> >> > Which variables did you have in mind in this context?  Can you show an
>> >> > example?
>> >> 
>> >> It'd just traverse all the variables and compute the "largest" ones.
>> >
>> > Again, they are included in the GC report.  So what do you expect to
>> > see?
>> 
>> I expect to see what variables take "all that size"?  I'm not sure what
>> you're asking here.
>
> I guess I don't understand the utility of knowing, e.g., which Lisp
> string in the current session is the longest one.  What would you do
> with such information?
>
>> > So, while a more detailed report might be nice to have, I don't see
>> > how it would help to diagnose "leaks" of the kind we are discussing
>> > now on the bug list.
>> 
>> I am not talking about any bug report in particular.  If I were, that is
>> where I would have posted this.
>
> Then I guess I don't understand the purpose of the features you'd like
> to have.  Can you explain?

Re: Memory usage report

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Date: Fri, 18 Sep 2020 21:32:04 +0800
> Cc: emacs-devel@gnu.org
> 
> > I guess I don't understand the utility of knowing, e.g., which Lisp
> > string in the current session is the longest one.  What would you do
> > with such information?
> 
> Knowing a buffer with largest total memory occupied by buffer-local
> variables might be useful.

Well, Lars basically said the same, but neither of you explain _why_
would something like that be useful.  Do tell, please.

Re: Memory usage report

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Well, Lars basically said the same, but neither of you explain _why_
> would something like that be useful.  Do tell, please.

If you see that you've got 13GBs worth of cons cells, you might wonder
what variable that's holding on to them.  The list of "largest
variables", with `some-stupid-mode-I'd-forgotten-about-cache' on top
will give you a hint.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Ihor Radchenko]

> Well, Lars basically said the same, but neither of you explain _why_
> would something like that be useful.  Do tell, please.

I have noticed that loading my org files increases my emacs memory from
3% (out of 8Gb) to around 7%. Later, the memory grows to around 13-20%
(with most of memory reported by garbage-collect). I have no idea what
causes it, but I suspect org eating the memory. Seeing how much memory
is used by buffer-local variables in org buffers would help. Similarly,
total memory consumption in variables from individual packages would be
helpful to pinpoint why the memory usage is high.

Best,
Ihor


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Date: Fri, 18 Sep 2020 21:32:04 +0800
>> Cc: emacs-devel@gnu.org
>> 
>> > I guess I don't understand the utility of knowing, e.g., which Lisp
>> > string in the current session is the longest one.  What would you do
>> > with such information?
>> 
>> Knowing a buffer with largest total memory occupied by buffer-local
>> variables might be useful.
>
> Well, Lars basically said the same, but neither of you explain _why_
> would something like that be useful.  Do tell, please.

Re: Memory usage report

[Stefan Monnier]

>> Well, Lars basically said the same, but neither of you explain _why_
>> would something like that be useful.  Do tell, please.
>
> If you see that you've got 13GBs worth of cons cells, you might wonder
> what variable that's holding on to them.  The list of "largest
> variables", with `some-stupid-mode-I'd-forgotten-about-cache' on top
> will give you a hint.

Eli's question boils down to: can you point to a concrete case where
this would have been useful?

I have done such digging once or twice when the Lisp heap was taking
more space than I expected, but it's quite rare IME (and apparently in
Eli's as well).

So, while it might be nice to add it to `memory-usage` it's unlikely to
help in the more common cases of excessive memory use where the memory
is eaten up by objects allocated in the C part of the heap.


        Stefan

Re: Memory usage report

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Eli's question boils down to: can you point to a concrete case where
> this would have been useful?

Yes.  :-)  

https://www.reddit.com/r/emacs/comments/iuott9/emacs_memory_leak_debugging/

Relevant quote:

> Over in #guix irc, the guix people seemed to think it was a memory leak with
> helm.
>
> I was watching my emacs consume about 0.1% more system memory every 2 or 3
> seconds. Setting
>
> (setq helm-ff-keep-cached-candidates nil)
>
> Seemed to make the problem go away.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Ihor Radchenko <yantar92@gmail.com>,  emacs-devel@gnu.org
> Date: Fri, 18 Sep 2020 16:08:22 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Well, Lars basically said the same, but neither of you explain _why_
> > would something like that be useful.  Do tell, please.
> 
> If you see that you've got 13GBs worth of cons cells, you might wonder
> what variable that's holding on to them.

In that case, yes.  But I don't think we've ever seen such a
situation.

Re: Memory usage report

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Fri, 18 Sep 2020 22:30:11 +0800
> 
> Seeing how much memory is used by buffer-local variables in org
> buffers would help. Similarly, total memory consumption in variables
> from individual packages would be helpful to pinpoint why the memory
> usage is high.

That'd require some filtering of the variables, not just finding the
largest one.

Did you actually see cases where something like that happens?  I mean
that loading some library, or all of your Org files, causes increase
in memory usage of hundreds of MBs (320MB in your example)?  I only
see this when the buffers themselves are near that value.

Re: Memory usage report

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Seeing how much memory is used by buffer-local variables in org
>> buffers would help. Similarly, total memory consumption in variables
>> from individual packages would be helpful to pinpoint why the memory
>> usage is high.
>
> That'd require some filtering of the variables, not just finding the
> largest one.

I don't think anybody has said anything about just finding the largest
variable...

For a buffer display, the natural thing would be to list the size of the
buffer itself, and then a sum of the size of all the buffer-local
variables.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Ihor Radchenko <yantar92@gmail.com>,
>   emacs-devel@gnu.org
> Date: Fri, 18 Sep 2020 17:19:00 +0200
> 
> https://www.reddit.com/r/emacs/comments/iuott9/emacs_memory_leak_debugging/
> 
> Relevant quote:
> 
> > Over in #guix irc, the guix people seemed to think it was a memory leak with
> > helm.
> >
> > I was watching my emacs consume about 0.1% more system memory every 2 or 3
> > seconds. Setting
> >
> > (setq helm-ff-keep-cached-candidates nil)
> >
> > Seemed to make the problem go away.

That helm cache is stored in a hash-table, AFAIU, in which case we
should have seen that in the GC report.  But in all of the cases where
people complained about such huge RSS, the GC report shows a very
small Lisp memory usage, so that helm cache cannot be the culprit.
It's something else, whether related to helm-ff-keep-cached-candidates
or some other feature.

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Ihor Radchenko <yantar92@gmail.com>,  emacs-devel@gnu.org
> Date: Fri, 18 Sep 2020 17:37:32 +0200
> 
> I don't think anybody has said anything about just finding the largest
> variable...

You did, I think.  Or at least I interpreted what you said that way.

> For a buffer display, the natural thing would be to list the size of the
> buffer itself, and then a sum of the size of all the buffer-local
> variables.

What about text properties and overlays? what about the buffer's
markers? and what about the caches (like the newline_cache)?

Re: Memory usage report

[Ihor Radchenko]

> Did you actually see cases where something like that happens?  I mean
> that loading some library, or all of your Org files, causes increase
> in memory usage of hundreds of MBs (320MB in your example)?  I only
> see this when the buffers themselves are near that value.

Yep. I see it every time I run Emacs. I have done M-x memory-usage
before and after loading org-agenda (which loads all my working org
files).

Before:
Total in lisp objects: 77.4MB (live 47.7MB, dead 29.7MB)
Buffer ralloc memory usage:
16 buffers
39.3kB total (23.6kB in gaps)

After:
Total in lisp objects:  283MB (live  218MB, dead 64.7MB)
Buffer ralloc memory usage:
47 buffers
6.79MB total (78.6kB in gaps)

This is not all the memory increase. Some part seems to be in heap.

Also, the memory usage is even worse when org-mode is using overlays (I
am on org-mode branch using text properties instead of overlays in org).

Full memory-usage reports are shown below.

The memory consumption gets even worse after I use Emacs for several
hours. Recently, it tend to settle around 20% (of 8Gb) with lisp objects
taking around 1Gb.

This last part is the most annoying and also difficult to track - I have
no easy way to know if it is also caused by org-mode or it is something
else.

Best,
Ihor

Before:

Garbage collection stats:
((conses 16 934007 1096370) (symbols 48 73755 2041) (strings 32 265462 116861) (string-bytes 1 9693220) (vectors 16 115429) (vector-slots 8 1423224 1215629) (floats 8 1419 3034) (intervals 56 1366 249) (buffers 992 16))

 =>	14.3MB (+ 16.7MB dead) in conses
	3.38MB (+ 95.7kB dead) in symbols
	8.10MB (+ 3.57MB dead) in strings
	9.24MB in string-bytes
	1.76MB in vectors
	10.9MB (+ 9.27MB dead) in vector-slots
	11.1kB (+ 23.7kB dead) in floats
	74.7kB (+ 13.6kB dead) in intervals
	15.5kB in buffers

Total in lisp objects: 77.4MB (live 47.7MB, dead 29.7MB)

Buffer ralloc memory usage:
16 buffers
39.3kB total (23.6kB in gaps)
      Size	Gap	Name

     13213	2000	*scratch*
      1252	769	*Messages*
       676	1574	*Buffer Details*
       294	1940	 *straight-watcher*
       265	1756	*straight-process*
       124	1911	 *code-conversion-work*
       112	3941	*helm M-x*
        82	2000	*elfeed-log*
        67	2000	 *pdf-info-query--parse-response*
        53	2000	 *Echo Area 1*
        24	2001	 *Echo Area 0*
         8	12	 *pdf-info-query--escape*
         0	20	 *Minibuf-1*
         0	20	 *Minibuf-0*
         0	20	 *epdfinfo*
         0	2067	 tq-temp-epdfinfo

After:

Garbage collection stats:
((conses 16 9393579 2919763) (symbols 48 75965 293) (strings 32 323708 224589) (string-bytes 1 13017637) (vectors 16 154787) (vector-slots 8 2249055 1365412) (floats 8 1630 2854) (intervals 56 547977 53455) (buffers 992 47))

 =>	 143MB (+ 44.6MB dead) in conses
	3.48MB (+ 13.7kB dead) in symbols
	9.88MB (+ 6.85MB dead) in strings
	12.4MB in string-bytes
	2.36MB in vectors
	17.2MB (+ 10.4MB dead) in vector-slots
	12.7kB (+ 22.3kB dead) in floats
	29.3MB (+ 2.85MB dead) in intervals
	45.5kB in buffers

Total in lisp objects:  283MB (live  218MB, dead 64.7MB)

Buffer ralloc memory usage:
47 buffers
6.79MB total (78.6kB in gaps)
      Size	Gap	Name

   2345866	2000	nodeadline.org
   1914956	2000	notes.org
    695164	2000	TODO.org
    548214	2000	config.org
    487501	2000	get_started_org_mode.org
    361666	2000	CuNb-beamcrack.org
    340586	2000	References.org
     84150	2000	articles.org
     47507	2000	Journal.org
     45875	2000	Олди Генри Лайон.org
     31561	2000	mirror-text.org
     30235	2000	rss.org
     29662	2000	org-autosort.org
     18633	2000	[book]DFR.org
     14412	801	*scratch*
      9932	2000	schedule.org
      8719	2000	Abraham Daniel.org
      7292	2000	org-mode.org
      6172	2000	inbox.org
      4503	2000	contacts.org
      2564	2000	 *code-conversion-work*
      1944	2000	gnuplot.org
      1838	183	*Messages*
      1125	950	*Memory-Profiler-Report 2020-09-18 23:59:40*
       897	1131	 *emacsql-log*
       889	1727	 *Agenda Commands*
       681	1569	*Buffer Details*
       301	1720	*Org Agenda(d:)*
       294	1940	 *straight-watcher*
       265	1756	*straight-process*
       125	3928	*helm M-x*
       119	2000	frommobile.org
        82	2000	*elfeed-log*
        67	2000	 *pdf-info-query--parse-response*
        50	1973	*Helm Completions*
        41	1988	*helm-mode-profiler-report-write-profile*
        35	1990	 *Echo Area 0*
        31	1991	*helm-mode-profiler-start*
        13	2474	 *emacsql-sqlite*
         8	12	 *pdf-info-query--escape*
         6	14	 *temp*
         0	2030	 *Minibuf-1*
         0	20	 *Minibuf-0*
         0	2053	 *Echo Area 1*
         0	20	 *epdfinfo*
         0	2067	 tq-temp-epdfinfo
         0	20	 *server*



Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> Date: Fri, 18 Sep 2020 22:30:11 +0800
>> 
>> Seeing how much memory is used by buffer-local variables in org
>> buffers would help. Similarly, total memory consumption in variables
>> from individual packages would be helpful to pinpoint why the memory
>> usage is high.
>
> That'd require some filtering of the variables, not just finding the
> largest one.
>
> Did you actually see cases where something like that happens?  I mean
> that loading some library, or all of your Org files, causes increase
> in memory usage of hundreds of MBs (320MB in your example)?  I only
> see this when the buffers themselves are near that value.

Re: Memory usage report

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Sat, 19 Sep 2020 00:14:34 +0800
> 
> Before:
> Total in lisp objects: 77.4MB (live 47.7MB, dead 29.7MB)
> Buffer ralloc memory usage:
> 16 buffers
> 39.3kB total (23.6kB in gaps)
> 
> After:
> Total in lisp objects:  283MB (live  218MB, dead 64.7MB)
> Buffer ralloc memory usage:
> 47 buffers
> 6.79MB total (78.6kB in gaps)
> 
> This is not all the memory increase. Some part seems to be in heap.

I see almost all of the 200MB of additional memory in Lisp objects:
mainly conses, the rest in vectors and intervals (i.e. text
properties).  Where do you see evidence for a significant memory in
the heap?

I guess you could ask Org developers a question regarding this high
memory usage.  In any case, this is not what people report in
bug#43389, or so it seems.

> Also, the memory usage is even worse when org-mode is using overlays (I
> am on org-mode branch using text properties instead of overlays in org).

Looks like you have 30 Org buffers whose total size is below 10MB?  If
they use a lot of text properties and overlays, it's a small wonder
you get high memory usage.  I don't necessarily see a problem here.

> The memory consumption gets even worse after I use Emacs for several
> hours. Recently, it tend to settle around 20% (of 8Gb) with lisp objects
> taking around 1Gb.

So the interesting question is: where are those 7GB used?

> This last part is the most annoying and also difficult to track - I have
> no easy way to know if it is also caused by org-mode or it is something
> else.

The only way to investigate this is to use a good memory-mapping
utility and perhaps also a debugging malloc library.

And that is actually the bottom line here: the GBytes of memory used
by such large Emacs sessions don't seem to come from Lisp objects,
they are used up in some other way.  The question is how and where in
the code this happens.

Re: Memory usage report

[Ihor Radchenko]

> Where do you see evidence for a significant memory in
> the heap?

top shows more than 200Mb memory increase.

> So the interesting question is: where are those 7GB used?

1Gb in lisp objects and 20%*8Gb=1.6Gb total. I am not sure about those
extra 0.6Gb, but answering about the 1Gb in lisp objects would also
help. That's why I asked earlier about buffer-local variable summary and
summary about library variables.

> And that is actually the bottom line here: the GBytes of memory used
> by such large Emacs sessions don't seem to come from Lisp objects,
> they are used up in some other way.  The question is how and where in
> the code this happens.

While I would also like to answer this question, I also have large
fraction of lisp objects. As I mentioned in previous emails (sorry if I
was not clear), I would appreciate some way to understand why my **lisp
object** memory grew from 283Mb right after loading my org files to 1Gb
later.

P.S.

I tried to use memory profiler to understand the memory usage, but the
memory profiler report appear to show peak memory usage (there tend to
be functions with many let-bindings). So, it seems not be to useful for
me (correct me if I am wrong).

Best,
Ihor


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> Date: Sat, 19 Sep 2020 00:14:34 +0800
>> 
>> Before:
>> Total in lisp objects: 77.4MB (live 47.7MB, dead 29.7MB)
>> Buffer ralloc memory usage:
>> 16 buffers
>> 39.3kB total (23.6kB in gaps)
>> 
>> After:
>> Total in lisp objects:  283MB (live  218MB, dead 64.7MB)
>> Buffer ralloc memory usage:
>> 47 buffers
>> 6.79MB total (78.6kB in gaps)
>> 
>> This is not all the memory increase. Some part seems to be in heap.
>
> I see almost all of the 200MB of additional memory in Lisp objects:
> mainly conses, the rest in vectors and intervals (i.e. text
> properties).  Where do you see evidence for a significant memory in
> the heap?
>
> I guess you could ask Org developers a question regarding this high
> memory usage.  In any case, this is not what people report in
> bug#43389, or so it seems.
>
>> Also, the memory usage is even worse when org-mode is using overlays (I
>> am on org-mode branch using text properties instead of overlays in org).
>
> Looks like you have 30 Org buffers whose total size is below 10MB?  If
> they use a lot of text properties and overlays, it's a small wonder
> you get high memory usage.  I don't necessarily see a problem here.
>
>> The memory consumption gets even worse after I use Emacs for several
>> hours. Recently, it tend to settle around 20% (of 8Gb) with lisp objects
>> taking around 1Gb.
>
> So the interesting question is: where are those 7GB used?
>
>> This last part is the most annoying and also difficult to track - I have
>> no easy way to know if it is also caused by org-mode or it is something
>> else.
>
> The only way to investigate this is to use a good memory-mapping
> utility and perhaps also a debugging malloc library.
>
> And that is actually the bottom line here: the GBytes of memory used
> by such large Emacs sessions don't seem to come from Lisp objects,
> they are used up in some other way.  The question is how and where in
> the code this happens.

Re: Memory usage report

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Sat, 19 Sep 2020 08:29:37 +0800
> 
> > Where do you see evidence for a significant memory in
> > the heap?
> 
> top shows more than 200Mb memory increase.

Which are explained by the statistics produced by GC.  IOW, you have
many more live Lisp objects, which take up those megabytes.

> While I would also like to answer this question, I also have large
> fraction of lisp objects. As I mentioned in previous emails (sorry if I
> was not clear), I would appreciate some way to understand why my **lisp
> object** memory grew from 283Mb right after loading my org files to 1Gb
> later.

Since that's related to Org buffers, the best place to discuss this is
on Org mailing lists.  Perhaps there are ways to make Org use less
memory, but the expertise for that is there.

Producing memory usage for buffer-local variables will AFAIU need
support on the C level, so if someone wants to work on that, please
do.  However, issues like this one are very infrequent in Emacs
maintenance; most of the memory-related problems are not due to some
package using up many Lisp objects.

> I tried to use memory profiler to understand the memory usage, but the
> memory profiler report appear to show peak memory usage (there tend to
> be functions with many let-bindings). So, it seems not be to useful for
> me (correct me if I am wrong).

The "memory" profiler doesn't measure the usage of memory, it measures
CPU usage triggered by memory allocation calls (instead of the
periodic profiling signal).  So this profile is not supposed to be
useful for profiling memory usage.

Re: Memory usage report

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't think anybody has said anything about just finding the largest
>> variable...
>
> You did, I think.  Or at least I interpreted what you said that way.

No, I said the buffer should display a list of the largest variables.

>> For a buffer display, the natural thing would be to list the size of the
>> buffer itself, and then a sum of the size of all the buffer-local
>> variables.
>
> What about text properties and overlays? what about the buffer's
> markers? and what about the caches (like the newline_cache)?

Sure, all that could be included.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> That helm cache is stored in a hash-table, AFAIU, in which case we
> should have seen that in the GC report.

But that wouldn't have helped anybody pinpoint the issue with Helm.

> But in all of the cases where people complained about such huge RSS,
> the GC report shows a very small Lisp memory usage, so that helm cache
> cannot be the culprit.  It's something else, whether related to
> helm-ff-keep-cached-candidates or some other feature.

I just gave you an example of the opposite, and then you say "in all of
the cases"?  I'm not sure what you are arguing here, or even why you are
arguing about this: Giving users a convenient way to examine (and rule
out) certain memory usage problems surely can't be bad in itself.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Producing memory usage for buffer-local variables will AFAIU need
> support on the C level, so if someone wants to work on that, please
> do. 

I thought I'd just visit all the buffers and look at which variables
available from each buffer are buffer-local.  I think all of that is
available from Lisp?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Ihor Radchenko]

>> top shows more than 200Mb memory increase.
>
> Which are explained by the statistics produced by GC.  IOW, you have
> many more live Lisp objects, which take up those megabytes.

I meant that 200Mb is Lisp objects, but top shows more then 200Mb
(around 300Mb). So, there is extra 100Mb coming from somewhere else.

> Since that's related to Org buffers, the best place to discuss this is
> on Org mailing lists.  Perhaps there are ways to make Org use less
> memory, but the expertise for that is there.

The problem is how to identify where the memory usage is coming from.
Indeed, org is using overlays and text properties extensively. But how
much do those influence the memory usage? I am somewhat familiar with
org-mode codebase, but I have no idea how to debug memory usage issues
on practice. Total memory usage report provides little clue about what
should be changed.

> The "memory" profiler doesn't measure the usage of memory, it measures
> CPU usage triggered by memory allocation calls (instead of the
> periodic profiling signal).  So this profile is not supposed to be
> useful for profiling memory usage.

Thanks! I did not know that. Would it be possible to have a "real"
memory profiler showing how much the memory usage changed after-before
running separate functions? This could help debugging memory usage (i.e.
in org-mode).

Best,
Ihor

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> Date: Sat, 19 Sep 2020 08:29:37 +0800
>> 
>> > Where do you see evidence for a significant memory in
>> > the heap?
>> 
>> top shows more than 200Mb memory increase.
>
> Which are explained by the statistics produced by GC.  IOW, you have
> many more live Lisp objects, which take up those megabytes.
>
>> While I would also like to answer this question, I also have large
>> fraction of lisp objects. As I mentioned in previous emails (sorry if I
>> was not clear), I would appreciate some way to understand why my **lisp
>> object** memory grew from 283Mb right after loading my org files to 1Gb
>> later.
>
> Since that's related to Org buffers, the best place to discuss this is
> on Org mailing lists.  Perhaps there are ways to make Org use less
> memory, but the expertise for that is there.
>
> Producing memory usage for buffer-local variables will AFAIU need
> support on the C level, so if someone wants to work on that, please
> do.  However, issues like this one are very infrequent in Emacs
> maintenance; most of the memory-related problems are not due to some
> package using up many Lisp objects.
>
>> I tried to use memory profiler to understand the memory usage, but the
>> memory profiler report appear to show peak memory usage (there tend to
>> be functions with many let-bindings). So, it seems not be to useful for
>> me (correct me if I am wrong).
>
> The "memory" profiler doesn't measure the usage of memory, it measures
> CPU usage triggered by memory allocation calls (instead of the
> periodic profiling signal).  So this profile is not supposed to be
> useful for profiling memory usage.

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Ihor Radchenko <yantar92@gmail.com>,  emacs-devel@gnu.org
> Date: Sat, 19 Sep 2020 16:29:34 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Producing memory usage for buffer-local variables will AFAIU need
> > support on the C level, so if someone wants to work on that, please
> > do. 
> 
> I thought I'd just visit all the buffers and look at which variables
> available from each buffer are buffer-local.  I think all of that is
> available from Lisp?

You mean, use mapatoms or somesuch?

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 19 Sep 2020 16:27:20 +0200
> Cc: emacs-devel@gnu.org, monnier@iro.umontreal.ca, yantar92@gmail.com
> 
> > But in all of the cases where people complained about such huge RSS,
> > the GC report shows a very small Lisp memory usage, so that helm cache
> > cannot be the culprit.  It's something else, whether related to
> > helm-ff-keep-cached-candidates or some other feature.
> 
> I just gave you an example of the opposite, and then you say "in all of
> the cases"?  I'm not sure what you are arguing here, or even why you are
> arguing about this: Giving users a convenient way to examine (and rule
> out) certain memory usage problems surely can't be bad in itself.

We are talking about two different use cases, I think: I was talking
about the problems described in bug#43389.

I have no objections to producing more detailed memory usage reports
based on Lisp data.  I'm just saying that I usually find that not very
helpful for investigating problems like the bug mentioned above.

I'm sorry I annoyed you.

Re: Memory usage report

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Sat, 19 Sep 2020 22:34:28 +0800
> 
> >> top shows more than 200Mb memory increase.
> >
> > Which are explained by the statistics produced by GC.  IOW, you have
> > many more live Lisp objects, which take up those megabytes.
> 
> I meant that 200Mb is Lisp objects, but top shows more then 200Mb
> (around 300Mb). So, there is extra 100Mb coming from somewhere else.

I got a different (smaller) value, but it isn't worth to continue this
argument.

> > Since that's related to Org buffers, the best place to discuss this is
> > on Org mailing lists.  Perhaps there are ways to make Org use less
> > memory, but the expertise for that is there.
> 
> The problem is how to identify where the memory usage is coming from.
> Indeed, org is using overlays and text properties extensively. But how
> much do those influence the memory usage?

I think the more useful question is: are all those overlays and text
properties necessary?  If they are, they take the memory they are
supposed to take.

> > The "memory" profiler doesn't measure the usage of memory, it measures
> > CPU usage triggered by memory allocation calls (instead of the
> > periodic profiling signal).  So this profile is not supposed to be
> > useful for profiling memory usage.
> 
> Thanks! I did not know that. Would it be possible to have a "real"
> memory profiler showing how much the memory usage changed after-before
> running separate functions?

If you mean memory used by Lisp objects, then GC reports that.

If you mean memory used by non-Lisp objects, I don't see how we could
produce that without having infrastructure for tracking memory
allocation, something that debugging malloc libraries already do.

Re: Memory usage report

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> You mean, use mapatoms or somesuch?

Yup.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Memory usage report

[Ihor Radchenko]

> I think the more useful question is: are all those overlays and text
> properties necessary?  If they are, they take the memory they are
> supposed to take.

Some of the overlays and text properties are not strictly necessary.
For example, text properties are sometimes used as cache to avoid
parsing text multiple times. Is the resulting speedup worth extra memory
usage? It is not clear since we do not have an easy way to determine the
extra memory usage. 

>> Thanks! I did not know that. Would it be possible to have a "real"
>> memory profiler showing how much the memory usage changed after-before
>> running separate functions?

> If you mean memory used by Lisp objects, then GC reports that.
>
> If you mean memory used by non-Lisp objects, I don't see how we could
> produce that without having infrastructure for tracking memory
> allocation, something that debugging malloc libraries already do.

Hmm. I am looking again at the profiler-report output. It seems to
report the memory allocation for individual function calls (that's what
I meant by "real" memory profiler). Do I miss something?

Function                                                  Bytes        %

- command-execute                                         516,227,475  77%
 - call-interactively                                     516,222,195  77%
  - funcall-interactively                                 516,222,195  77%
   - helm-M-x                                             203,107,508  30%
    - helm-M-x-read-extended-command                      203,107,508  30%
     - helm                                               202,313,652  30%
      - apply                                             202,310,580  30%
       - helm                                             202,310,580  30%
        - apply                                           202,310,580  30%
         - helm-internal                                  202,310,580  30%
          - helm-display-buffer                           112,541,910  16%
           - helm-default-display-buffer                  112,540,854  16%
            - display-buffer                              112,540,854  16%
             - display-buffer--maybe-pop-up-frame-or-window   112,540,854  16%
              - display-buffer--maybe-pop-up-window       112,540,854  16%
               - display-buffer-pop-up-window             112,540,854  16%
                + window--try-to-split-window              56,401,971   8%
                  window--display-buffer                   56,130,435   8%
                  window--maybe-raise-frame                     5,280   0%

Best,
Ihor


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> Date: Sat, 19 Sep 2020 22:34:28 +0800
>> 
>> >> top shows more than 200Mb memory increase.
>> >
>> > Which are explained by the statistics produced by GC.  IOW, you have
>> > many more live Lisp objects, which take up those megabytes.
>> 
>> I meant that 200Mb is Lisp objects, but top shows more then 200Mb
>> (around 300Mb). So, there is extra 100Mb coming from somewhere else.
>
> I got a different (smaller) value, but it isn't worth to continue this
> argument.
>
>> > Since that's related to Org buffers, the best place to discuss this is
>> > on Org mailing lists.  Perhaps there are ways to make Org use less
>> > memory, but the expertise for that is there.
>> 
>> The problem is how to identify where the memory usage is coming from.
>> Indeed, org is using overlays and text properties extensively. But how
>> much do those influence the memory usage?
>
> I think the more useful question is: are all those overlays and text
> properties necessary?  If they are, they take the memory they are
> supposed to take.
>
>> > The "memory" profiler doesn't measure the usage of memory, it measures
>> > CPU usage triggered by memory allocation calls (instead of the
>> > periodic profiling signal).  So this profile is not supposed to be
>> > useful for profiling memory usage.
>> 
>> Thanks! I did not know that. Would it be possible to have a "real"
>> memory profiler showing how much the memory usage changed after-before
>> running separate functions?
>
> If you mean memory used by Lisp objects, then GC reports that.
>
> If you mean memory used by non-Lisp objects, I don't see how we could
> produce that without having infrastructure for tracking memory
> allocation, something that debugging malloc libraries already do.

Re: Memory usage report

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: yantar92@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 19 Sep 2020 16:54:11 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > You mean, use mapatoms or somesuch?
> 
> Yup.

Wouldn't that miss stuff that can be of interest?  For example, what
about variables temporarily bound to large values?

More generally, what about stuff that is not in obarray, like buffer's
markers?

Re: Memory usage report

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Sat, 19 Sep 2020 23:14:52 +0800
> 
> > I think the more useful question is: are all those overlays and text
> > properties necessary?  If they are, they take the memory they are
> > supposed to take.
> 
> Some of the overlays and text properties are not strictly necessary.
> For example, text properties are sometimes used as cache to avoid
> parsing text multiple times. Is the resulting speedup worth extra memory
> usage? It is not clear since we do not have an easy way to determine the
> extra memory usage. 

These are exactly the questions to ask the Org developers, I think.

> > If you mean memory used by non-Lisp objects, I don't see how we could
> > produce that without having infrastructure for tracking memory
> > allocation, something that debugging malloc libraries already do.
> 
> Hmm. I am looking again at the profiler-report output. It seems to
> report the memory allocation for individual function calls (that's what
> I meant by "real" memory profiler). Do I miss something?

What that profiler counts is calls to memory-allocation functions,
that's all.  Without being able to account for memory which was freed
after being allocated, these counts are useless.

Re: Memory usage report

[Stefan Monnier]

> I thought I'd just visit all the buffers and look at which variables
> available from each buffer are buffer-local.  I think all of that is
> available from Lisp?

You mean via `buffer-local-variables` ;-)


        Stefan

Re: Memory usage report

[Stefan Monnier]

>>> top shows more than 200Mb memory increase.
>> Which are explained by the statistics produced by GC.  IOW, you have
>> many more live Lisp objects, which take up those megabytes.
> I meant that 200Mb is Lisp objects, but top shows more then 200Mb
> (around 300Mb).  So, there is extra 100Mb coming from somewhere else.

Personally, when that extra data is smaller than the Lisp heap, I don't
care to know where it's spent.


        Stefan

Re: Memory usage report

[Ihor Radchenko]

>> Some of the overlays and text properties are not strictly necessary.
>> For example, text properties are sometimes used as cache to avoid
>> parsing text multiple times. Is the resulting speedup worth extra memory
>> usage? It is not clear since we do not have an easy way to determine the
>> extra memory usage. 
>
> These are exactly the questions to ask the Org developers, I think.

As an org developer, how can I know the extra memory usage by that one
specific type of text properties?

> What that profiler counts is calls to memory-allocation functions,
> that's all.  Without being able to account for memory which was freed
> after being allocated, these counts are useless.

Got it. Then, I imagine that a simple delta of before/after running a
function could be used to build a more useful memory report. Current GC
report only provides the total usage, but no information on how
individual function calls increased the memory usage. This might even be
done in Elisp comparing `cons-cells-consed' and similar variables
before/after each function call.

Best,
Ihor

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> Date: Sat, 19 Sep 2020 23:14:52 +0800
>> 
>> > I think the more useful question is: are all those overlays and text
>> > properties necessary?  If they are, they take the memory they are
>> > supposed to take.
>> 
>> Some of the overlays and text properties are not strictly necessary.
>> For example, text properties are sometimes used as cache to avoid
>> parsing text multiple times. Is the resulting speedup worth extra memory
>> usage? It is not clear since we do not have an easy way to determine the
>> extra memory usage. 
>
> These are exactly the questions to ask the Org developers, I think.
>
>> > If you mean memory used by non-Lisp objects, I don't see how we could
>> > produce that without having infrastructure for tracking memory
>> > allocation, something that debugging malloc libraries already do.
>> 
>> Hmm. I am looking again at the profiler-report output. It seems to
>> report the memory allocation for individual function calls (that's what
>> I meant by "real" memory profiler). Do I miss something?
>
> What that profiler counts is calls to memory-allocation functions,
> that's all.  Without being able to account for memory which was freed
> after being allocated, these counts are useless.

Re: Memory usage report

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Sat, 19 Sep 2020 23:45:37 +0800
> 
> >> Some of the overlays and text properties are not strictly necessary.
> >> For example, text properties are sometimes used as cache to avoid
> >> parsing text multiple times. Is the resulting speedup worth extra memory
> >> usage? It is not clear since we do not have an easy way to determine the
> >> extra memory usage. 
> >
> > These are exactly the questions to ask the Org developers, I think.
> 
> As an org developer, how can I know the extra memory usage by that one
> specific type of text properties?

No, you should be able to tell whether "the resulting speedup worth
extra memory usage".  The memory used by a single overlay can be
easily determined by looking at the contents of an overlay object on
the C level, and similarly with text properties.

> > What that profiler counts is calls to memory-allocation functions,
> > that's all.  Without being able to account for memory which was freed
> > after being allocated, these counts are useless.
> 
> Got it. Then, I imagine that a simple delta of before/after running a
> function could be used to build a more useful memory report. Current GC
> report only provides the total usage, but no information on how
> individual function calls increased the memory usage.

Call garbage-collect before and after the function, and you can have
that.


> This might even be done in Elisp comparing `cons-cells-consed' and
> similar variables before/after each function call.

Not everything is exposed to Lisp.  For example, how many bytes are
consumed by a cons cell.

Re: Memory usage report

[Ihor Radchenko]

> Personally, when that extra data is smaller than the Lisp heap, I don't
> care to know where it's spent.

It indeed does not matter when the heap size is small. However, memory
usage tends to grow over time. My Emacs instance currently uses 976Mb
memory with 463MB in lisp objects and 25.8MB in buffers. The extra
non-lisp object size is almost same with the lisp object size (487.2MB). 

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> top shows more than 200Mb memory increase.
>>> Which are explained by the statistics produced by GC.  IOW, you have
>>> many more live Lisp objects, which take up those megabytes.
>> I meant that 200Mb is Lisp objects, but top shows more then 200Mb
>> (around 300Mb).  So, there is extra 100Mb coming from somewhere else.
>
> Personally, when that extra data is smaller than the Lisp heap, I don't
> care to know where it's spent.
>
>
>         Stefan

Re: Memory usage report

[Stefan Monnier]

>> Personally, when that extra data is smaller than the Lisp heap, I don't
>> care to know where it's spent.
>
> It indeed does not matter when the heap size is small. However, memory
> usage tends to grow over time. My Emacs instance currently uses 976Mb
> memory with 463MB in lisp objects and 25.8MB in buffers. The extra
> non-lisp object size is almost same with the lisp object size (487.2MB). 

That's reaching the level where I'd begin to care about it, indeed.
But being lazy, I'd wait to see a worse case being I investigate.

The recent reports of 2GB memory use with 500MB of heap are the kinds of
situations where the existence of a problem is much more clear.


        Stefan

Re: Memory usage report

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I thought I'd just visit all the buffers and look at which variables
>> available from each buffer are buffer-local.  I think all of that is
>> available from Lisp?
>
> You mean via `buffer-local-variables` ;-)

Even better.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no