How to debug memory leaks

How to debug memory leaks

[edgar]

[-- Attachment #1: Type: text/plain, Size: 2219 bytes --]

Hello,

I have noticed that my Emacs setup keeps using all my RAM, and it then 
goes for my swap memory. I think that it is leaking memory.

My use of Emacs at the moment is to work with a C++ finite element 
library (libMesh), and I am using Org to create literate programming. I 
am currently not doing any LaTeX (or otherwise fancy) exporting, but 
Org-ref does get loaded, among others and I also do unicode text 
exporting to make sure that some things would be included (#+include: 
my_file::block_name src cpp). I work with Makefiles and use the compile 
command from Emacs, as much as shell-command if needed. I am also trying 
to use helm-gtags. There is company running as well.

I noticed that when the 17.9 MB TAGS file (Global gtags) is read, the 
memory use jumps a lot and keeps increasing. It seems that font leaking 
is a common issue as well. For both reasons, I already deactivated 
moe-theme and projectile, and I keep doing garbage-collect constantly, 
all of which seems to have helped. However, the problem subsists.

I tried to follow what seemed to be the most instructive way of 
debugging this 
http://notes.secretsauce.net/notes/2015/10/05_memory-leak-debugging-tools.html). 
However, I get errors unbeknownst to me. Namely:

     The /usr/lib/libc-2.33.so file has no debug information.
Rebuild with -g, or install an appropriate debuginfo package.
   Error: Failed to add events.

     Error:	File /sys/kernel/debug/tracing/events/probe_libc/free not 
found.
Hint:	Perhaps this kernel misses some CONFIG_ setting to enable this 
feature?.

It is clear that I would have to recompile my kernel with debugging 
symbols, but that seems too much.

I attach a file with info on my Linux, Emacs version and the packages 
that I use. I hope that someone can help me to find and correct the 
memory leak, or point me in the right direction. Thanks.

-------------------------------------------------
This free account was provided by VFEmail.net - report spam to abuse@vfemail.net
 
ONLY AT VFEmail! - Use our Metadata Mitigator to keep your email out of the NSA's hands!
$24.95 ONETIME Lifetime accounts with Privacy Features!  
15GB disk! No bandwidth quotas!
Commercial and Bulk Mail Options!  

[-- Attachment #2: emacs-config-mem-leak.txt.gz --]
[-- Type: application/x-gzip, Size: 2693 bytes --]

Re: How to debug memory leaks

[Eli Zaretskii]

> Date: Sun, 21 Mar 2021 18:17:10 +0000
> From: edgar@openmail.cc
> 
> I have noticed that my Emacs setup keeps using all my RAM, and it then 
> goes for my swap memory. I think that it is leaking memory.

It's unlikely.

> I attach a file with info on my Linux, Emacs version and the packages 
> that I use. I hope that someone can help me to find and correct the 
> memory leak, or point me in the right direction. Thanks.

I'd start by looking in your init files for any customization that
affects GC.

Also, if you use Emacs 27.1, try the RC2 release candidate of Emacs
27.2, I think it fixed a problem which could create a situation where
GC is prevented.

Re: How to debug memory leaks

[edgar]

 From: 	Eli Zaretskii

> It's unlikely.

Yet, it happened--more than once (did you notice that I said "my Emacs 
setup" and not "Emacs"?). The good news: it's not happening anymore :)

> I'd start by looking in your init files for any customization that 
> affects GC.
> 
> Also, if you use Emacs 27.1, try the RC2 release candidate of Emacs 
> 27.2, I think it fixed a problem which could create a situation where 
> GC is prevented.

So very interesting. I did have an Emacs compiled by my system, but when 
the problem surfaced, I went for my distro's package to see if that 
would solve it (not really). When I brought moe-theme back, I disabled 
the DejaVu Sans Mono font, and Emacs behaved well. It was really loading 
the TAGS which added like 400 MB on the stack (virtual and 
resident--yeah, sounds unlikely, right?). Therefore, I went to look for 
an alternative.

I am not a fan of having a subprocess running, but after trying eglot 
with clangd, I was convinced that 150 MB more on my stack would be 
acceptable. Even more, because it goes away once the buffer is closed.

I haven't done a garbage collect, and I know that having near 750 MB in 
virtual and 140 MB in resident memory is still a lot, but it is at least 
not +4 GB in between RAM and swap.

I would next compile Emacs, but I'll settle for this right now, and get 
some work done.

Thank you, Eli and all the developers for providing the world with the 
most customisable editor ever! (sorry to Mr. Stallman for calling it an 
editor)

Re: How to debug memory leaks

[Stefan Monnier]

>> It's unlikely.
> Yet, it happened--more than once (did you notice that I said "my Emacs
> setup" and not "Emacs"?). The good news: it's not happening anymore :)

"unlikely" that this is a memory leak.  Not unlikely that it happens.

> Mono font, and Emacs behaved well. It was really loading the TAGS which
> added like 400 MB on the stack (virtual and resident--yeah, sounds unlikely,
> right?). Therefore, I went to look for an alternative.

Using a lot of memory is not the same as having a memory leak.

When you take a bath you will typically consume several hundred liters
of water.  Would you call that a leak?


        Stefan

Re: How to debug memory leaks

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2021-03-25 17:24]:
> >> It's unlikely.
> > Yet, it happened--more than once (did you notice that I said "my Emacs
> > setup" and not "Emacs"?). The good news: it's not happening anymore :)
> 
> "unlikely" that this is a memory leak.  Not unlikely that it happens.
> 
> > Mono font, and Emacs behaved well. It was really loading the TAGS which
> > added like 400 MB on the stack (virtual and resident--yeah, sounds unlikely,
> > right?). Therefore, I went to look for an alternative.
> 
> Using a lot of memory is not the same as having a memory leak.
> 
> When you take a bath you will typically consume several hundred liters
> of water.  Would you call that a leak?

:-)

My water tank is leaking if somebody can help...

But memory is also not "leaking", so chips are normally there and
remain usable for next time. Leaking is probably when it goes out of
defined containment. But what is the containment? 

We had a long thread on this and tried to debug but I do not know how
it ended, I have done my best.

What I can observe now since I switched from Hyperbola GNU/Linux-libre
to Parabola GNU/Linux-libre is that my Emacs uptime is more than 9
days. And I do use helm and helm-system-packages, as when I used those
previously I had such a huge swapping and impossibility to use the OS
to the point that I had to shut it down physically.

So maybe that memory effects, whatever they are, leaking, plumbing,
I do not observe it now, but who knows...

Jean

Re: How to debug memory leaks

[Stefan Monnier]

> So maybe that memory effects, whatever they are, leaking, plumbing,
> I do not observe it now, but who knows...

Call it "excessive memory use".  That term even has the advantage of
including a notion of personal judgment (in the term "excessive") which
indeed reflects the fact that maybe this is not the result of a bug, but
you still find it excessive (and other people may disagree without
making your judgment wrong since it's just an opinion).


        Stefan

Re: How to debug memory leaks

[edgar]

 From: 	Stefan Monnier
> > So maybe that memory effects, whatever they are, leaking, plumbing,
> > I do not observe it now, but who knows...
> 
> Call it "excessive memory use".  That term even has the advantage of
> including a notion of personal judgment (in the term "excessive") which
> indeed reflects the fact that maybe this is not the result of a bug, 
> but
> you still find it excessive (and other people may disagree without
>                                  making your judgment wrong since it's 
> just an opinion).
> 

The risk of leaving an unfruitful discussion is to let others think that 
a point has been made which could possibly influence or confuse 
outsiders and have unexpected adverse consequenes.

May be "management" is a better term, but obviously Stefan has a 
different idea of "leak" (I guess as related to memory injection--I 
don't know, and would not like to add yet another level of needless 
argument). I did write

     I think that it is leaking memory

which implies that I did not know if that was certain. I apologise for 
any misconstrued conclusions from my statement

     It seems that font leaking is a common issue as well

I meant that, in general terms, /font leaking has been found by other 
users to happen as well/. If by the end of:

     find(ing) and correct(ing) the memory leak, or point me in the right 
direction

it would have been found that there was no such leak, the conclusion 
would have been something like: oh, this was not a leak. However, we 
cannot really conclude this, because the debugging process never 
happened. Whether it was a leak or not is completely uncertain; I just 
replaced the pieces which I thought that were using memory excessively.

The point which is meant is that a process running in Emacs was probably 
using memory, did not liberate that memory after it finished, and used 
more memory the next time it was used, all of which consumed a lot of 
RAM and swap memory until the computer froze.

Can we leave it at that? It would have been nice to learn how to debug 
this, but we all have higher priorities, I think.

Again, thank you very much. I really enjoy my Emacs, and I owe it to you 
people :D .

Re: How to debug memory leaks

[Stefan Monnier]

> Can we leave it at that?

Sorry for the thread misdirection, indeed.

> It would have been nice to learn how to debug this,

It's actually really hard.  So the best is to `M-x report-emacs-bug` and
contribute as much concrete data as you can.  If you use an Emacs built
from Git, you can also include `M-x memory-report` as part of the hard data.


        Stefan

Re: How to debug memory leaks

[Jean Louis]

[-- Attachment #1: Type: text/plain, Size: 1575 bytes --]

* edgar <edgarlux@cryptolab.net> [2021-03-25 20:56]:
> From: 	Stefan Monnier
> > > So maybe that memory effects, whatever they are, leaking, plumbing,
> > > I do not observe it now, but who knows...
> > 
> > Call it "excessive memory use".  That term even has the advantage of
> > including a notion of personal judgment (in the term "excessive") which
> > indeed reflects the fact that maybe this is not the result of a bug, but
> > you still find it excessive (and other people may disagree without
> >                                  making your judgment wrong since it's
> > just an opinion).
> > 
> 
> The risk of leaving an unfruitful discussion is to let others think that a
> point has been made which could possibly influence or confuse outsiders and
> have unexpected adverse consequenes.
> 
> May be "management" is a better term, but obviously Stefan has a different
> idea of "leak" (I guess as related to memory injection--I don't know, and
> would not like to add yet another level of needless argument). I did write
> 
>     I think that it is leaking memory
> 
> which implies that I did not know if that was certain. I apologise for any
> misconstrued conclusions from my statement
> 
>     It seems that font leaking is a common issue as well
> 
> I meant that, in general terms, /font leaking has been found by other users
> to happen as well/. If by the end of:

I have 2 pictures attached on what happens sometimes on my side.

Is that maybe font leaking? It looks like some fonts leaked somewhere
and I don't know where... there is hole somewhere...


[-- Attachment #2: 2021-02-21-00:35:41a.png --]
[-- Type: image/png, Size: 247558 bytes --]

[-- Attachment #3: Screenshot-from-2021-03-14-11-02-46.png --]
[-- Type: image/png, Size: 67591 bytes --]

Re: How to debug memory leaks

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier wrote:

>> So maybe that memory effects, whatever they are, leaking,
>> plumbing, I do not observe it now, but who knows...
>
> Call it "excessive memory use". That term even has the
> advantage of including a notion of personal judgment (in the
> term "excessive") which indeed reflects the fact that maybe
> this is not the result of a bug, but you still find it
> excessive (and other people may disagree without making your
> judgment wrong since it's just an opinion).

In so many words - yes.

:)

-- 
underground experts united
https://dataswamp.org/~incal

Re: How to debug memory leaks

[edgar]

 From: 	Stefan Monnier
> Sorry for the thread misdirection, indeed.
> 
> It's actually really hard.  So the best is to `M-x report-emacs-bug` 
> and
> contribute as much concrete data as you can.  If you use an Emacs built
> from Git, you can also include `M-x memory-report` as part of the hard 
> data.

Wow! A real gentleman... on internet? Nah, I must be dreaming ;) . Thank 
you, Stefan.

Re: How to debug memory leaks

[edgar]

 From:  Jean Louis
> I have 2 pictures attached on what happens sometimes on my side.
> 
> Is that maybe font leaking? It looks like some fonts leaked somewhere
> and I don't know where... there is hole somewhere...

I think that Stefan already kind of gave a hint on this being a 
cumbersome task. If you want to try, may be you can help yourself with 
the link that I posted originally (where they show how to deal with this 
types of issues, especially a font leak). Bonne chance! :D .

Re: How to debug memory leaks

[Robert Thorpe]

edgar <edgarlux@cryptolab.net> writes:

>  From:  Jean Louis
>> I have 2 pictures attached on what happens sometimes on my side.
>> 
>> Is that maybe font leaking? It looks like some fonts leaked somewhere
>> and I don't know where... there is hole somewhere...
>
> I think that Stefan already kind of gave a hint on this being a 
> cumbersome task. If you want to try, may be you can help yourself with 
> the link that I posted originally (where they show how to deal with this 
> types of issues, especially a font leak). Bonne chance! :D .

About your problem Edgar....

I think you mentioned a while ago that you use shell-mode.  In that mode
Emacs keeps a buffer with all of the shell activity in it.  By default
it doesn't remove anything, at least on my Emacs.

Some programs give out a lot of text on standard output.  These programs
can make your shell buffers very large and cause them to consume lots of
memory.

In the Emacs manual it gives ways of limiting the size of these buffers.

BR,
Robert Thorpe

Re: How to debug memory leaks

[edgar]

Robert Thorpe wrote:
> I think you mentioned a while ago that you use shell-mode.  In that 
> mode
> Emacs keeps a buffer with all of the shell activity in it.  By default
> it doesn't remove anything, at least on my Emacs.
> 
> Some programs give out a lot of text on standard output.  These 
> programs
> can make your shell buffers very large and cause them to consume lots 
> of
> memory.
> 
> In the Emacs manual it gives ways of limiting the size of these 
> buffers.
> 
> BR,
> Robert Thorpe

Thanks, Robert. I'll look into it :) .

Re: How to debug memory leaks

[Stefan Monnier]

> I think you mentioned a while ago that you use shell-mode.  In that mode
> Emacs keeps a buffer with all of the shell activity in it.  By default
> it doesn't remove anything, at least on my Emacs.

The `M-x memory-report` in Emacs's Git version makes these issues
immediately visible.

Sadly, in my experience the kinds of excessive memory use problems
people experience come from elsewhere (such as the occasional need to
load all the fonts that exist in your system to try and figure out if
you have a font that can display that damn emoticon that your brother
found cute).


        Stefan

Re: How to debug memory leaks

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier wrote:

> Sadly, in my experience the kinds of excessive memory use
> problems people experience come from elsewhere (such as the
> occasional need to load all the fonts that exist in your
> system to try and figure out if you have a font that can
> display that damn emoticon that your brother found cute).

HA! :)

Don't forget to tell me if and when you feel I do stuff like
that...

Can't recall a memory problem tho? ... hoho

(actually my memory was close to photographic before I got
this nerve damage or whatever it is. but it is still better
than everyone else around me. maybe I hang out with a bunch of
bozos tho)

Wasn't it the great super-engineer Steve Wozniak who crashed
his aeroplane and lost the ability to store new memories in
his brain?

@book{iwoz,
  author     = {Steve Wozniak},
  isbn       = {0-393-06143-4},
  publisher  = {Norton \& Company},
  title      = {iWoz},
  year       = {2006}
}

-- 
underground experts united
https://dataswamp.org/~incal

Re: How to debug memory leaks

[Emanuel Berg via Users list for the GNU Emacs text editor]

Sorry about sending this twice, I forgot I could CC it to
Steve Wozniak himself, I found his mail with Google, maybe it
is bogus, no idea. Well, never mention anyone in a mail
without CC it to him/her, if you can. Is a good rule I'm proud
of :) I hope he doesn't find this mail too silly or
just... irrelevant.

Stefan Monnier wrote:

> Sadly, in my experience the kinds of excessive memory use
> problems people experience come from elsewhere (such as the
> occasional need to load all the fonts that exist in your
> system to try and figure out if you have a font that can
> display that damn emoticon that your brother found cute).

HA! :)

Don't forget to tell me if and when you feel I do stuff
like that...

Can't recall a memory problem tho? ... hoho

(actually my memory was close to photographic before I got
this nerve damage or whatever it is. but it is still better
than everyone else around me. maybe I hang out with a bunch of
bozos tho)

Wasn't it the great super-engineer Steve Wozniak who crashed
his aeroplane and lost the ability to store new memories in
his brain?

@book{iwoz,
  author     = {Steve Wozniak},
  isbn       = {0-393-06143-4},
  publisher  = {Norton \& Company},
  title      = {iWoz},
  year       = {2006}
}

-- 
underground experts united
https://dataswamp.org/~incal

Re: How to debug memory leaks

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I think you mentioned a while ago that you use shell-mode.  In that mode
>> Emacs keeps a buffer with all of the shell activity in it.  By default
>> it doesn't remove anything, at least on my Emacs.
>
> The `M-x memory-report` in Emacs's Git version makes these issues
> immediately visible.
>
> Sadly, in my experience the kinds of excessive memory use problems
> people experience come from elsewhere (such as the occasional need to
> load all the fonts that exist in your system to try and figure out if
> you have a font that can display that damn emoticon that your brother
> found cute).

:-)

Would there be any way for Emacs to unload unused fonts? GC?

As a thought, could it be possible to implement something like "clean
emacs state" or/and "clean buffer state". It would throw away undo
history, unload unused stuff, for example in eshell/term/ansi-term kill
the buffer text, etc. Something that would work either on entire Emacs state,
or just on a buffer state; or maybe on both.

Is it just to set some variables to nil and collect garbage or is it
more work than so?

Re: How to debug memory leaks

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Date: Fri, 26 Mar 2021 15:57:09 +0100
> Cc: help-gnu-emacs@gnu.org
> 
> Would there be any way for Emacs to unload unused fonts? GC?

We already do that.

> As a thought, could it be possible to implement something like "clean
> emacs state" or/and "clean buffer state". It would throw away undo
> history, unload unused stuff, for example in eshell/term/ansi-term kill
> the buffer text, etc. Something that would work either on entire Emacs state,
> or just on a buffer state; or maybe on both.

That's what GC does.

Re: How to debug memory leaks

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Fri, 26 Mar 2021 15:57:09 +0100
>> Cc: help-gnu-emacs@gnu.org
>> 
>> Would there be any way for Emacs to unload unused fonts? GC?
>
> We already do that.
>
>> As a thought, could it be possible to implement something like "clean
>> emacs state" or/and "clean buffer state". It would throw away undo
>> history, unload unused stuff, for example in eshell/term/ansi-term kill
>> the buffer text, etc. Something that would work either on entire Emacs state,
>> or just on a buffer state; or maybe on both.
>
> That's what GC does.
Yes, but GC can't know if I don't need undo history, right, or old text
in eshell buffer?

Can we maybe get a "cleanup-hook" so we can specify per mode which
data/variables shoudl be set to nil so GC can do it's job?

Re: How to debug memory leaks

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 26 Mar 2021 16:09:33 +0100
> 
> > That's what GC does.
> Yes, but GC can't know if I don't need undo history, right

And neither do you.  GC compacts the undo history, though, and frees
the slack.

> or old text in eshell buffer?

Delete it, and it will be free'd.

> Can we maybe get a "cleanup-hook" so we can specify per mode which
> data/variables shoudl be set to nil so GC can do it's job?

It's much easier to delete unneeded stuff, and let GC recycle the
memory.

Re: How to debug memory leaks

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Fri, 26 Mar 2021 16:09:33 +0100
>> 
>> > That's what GC does.
>> Yes, but GC can't know if I don't need undo history, right
>
> And neither do you.  GC compacts the undo history, though, and frees
> the slack.

Ok, good if GC compacts the undo, but I could explicitly wish/know
sometimes that I could "freeze" the undo. I could just put on my own
interactive command that does this instead of using
M-: (setq buffer-undo-list nil), but there could be some other stuff
based on mode, like eshell or term or I don't know what. That could help
GC.

>> or old text in eshell buffer?
>
> Delete it, and it will be free'd.

Yes, so that is what I am saying; a hook to put all kind of various
"delete" stuff for various modes so we can run interactive command, say
"clean-buffer" or whatever, and it will run the apropriate "deletes" and
maybe other stuff based on mode.

>> Can we maybe get a "cleanup-hook" so we can specify per mode which
>> data/variables shoudl be set to nil so GC can do it's job?
>
> It's much easier to delete unneeded stuff, and let GC recycle the
> memory.

:-). Yes that is exactly what I ask for; a hook where we could put what
will be deleted and that will be called on user request interactively.

Just as a note: no idea if you have found the memory leak or not, the
big one that everyone was repporting few weeks ago, but I haven't
experienced any leaks or slugishness with lately builds. I guess you
found it? :).

Re: How to debug memory leaks

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Fri, 26 Mar 2021 16:53:58 +0100
> 
> >> Yes, but GC can't know if I don't need undo history, right
> >
> > And neither do you.  GC compacts the undo history, though, and frees
> > the slack.
> 
> Ok, good if GC compacts the undo, but I could explicitly wish/know
> sometimes that I could "freeze" the undo. I could just put on my own
> interactive command that does this instead of using
> M-: (setq buffer-undo-list nil), but there could be some other stuff
> based on mode, like eshell or term or I don't know what. That could help
> GC.

You will have to come up with more specific ideas, the above is too
general to be useful.  If you suggest that some modes don't need undo,
then I don't think I agree.

> >> or old text in eshell buffer?
> >
> > Delete it, and it will be free'd.
> 
> Yes, so that is what I am saying; a hook to put all kind of various
> "delete" stuff for various modes so we can run interactive command, say
> "clean-buffer" or whatever, and it will run the apropriate "deletes" and
> maybe other stuff based on mode.

You have post-command-hook already; if you want to keep only a small
part of a shell buffer, you can use that hook to delete the old
stuff.  This is clearly specific to your use patterns, though.

And after all that, please keep in mind that in all the reports about
multi-GB memory footprints that people complained about, the actual
memory used by Lisp objects (what GC can free) was quite small, almost
negligible.  The real problem, whatever it was, wasn't with GC not
freeing enough.

> Just as a note: no idea if you have found the memory leak or not, the
> big one that everyone was repporting few weeks ago, but I haven't
> experienced any leaks or slugishness with lately builds. I guess you
> found it? :).

Not really.  We fixed a problem where fiddling with some GC-related
options could cause Emacs stop calling GC, but that's all.

Re: How to debug memory leaks

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Fri, 26 Mar 2021 16:53:58 +0100
>> 
>> >> Yes, but GC can't know if I don't need undo history, right
>> >
>> > And neither do you.  GC compacts the undo history, though, and frees
>> > the slack.
>> 
>> Ok, good if GC compacts the undo, but I could explicitly wish/know
>> sometimes that I could "freeze" the undo. I could just put on my own
>> interactive command that does this instead of using
>> M-: (setq buffer-undo-list nil), but there could be some other stuff
>> based on mode, like eshell or term or I don't know what. That could help
>> GC.
>
> You will have to come up with more specific ideas, the above is too
> general to be useful.  If you suggest that some modes don't need undo,
> then I don't think I agree.

No, no, I don't mean that some modes don't need undo. I mean, if I am
working in a buffer, say some C file, and I think Emacs get sluggish, I
could do some M-x clean-buffer and have it setq bunch of variables to
nil, amongst them undo so GC can collect them. It is just that these
variables might be different depending of type of buffer, active modes
etc. I can of course do my own interactive command. But it is maybe
better if there was a "standard" hook which mode writers can use, and
standard command, so users like me don't need to write their own.

>> >> or old text in eshell buffer?
>> >
>> > Delete it, and it will be free'd.
>> 
>> Yes, so that is what I am saying; a hook to put all kind of various
>> "delete" stuff for various modes so we can run interactive command, say
>> "clean-buffer" or whatever, and it will run the apropriate "deletes" and
>> maybe other stuff based on mode.
>
> You have post-command-hook already; if you want to keep only a small
> part of a shell buffer, you can use that hook to delete the old
> stuff.  This is clearly specific to your use patterns, though.

Ok, I didn't know I can use post-command-hook for that. I'll try and
play with it. Thanks.

> And after all that, please keep in mind that in all the reports about
> multi-GB memory footprints that people complained about, the actual
> memory used by Lisp objects (what GC can free) was quite small, almost
> negligible.  The real problem, whatever it was, wasn't with GC not
> freeing enough.

To be honest, I don't really think it is very much needed, more of a "in
case of" thing. Lately I haven't experienced any memory problems with
Emacs, and it feels fast and responsive as ever before. So well done job
there.

>> Just as a note: no idea if you have found the memory leak or not, the
>> big one that everyone was repporting few weeks ago, but I haven't
>> experienced any leaks or slugishness with lately builds. I guess you
>> found it? :).
>
> Not really.  We fixed a problem where fiddling with some GC-related
> options could cause Emacs stop calling GC, but that's all.

Ok, seems like it works well now.

Re: How to debug memory leaks

[Stefan Monnier]

> No, no, I don't mean that some modes don't need undo. I mean, if I am
> working in a buffer, say some C file, and I think Emacs get sluggish, I
> could do some M-x clean-buffer and have it setq bunch of variables to
> nil, amongst them undo so GC can collect them. It is just that these
> variables might be different depending of type of buffer, active modes
> etc.

I suspect that it would be "ineffective by design": in order to realize
that such-and-such setting might solve a performance problem, you'll
generally need first to be aware of that performance problem, and
usually once you're aware of it you can just fix the problem itself.

E.g. I don't think flushing buffer-undo-list will ever help
with sluggishness.

> Ok, I didn't know I can use post-command-hook for that. I'll try and
> play with it. Thanks.

Note also that if you want to make Emacs sluggish, a good place to start
is by adding stuff to post-command-hook ;-)


        Stefan

Re: How to debug memory leaks

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> No, no, I don't mean that some modes don't need undo. I mean, if I am
>> working in a buffer, say some C file, and I think Emacs get sluggish, I
>> could do some M-x clean-buffer and have it setq bunch of variables to
>> nil, amongst them undo so GC can collect them. It is just that these
>> variables might be different depending of type of buffer, active modes
>> etc.
>
> I suspect that it would be "ineffective by design":

It is inspired by 3D modelling/animation packages. They usually keep an
undo stack of transformation matrices for models, materials etc. Since
it quite fast becomes lots of data, they usually have something like
"freeze transforms" or "clean scene" etc. One that I worked with used
modified TCL interpretter with garbage collector to manage the scene
graph, which is similar appliclation architecture to Emacs + Lisp.

> in order to realize
> that such-and-such setting might solve a performance problem, you'll
> generally need first to be aware of that performance problem, and
> usually once you're aware of it you can just fix the problem itself.

Yes, that is why I ment a hook that will be avaialble for mode writers
to put potential "cleanup" into since users probably don't know and
should not need to know the details.

I used undo history as example, but I ment bunch of other stuff. No idea
how much it would give in practice though.

Re: How to debug memory leaks

[Stefan Monnier]

> Yes, that is why I ment a hook that will be avaialble for mode writers
> to put potential "cleanup" into since users probably don't know and
> should not need to know the details.

What I meant is that if the mode's writer knows that this variable can
be a source of problem, he can just as well fix those problem (or
report the problem and have someone else fix the problem).

> I used undo history as example, but I ment bunch of other stuff. No idea
> how much it would give in practice though.

So, there's no concrete existing example of stuff that could be added to
this "cleanup hook" :-(


        Stefan

Re: How to debug memory leaks

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier wrote:

>> I used undo history as example, but I ment bunch of other
>> stuff. No idea how much it would give in practice though.
>
> So, there's no concrete existing example of stuff that could
> be added to this "cleanup hook" :-(

Yeah, what is the "bunch of other stuff"? Do tell so we can
all check :)

But: is that kind of data really in such quantities that it
would do this?

Also, is data that piles up a memory leak? What's the
definition of a memory leak?

-- 
underground experts united
https://dataswamp.org/~incal

Re: How to debug memory leaks

[Emanuel Berg via Users list for the GNU Emacs text editor]

> Also, is data that piles up a memory leak? What's the
> definition of a memory leak?

Aren't there tools to detect memory leaks, how do they work?

-- 
underground experts united
https://dataswamp.org/~incal

Re: How to debug memory leaks

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Yes, that is why I ment a hook that will be avaialble for mode writers
>> to put potential "cleanup" into since users probably don't know and
>> should not need to know the details.
>
> What I meant is that if the mode's writer knows that this variable can
> be a source of problem, he can just as well fix those problem (or
> report the problem and have someone else fix the problem).

"Source of a problem" is probably a bit too heavy expression :). For
example if I have had a file with lots of undo history, I wouldn't say
it was a source of a problem, but I might decide that I wish to get rid
of the undo history, and start it from clean, without killing the buffer
and opening it a new.

>> I used undo history as example, but I ment bunch of other stuff. No idea
>> how much it would give in practice though.
>
> So, there's no concrete existing example of stuff that could be added to
> this "cleanup hook" :-(

I was trying to reason in terms of a general facility. What do I know
what people use, no idea what can be "throwable" in different modes.

Re: How to debug memory leaks

[Stefan Monnier]

> "Source of a problem" is probably a bit too heavy expression :).  For
> example if I have had a file with lots of undo history, I wouldn't say
> it was a source of a problem, but I might decide that I wish to get rid
> of the undo history, and start it from clean, without killing the buffer
> and opening it a new.

I don't think I've ever seen the undo history being a source of sluggishness ;-)

>>> I used undo history as example, but I ment bunch of other stuff. No idea
>>> how much it would give in practice though.
>> So, there's no concrete existing example of stuff that could be added to
>> this "cleanup hook" :-(
> I was trying to reason in terms of a general facility. What do I know
> what people use, no idea what can be "throwable" in different modes.

The problem is that sluggishness (just like excessive memory use) can
come from many many different places.  So it's hard to come up with
a tool that handles "the usual suspects" because there are too many
usual suspects.

What we have instead is `M-x profiler-start/report` which should(?) let
you find out what is the source of the sluggishness.  Similarly we have
a `M-x memory-report` for excessive memory use.

They don't work great, admittedly, but this is a hard problem.


        Stefan

Re: How to debug memory leaks

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> "Source of a problem" is probably a bit too heavy expression :).  For
>> example if I have had a file with lots of undo history, I wouldn't say
>> it was a source of a problem, but I might decide that I wish to get rid
>> of the undo history, and start it from clean, without killing the buffer
>> and opening it a new.
>
> I don't think I've ever seen the undo history being a source of sluggishness ;-)
>
>>>> I used undo history as example, but I ment bunch of other stuff. No idea
>>>> how much it would give in practice though.
>>> So, there's no concrete existing example of stuff that could be added to
>>> this "cleanup hook" :-(
>> I was trying to reason in terms of a general facility. What do I know
>> what people use, no idea what can be "throwable" in different modes.
>
> The problem is that sluggishness (just like excessive memory use) can
> come from many many different places.  So it's hard to come up with
> a tool that handles "the usual suspects" because there are too many
> usual suspects.

Indeed, and I didn't ment undo as a single of slugishness either, not
does it even have to be slughisness. That is why I say "bunch of stuff";
unspecified :-). I ment a single place where user, mode creators etc,
can add bunch of setq vars to nil and would be explicitly called by
user from some function; something like reset to starting state or some
kind of that. I said in some previous answer to Eli, I don't even
percieve later versions as sluggish. I got ideas when there was that
memory leak, and I just suggest it as a mechanism for the future, in
case of.

> What we have instead is `M-x profiler-start/report` which should(?) let
> you find out what is the source of the sluggishness.  Similarly we have
> a `M-x memory-report` for excessive memory use.
>
> They don't work great, admittedly, but this is a hard problem.

Better than nothing! :):

Re: How to debug memory leaks

[Emanuel Berg via Users list for the GNU Emacs text editor]

Stefan Monnier wrote:

> What we have instead is `M-x profiler-start/report` which
> should(?) let you find out what is the source of the
> sluggishness. Similarly we have a `M-x memory-report` for
> excessive memory use.

OT (?): remember playing games? You never had the required
computer RAM and stuff but that was solved by downgrading the
graphics in particular, reducing the amount of details etc, and
suddenly it went really fast! And then even the guys that had
really strong computers got envious because even if it worked
great for them it still didn't have that "snap" which you had
achieved by all the methods applied at once (or
simultaneously).

So is there a way to downgrade Emacs in much the same way?

What happens with -Q?

Can't we have:

$ emacs                                    \
    --not-module "uwe-brauer-mac-os-hacks" \
    --not-module "jeans-super-key"         \
    etc etc 

and then you could experiment what was needed and not for you?

Downsides to that are easy to see but would that increase the
speed, i.e. the interactive feel?

What is it that -Q removes? but it is too much

$ emacs -Qp 0.5 # Q percent, only remove 50% of what -Q does?

A file of Elisp that is not used, why should that slow
down anything? It really does? If so, why?

E.g., I don't move around in my apartment slower for every
book I put in my bookshelf...

-- 
underground experts united
https://dataswamp.org/~incal