memory leaks

memory leaks

[Madhu]

There is obviously some memory leak in emacs and I don't know how to
trace the culprit.  Emacs lops up 2GB RSS: ps axuw |grep emacs

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
madhu     5434  6.9 54.2 2235612 2115564 ?     SNsl Sep19  66:42 emacs --daemon --debug-init

(garbage-collect) shows

((conses 16 981544 136697)
 (symbols 48 56864 5)
 (strings 32 312273 12669)
 (string-bytes 1 18002558)
 (vectors 16 68285)
 (vector-slots 8 1780950 150102)
 (floats 8 1089 2405)
 (intervals 56 6605 2469)
 (buffers 992 79))

So does this look like a X fonts issue?  How can I figure out which
package is the culprit.

1. Emacs has a bug where I can't look at /proc/5434/emacs from within
emacs it will only read 16384 bytes from the proc filesystem

2. Trying to Unload a feature hits all sorts of design and
implementation bugs with the cl-generics scourge which is unfortunately
used to implement emacs.  Is unloading features likely to help at all in
reducing the memory and getting a clue about the errant package?

Re: memory leaks

[Jean-Christophe Helary]

I had something similar happening the other day and Eli told me it was probably fixed with some code recently committed to master. Have you tried a recent build ?

Jean-Christophe Helary 

> On Sep 20, 2019, at 16:23, Madhu <enometh@meer.net> wrote:
> 
> 
> There is obviously some memory leak in emacs and I don't know how to
> trace the culprit.  Emacs lops up 2GB RSS: ps axuw |grep emacs
> 
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> madhu     5434  6.9 54.2 2235612 2115564 ?     SNsl Sep19  66:42 emacs --daemon --debug-init

Jean-Christophe Helary
-----------------------------------------------
http://mac4translators.blogspot.com @brandelune

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Fri, 20 Sep 2019 12:53:46 +0530
> 
> 
> There is obviously some memory leak in emacs and I don't know how to
> trace the culprit.  Emacs lops up 2GB RSS: ps axuw |grep emacs
> 
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> madhu     5434  6.9 54.2 2235612 2115564 ?     SNsl Sep19  66:42 emacs --daemon --debug-init

Which version of Emacs?  If this is a development snapshot, when was
it built?

> 1. Emacs has a bug where I can't look at /proc/5434/emacs from within
> emacs it will only read 16384 bytes from the proc filesystem

Please report as a bug using "M-x report-emacs-bug".

> 2. Trying to Unload a feature hits all sorts of design and
> implementation bugs with the cl-generics scourge which is unfortunately
> used to implement emacs.  Is unloading features likely to help at all in
> reducing the memory and getting a clue about the errant package?

Likewise.

Re: memory leaks

[Madhu]

* Eli Zaretskii <mailman.572.1568967069.2190.help-gnu-emacs@gnu.org> :
Wrote on Fri, 20 Sep 2019 11:11:11 +0300:

>> From: Madhu <enometh@meer.net>
>> Date: Fri, 20 Sep 2019 12:53:46 +0530
>> There is obviously some memory leak in emacs and I don't know how to
>> trace the culprit.  Emacs lops up 2GB RSS: ps axuw |grep emacs
>>
>> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>> madhu 5434 6.9 54.2 2235612 2115564 ?  SNsl Sep19 66:42 emacs
>> --daemon --debug-init
>
> Which version of Emacs?  If this is a development snapshot, when was
> it built?

Sorry I neglected to include that info: it is master from around
20-Aug-2019: (commit 08d7cab)

GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw scroll
bars) of 2019-08-20

If there isn't a known problem I'd still appreciate a clue or two in
trying to figure out what the problem is.

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Fri, 20 Sep 2019 14:01:05 +0530
> 
> >> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> >> madhu 5434 6.9 54.2 2235612 2115564 ?  SNsl Sep19 66:42 emacs
> >> --daemon --debug-init
> >
> > Which version of Emacs?  If this is a development snapshot, when was
> > it built?
> 
> Sorry I neglected to include that info: it is master from around
> 20-Aug-2019: (commit 08d7cab)

Then what you see is almost certainly a problem with GC that has been
solved recently.  Update from Git and rebuild.

Re: memory leaks

[Madhu]

* Eli Zaretskii <mailman.578.1568971399.2190.help-gnu-emacs@gnu.org> :
Wrote on Fri, 20 Sep 2019 12:21:23 +0300:

>> From: Madhu <enometh@meer.net>
>> Date: Fri, 20 Sep 2019 14:01:05 +0530
>>
>> >> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>> >> madhu 5434 6.9 54.2 2235612 2115564 ?  SNsl Sep19 66:42 emacs
>> >> --daemon --debug-init
>> >
>> > Which version of Emacs?  If this is a development snapshot, when was
>> > it built?
>>
>> Sorry I neglected to include that info: it is master from around
>> 20-Aug-2019: (commit 08d7cab)
>
> Then what you see is almost certainly a problem with GC that has been
> solved recently.  Update from Git and rebuild.

One year on I'm having the same problems on master (and I've been
noticing it for a few months now)

virtual memory reports 2.9G resident 2.5G shared 10.9M.

$ ps o pid,rss,drs,sz,share,start_time,vsize,cmd 26285

  PID   RSS   DRS  SIZE - START    VSZ CMD
26285 2643236 2996379 2664940 - Sep09 2998948 /7/gtk/emacs/build-xt-xft/src/emacs --debug-init --daemon

GC reports
Memory information:
((conses 16 3591911 2541685)
 (symbols 48 87049 452)
 (strings 32 528119 452566)
 (string-bytes 1 30189681)
 (vectors 16 217149)
 (vector-slots 8 3232842 6057920)
 (floats 8 1637 5252)
 (intervals 56 501483 50429)
 (buffers 992 581))

which is a small fraction of what emacs is using.

This is on a recent emacs from master
In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d
scroll bars)  of 2020-09-06 built on maher
Base Repository revision: 6fc502c1ef327ab357c971b9bffbbd7cb6a436f1
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000


Perhaps there is already a bug report for this on debbugs?

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Sat, 12 Sep 2020 18:32:11 +0530
> 
> One year on I'm having the same problems on master (and I've been
> noticing it for a few months now)
> 
> virtual memory reports 2.9G resident 2.5G shared 10.9M.

Fascinating.  Why no one else has these problems, I wonder.

> Perhaps there is already a bug report for this on debbugs?

I don't think so, so please file a bug report with all the details
(mainly how do you get from starting Emacs to such a huge RSS).

Re: memory leaks

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > virtual memory reports 2.9G resident 2.5G shared 10.9M.
>
> Fascinating.  Why no one else has these problems, I wonder.

I have the problem.  From time to time I see that my memory is getting
out and Emacs uses 4 GB of it.  I use to restart Emacs then.

Michael.

Re: memory leaks

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Sat, 12 Sep 2020 15:55:33 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > > virtual memory reports 2.9G resident 2.5G shared 10.9M.
> >
> > Fascinating.  Why no one else has these problems, I wonder.
> 
> I have the problem.  From time to time I see that my memory is getting
> out and Emacs uses 4 GB of it.  I use to restart Emacs then.

Then I wonder how come you didn't report it earlier, with all the
details.  Please do that now, TIA.

Re: memory leaks

[Igor Sosa Mayor]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Madhu <enometh@meer.net>
>> Date: Sat, 12 Sep 2020 18:32:11 +0530
>> 
>> One year on I'm having the same problems on master (and I've been
>> noticing it for a few months now)
>> 
>> virtual memory reports 2.9G resident 2.5G shared 10.9M.
>
> Fascinating.  Why no one else has these problems, I wonder.

[...]

I am also experiencing a similar problem (version 27.1 in archlinux)...

Re: memory leaks

[Eli Zaretskii]

> From: Igor Sosa Mayor <joseleopoldo1792@gmail.com>
> Date: Sat, 12 Sep 2020 21:14:01 +0200
> 
> I am also experiencing a similar problem (version 27.1 in archlinux)...

Then I'd appreciate if everyone who is experiencing these problems
will please describe them in a full-fledged bug report, including how
long it takes to get to such large RSS values from the start.

Please don't leave such problems unreported.  They will not fix
themselves.

TIA

Re: memory leaks

[Igor Sosa Mayor]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Igor Sosa Mayor <joseleopoldo1792@gmail.com>
>> Date: Sat, 12 Sep 2020 21:14:01 +0200
>> 
>> I am also experiencing a similar problem (version 27.1 in archlinux)...
>
> Then I'd appreciate if everyone who is experiencing these problems
> will please describe them in a full-fledged bug report, including how
> long it takes to get to such large RSS values from the start.
>
> Please don't leave such problems unreported.  They will not fix
> themselves.

Of course. But I'm doing some preliminary research on my own, because I
thought, maybe it is a problem with one of the hundreds of packages I
load...

Re: memory leaks

[Eli Zaretskii]

> From: Igor Sosa Mayor <joseleopoldo1792@gmail.com>
> Date: Sat, 12 Sep 2020 21:28:27 +0200
> 
> > Please don't leave such problems unreported.  They will not fix
> > themselves.
> 
> Of course. But I'm doing some preliminary research on my own, because I
> thought, maybe it is a problem with one of the hundreds of packages I
> load...

Thank you.

Re: memory leaks

[Madhu]

* Eli Zaretskii <83imcidcfp.fsf@gnu.org> :
Wrote on Sat, 12 Sep 2020 22:26:02 +0300:
> Then I'd appreciate if everyone who is experiencing these problems
> will please describe them in a full-fledged bug report, including how
> long it takes to get to such large RSS values from the start.
>
> Please don't leave such problems unreported.  They will not fix
> themselves.

The problem is that there is nothing to report except that the RSS is
growing unreasonably.  In normal operation with all my packages loaded
emacs wouldn't go beyond around 500M RSS even when running for a week.

I usually only notice the leak when it has gone beyond 2G - when linux
refuses to suspend because I have limited swap.  In most cases emacs
would be running for a few days.  There are over 600 `features' in these
emacs but the major memory hog packages I was using are magit, sly,
gnus, and ivy - but I don't think these can be the culprits.

ATM I have a gtk emacs instance that is running at ~900M RSS, which may
be an early stage.  If you have any suggestions on how to go about
debugging GC - it would help in submitting a useful bug repoprt. But as
of now I have nothing more than "it leaks"

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Sun, 13 Sep 2020 08:47:29 +0530
> 
> > Please don't leave such problems unreported.  They will not fix
> > themselves.
> 
> The problem is that there is nothing to report except that the RSS is
> growing unreasonably.  In normal operation with all my packages loaded
> emacs wouldn't go beyond around 500M RSS even when running for a week.

When you file a bug report, Emacs collects important information about
your build and the environment in which you run Emacs.  That already
tells the Emacs developers a lot, and can give ideas regarding the
possible reasons.

Then I suggest to tell in your report how many buffers you have in
that session, and what is their total size.

And finally, I'd encourage you to try to describe your use patterns,
especially regarding operations that could require significant amounts
of memory: visiting large file, loading many fonts, creating large
buffers that don't visit files, etc.  If you can afford keeping notes
about what you did immediately prior to RSS becoming significantly
larger, that is even better.

> ATM I have a gtk emacs instance that is running at ~900M RSS, which may
> be an early stage.  If you have any suggestions on how to go about
> debugging GC - it would help in submitting a useful bug repoprt.

I don't think the problem is in GC, and the information about that is
part of the info collected by report-emacs-bug anyway.

Thanks.

Re: memory leaks

[Stefan Monnier]

> The problem is that there is nothing to report except that the RSS is
> growing unreasonably.  In normal operation with all my packages loaded
> emacs wouldn't go beyond around 500M RSS even when running for a week.

The `garbage-collect` command actually can return info about the current
Lisp heap use.  The `memory-usage` package can display it in a more
user-friendly format.

That can be very useful to see if those 500MB belong to the Lisp heap or
to ... something else (typically libraries, like those of the GUI or
those used to display images).


        Stefan

Re: memory leaks

[Madhu]

* Stefan Monnier <jwvh7s1780o.fsf-monnier+emacs@gnu.org> :
Wrote on Sun, 13 Sep 2020 16:14:36 -0400:
> The `garbage-collect` command actually can return info about the current
> Lisp heap use.  The `memory-usage` package can display it in a more
> user-friendly format.
>
> That can be very useful to see if those 500MB belong to the Lisp heap or
> to ... something else (typically libraries, like those of the GUI or
> those used to display images).

The numbers I reported by garbage-collect do not account for the memory
usage. FWIW I've posted that info again at
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43395

In 2019 i suspected the culprit was the fonts system.  It doesn't seem
to be toolkit related. I don't use emacs as an imageviewer (but I have
to check that). Since the interest in CONFIG_PSI there are a number of
tools on github that monitor memory - it should be possible to come up
with some tool that alerts emacs of memory surges while keeping some
history.

Re: memory leaks

[Madhu]

* Madhu <m35z8gise6.fsf@leonis4.robolove.meer.net> :
Wrote on Mon, 14 Sep 2020 15:36:57 +0530:
> The numbers I reported by garbage-collect do not account for the memory
> usage. FWIW I've posted that info again at
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43395
>
> In 2019 i suspected the culprit was the fonts system.  It doesn't seem
> to be toolkit related. I don't use emacs as an imageviewer (but I have
> to check that). Since the interest in CONFIG_PSI there are a number of
> tools on github that monitor memory - it should be possible to come up
> with some tool that alerts emacs of memory surges while keeping some
> history.

I found test case for this bug and I sent mail to 43395@debbugs.gnu.org,
and qmail told me that the mail was accepted:
delivery 1: success:
209.51.188.43_accepted_message./Remote_host_said:_250_OK_id=1kc8Bf-00061T-3u/
But the bugreport didn't show up on gnu.emacs.bug.

Here is the test case:

#+BEGIN_SRC
$ cat > f.c
#include <stdio.h>
int
main()
{
  char c = ' ';
  while (c != 'q' && c != 'Q')
    {
      fprintf(stdout, "Press q then enter to quit: ");
      c = fgetc(stdin);
    }
  return 0;
}
^D

$ gcc f.c
$ emacs -Q
#+END_SRC

M-x shell-command ./a.out

Then the process hangs. But Emacs' memory grows unbounded.

WARNING. Be careful to interrupt it with ^G before the OOM killer
kicks in.  If you have swap you may lose control of your machine
indefinitely.

After you interrupt the sub process, Emacs is left with unreclaimed
gigabytes of RSS

Please let me know if you can reproduce this

Re: memory leaks

[Madhu]

[Test case follows. Sorry Drew for trying to post to help-gnu and
debbugs simultaneously, but this is my third attempt at sending this
test case - I've tried sending it to debbugs.gnu.org and to
gmane.emacs.help newsgroup - and my messages have been dropped]


* Madhu <m35z8gise6.fsf@leonis4.robolove.meer.net> :
Wrote on Mon, 14 Sep 2020 15:36:57 +0530:
> The numbers I reported by garbage-collect do not account for the memory
> usage. FWIW I've posted that info again at
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43395

I found test case for this bug and I sent mail to 43395@debbugs.gnu.org,
and qmail told me that the mail was accepted:
delivery 1: success:
209.51.188.43_accepted_message./Remote_host_said:_250_OK_id=1kc8Bf-00061T-3u/
But the bugreport didn't show up on gnu.emacs.bug.

Here is the test case:

#+BEGIN_SRC
$ cat > f.c
#include <stdio.h>
int
main()
{
  char c = ' ';
  while (c != 'q' && c != 'Q')
    {
      fprintf(stdout, "Press q then enter to quit: ");
      c = fgetc(stdin);
    }
  return 0;
}
^D

$ gcc f.c
$ emacs -Q
#+END_SRC

M-x shell-command ./a.out

Then the process hangs. But Emacs' memory grows unbounded.

WARNING. Be careful to interrupt it with ^G before the OOM killer
kicks in.  If you have swap you may lose control of your machine
indefinitely.

After you interrupt the sub process, Emacs is left with unreclaimed
gigabytes of RSS

Please let me know if you can reproduce this

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Tue, 10 Nov 2020 06:54:56 +0530
> Cc: 43395@debbugs.gnu.org
> 
> #+BEGIN_SRC
> $ cat > f.c
> #include <stdio.h>
> int
> main()
> {
>   char c = ' ';
>   while (c != 'q' && c != 'Q')
>     {
>       fprintf(stdout, "Press q then enter to quit: ");
>       c = fgetc(stdin);
>     }
>   return 0;
> }
> ^D
> 
> $ gcc f.c
> $ emacs -Q
> #+END_SRC
> 
> M-x shell-command ./a.out
> 
> Then the process hangs. But Emacs' memory grows unbounded.

Short answer: you use a command that cannot handle this case.  For the
details, please refer to the bug tracker.

Re: memory leaks

[Madhu]

| Your bug report #43389: 28.0.50; memory leak which was filed against the emacs package, has been closed.

Kudos!

Thanks Trevor and everyone else for tracking this down.  Your hard work
is much appreciated.

The debbugs report #43399 rendered under eww runs to over 100,800 words,
much longer than your average murder mystery; where the clue to the
culprit isn't given till near end.  It is a great relief to see that
justice is finally done and ones faith in humanity is restored.

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Mon, 09 Nov 2020 23:20:55 +0530
> 
> I found test case for this bug and I sent mail to 43395@debbugs.gnu.org,
> and qmail told me that the mail was accepted:
> delivery 1: success:
> 209.51.188.43_accepted_message./Remote_host_said:_250_OK_id=1kc8Bf-00061T-3u/
> But the bugreport didn't show up on gnu.emacs.bug.

That bug is archived, so any messages to it are discarded.  You need
to unarchive it first (see admin/notes/bugtracker for the details).

> Here is the test case:
> 
> #+BEGIN_SRC
> $ cat > f.c
> #include <stdio.h>
> int
> main()
> {
>   char c = ' ';
>   while (c != 'q' && c != 'Q')
>     {
>       fprintf(stdout, "Press q then enter to quit: ");
>       c = fgetc(stdin);
>     }
>   return 0;
> }
> ^D
> 
> $ gcc f.c
> $ emacs -Q
> #+END_SRC
> 
> M-x shell-command ./a.out
> 
> Then the process hangs. But Emacs' memory grows unbounded.
> 
> WARNING. Be careful to interrupt it with ^G before the OOM killer
> kicks in.  If you have swap you may lose control of your machine
> indefinitely.
> 
> After you interrupt the sub process, Emacs is left with unreclaimed
> gigabytes of RSS
> 
> Please let me know if you can reproduce this

This was already discussed in that bug, and you said back then that
when the shell buffer is killed, the memory is freed up.  So how come
you now say this is still a bug, and what is the difference between
the case discussed in that bug and what you describe now?

Re: memory leaks

[Madhu]

* Eli Zaretskii <83ee3ihs17.fsf@gnu.org> :
Wrote on Fri, 04 Mar 2022 13:32:36 +0200:
> This was already discussed in that bug, and you said back then that
> when the shell buffer is killed, the memory is freed up.  So how come
> you now say this is still a bug, and what is the difference between
> the case discussed in that bug and what you describe now?

That message was from nov 2020. It is a duplicate

I onky authorized gmane to send my message this week to emacs-help,
looks like I had a bunch of messages posted through gmane over the years
which were not posted by gmane because i was not authorized, which have
now been delivered, but which were kept in hold. I think in most cases i
posted the messages outside of gmane.

sorry for the noise

I've not seen the memory "leaks" recently. Maybe I stopped using the
packages that triggered them

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Sun, 06 Mar 2022 16:29:41 +0530
> 
> * Eli Zaretskii <83ee3ihs17.fsf@gnu.org> :
> Wrote on Fri, 04 Mar 2022 13:32:36 +0200:
> > This was already discussed in that bug, and you said back then that
> > when the shell buffer is killed, the memory is freed up.  So how come
> > you now say this is still a bug, and what is the difference between
> > the case discussed in that bug and what you describe now?
> 
> That message was from nov 2020. It is a duplicate
> 
> I onky authorized gmane to send my message this week to emacs-help,
> looks like I had a bunch of messages posted through gmane over the years
> which were not posted by gmane because i was not authorized, which have
> now been delivered, but which were kept in hold. I think in most cases i
> posted the messages outside of gmane.
> 
> sorry for the noise
> 
> I've not seen the memory "leaks" recently. Maybe I stopped using the
> packages that triggered them

OK, thanks.  And sorry for not noticing the old date.