Re: memory leaks

Re: memory leaks

[Madhu]

[Test case follows. Sorry Drew for trying to post to help-gnu and
debbugs simultaneously, but this is my third attempt at sending this
test case - I've tried sending it to debbugs.gnu.org and to
gmane.emacs.help newsgroup - and my messages have been dropped]


* Madhu <m35z8gise6.fsf@leonis4.robolove.meer.net> :
Wrote on Mon, 14 Sep 2020 15:36:57 +0530:
> The numbers I reported by garbage-collect do not account for the memory
> usage. FWIW I've posted that info again at
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43395

I found test case for this bug and I sent mail to 43395@debbugs.gnu.org,
and qmail told me that the mail was accepted:
delivery 1: success:
209.51.188.43_accepted_message./Remote_host_said:_250_OK_id=1kc8Bf-00061T-3u/
But the bugreport didn't show up on gnu.emacs.bug.

Here is the test case:

#+BEGIN_SRC
$ cat > f.c
#include <stdio.h>
int
main()
{
  char c = ' ';
  while (c != 'q' && c != 'Q')
    {
      fprintf(stdout, "Press q then enter to quit: ");
      c = fgetc(stdin);
    }
  return 0;
}
^D

$ gcc f.c
$ emacs -Q
#+END_SRC

M-x shell-command ./a.out

Then the process hangs. But Emacs' memory grows unbounded.

WARNING. Be careful to interrupt it with ^G before the OOM killer
kicks in.  If you have swap you may lose control of your machine
indefinitely.

After you interrupt the sub process, Emacs is left with unreclaimed
gigabytes of RSS

Please let me know if you can reproduce this

bug#43395: memory leaks

[Jean Louis]

* Madhu <enometh@meer.net> [2020-11-10 04:27]:
> [Test case follows. Sorry Drew for trying to post to help-gnu and
> debbugs simultaneously, but this is my third attempt at sending this
> test case - I've tried sending it to debbugs.gnu.org and to
> gmane.emacs.help newsgroup - and my messages have been dropped]
> 
> 
> * Madhu <m35z8gise6.fsf@leonis4.robolove.meer.net> :
> Wrote on Mon, 14 Sep 2020 15:36:57 +0530:
> > The numbers I reported by garbage-collect do not account for the memory
> > usage. FWIW I've posted that info again at
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43395
> 
> I found test case for this bug and I sent mail to 43395@debbugs.gnu.org,
> and qmail told me that the mail was accepted:
> delivery 1: success:
> 209.51.188.43_accepted_message./Remote_host_said:_250_OK_id=1kc8Bf-00061T-3u/
> But the bugreport didn't show up on gnu.emacs.bug.
> 
> Here is the test case:
> 
> #+BEGIN_SRC
> $ cat > f.c
> #include <stdio.h>
> int
> main()
> {
>   char c = ' ';
>   while (c != 'q' && c != 'Q')
>     {
>       fprintf(stdout, "Press q then enter to quit: ");
>       c = fgetc(stdin);
>     }
>   return 0;
> }
> ^D
> 
> $ gcc f.c
> $ emacs -Q
> #+END_SRC
> 
> M-x shell-command ./a.out
> 
> Then the process hangs. But Emacs' memory grows unbounded.
> 
> WARNING. Be careful to interrupt it with ^G before the OOM killer
> kicks in.  If you have swap you may lose control of your machine
> indefinitely.
> 
> After you interrupt the sub process, Emacs is left with unreclaimed
> gigabytes of RSS
> 
> Please let me know if you can reproduce this

Me I have been testing now. It all blocks. Then I had to switch to
console from X to kill process.

I can see memory is not reclaimed graphically. Not practically. I have
graphical indicator.

This same things happens from time to time in Emacs. Exactly like you
say when I have swap I may lose control of machine. Then I had to
power off with the hard key. Bad.

I hope this can be resolved.

bug#43395: memory leaks

[Eli Zaretskii]

> Date: Tue, 10 Nov 2020 17:22:46 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: 43395@debbugs.gnu.org
> 
> > #+BEGIN_SRC
> > $ cat > f.c
> > #include <stdio.h>
> > int
> > main()
> > {
> >   char c = ' ';
> >   while (c != 'q' && c != 'Q')
> >     {
> >       fprintf(stdout, "Press q then enter to quit: ");
> >       c = fgetc(stdin);
> >     }
> >   return 0;
> > }
> > ^D
> > 
> > $ gcc f.c
> > $ emacs -Q
> > #+END_SRC
> > 
> > M-x shell-command ./a.out
> > 
> > Then the process hangs. But Emacs' memory grows unbounded.

This is expected, and is not a bug.

'shell-command' is intended for running non-interactive programs,
those which produce output, but don't expect any input.  So it runs
the sub-process with its standard input connected to the null device.
Your program has a bug in that case: it doesn't detect the EOF
condition on its standard input (to see that, invoke it as
"./a.out < /dev/null").  So it loops indefinitely, with each iteration
pumping the prompts into Emacs, which obediently collects that in a
buffer, that grows and grows and grows, until it eats up all the
available memory.  And evidently, you didn't configure your system to
have resident size limitation on user processes, so instead of
reporting "memory full", Emacs is allowed to eat up all your memory
and swap.

In short: don't do that.

I see no bug here: Emacs cannot know in advance how much stuff will be
emitted by the sub-process, and it cannot know how much memory it cqan
swallow before OOMK will sporing into action.

And, of course, this is unrelated to the problems being discussed in
this bug report.

bug#43395: memory leaks

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2020-11-10 18:28]:
> > > M-x shell-command ./a.out
> > > 
> > > Then the process hangs. But Emacs' memory grows unbounded.
> 
> This is expected, and is not a bug.
> 
> 'shell-command' is intended for running non-interactive programs,
> those which produce output, but don't expect any input.  So it runs
> the sub-process with its standard input connected to the null device.
> Your program has a bug in that case: it doesn't detect the EOF
> condition on its standard input (to see that, invoke it as
> "./a.out < /dev/null").  So it loops indefinitely, with each iteration
> pumping the prompts into Emacs, which obediently collects that in a
> buffer, that grows and grows and grows, until it eats up all the
> available memory.  And evidently, you didn't configure your system to
> have resident size limitation on user processes, so instead of
> reporting "memory full", Emacs is allowed to eat up all your memory
> and swap.

Thank you, it helps!

Do I need to use `ulimit -m'?

I guess following should be for 1 GiB:
ulimit -m 1048576

and this for 2 GiB:
ulimit -m 2097152

Now I can execute ./a.out without getting blocked in Emacs. I will see
how to tweak that more.

bug#43395: memory leaks

[Madhu]

*  Eli Zaretskii <eliz@gnu.org> <83k0ut2pv6.fsf@gnu.org>
Wrote on Tue, 10 Nov 2020 17:28:13 +0200

> This is expected, and is not a bug.
>
> 'shell-command' is intended for running non-interactive programs,
> those which produce output, but don't expect any input.  So it runs
> the sub-process with its standard input connected to the null device.
> Your program has a bug in that case: it doesn't detect the EOF
> condition on its standard input (to see that, invoke it as
> "./a.out < /dev/null").  So it loops indefinitely, with each iteration
> pumping the prompts into Emacs, which obediently collects that in a
> buffer, that grows and grows and grows, until it eats up all the
> available memory.  And evidently, you didn't configure your system to
> have resident size limitation on user processes, so instead of
> reporting "memory full", Emacs is allowed to eat up all your memory
> and swap.
>
> In short: don't do that.
>
> I see no bug here: Emacs cannot know in advance how much stuff will be
> emitted by the sub-process, and it cannot know how much memory it cqan
> swallow before OOMK will sporing into action.
>
> And, of course, this is unrelated to the problems being discussed in
> this bug report.

Sorry, yes - the memory is indeed freed up when *Shell Command Output*
is killed.  I was mistaken in thinking that the memory was not freed
up when all the buffers had been deleted.

Apologies.

Re: memory leaks

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Tue, 10 Nov 2020 06:54:56 +0530
> Cc: 43395@debbugs.gnu.org
> 
> #+BEGIN_SRC
> $ cat > f.c
> #include <stdio.h>
> int
> main()
> {
>   char c = ' ';
>   while (c != 'q' && c != 'Q')
>     {
>       fprintf(stdout, "Press q then enter to quit: ");
>       c = fgetc(stdin);
>     }
>   return 0;
> }
> ^D
> 
> $ gcc f.c
> $ emacs -Q
> #+END_SRC
> 
> M-x shell-command ./a.out
> 
> Then the process hangs. But Emacs' memory grows unbounded.

Short answer: you use a command that cannot handle this case.  For the
details, please refer to the bug tracker.