bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Boruch Baum]

Version: emacs 25.2, debian

As emacs begins to exhaust memory, it begins to grind all system process's
performance to a halt, which is clearly undesirable. If the user notices
in time, either because the user is attentive to the mode line or the
system slow-down, then the user will have an opportunity to gracefully
exit and restart emacs, but such is not always the case, and should not
be presumed to always be the case.

I would like to suggest a optional feature to automatically save
buffers and shut down emacs / emacs-server when this condition occurs.
The user could toggle its operation by manipulating a boolean variable.

It's default condition is for the emacs developers to decide. My vote
would be for default on, in the interest of the greater good of
preserving an entire system from grinding to a halt.

I reliably encountered memory exhaustion when testing function
`apropos-value', which creates for me that condition within five
calls.

An alternate solution, external to emacs, would be to use the shell's
`ulimit' feature, but I don't know whether this is available on all
platforms (eg. Microsoft), and would require constant tinkering of the
memory threshold limit to account for any change in requirements in the
sum of all the other system processes.

-- 
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Andreas Schwab]

On Apr 08 2018, Boruch Baum <boruch_baum@gmx.com> wrote:

> I would like to suggest a optional feature to automatically save
> buffers and shut down emacs / emacs-server when this condition occurs.

How do you detect that condition?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Boruch Baum]

On 2018-04-08 09:51, Andreas Schwab wrote:
> How do you detect that condition?

Emacs already monitors the condition, and currently performs the following:

1] Adds a memory warning to the mode line;

2] Warns the user with a minibuffer message, along with the suggestion
to save the current buffer and exit emacs.

The proposed feature request would optionally change that behavior.

-- 
hkp://keys.gnupg.net CA45 09B5 5351 7C11 A9D1 7286 0036 9E45 1595 8BC0

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Andreas Schwab]

On Apr 08 2018, Boruch Baum <boruch_baum@gmx.com> wrote:

> On 2018-04-08 09:51, Andreas Schwab wrote:
>> How do you detect that condition?
>
> Emacs already monitors the condition, and currently performs the following:

It monitors its virtual memory.  But shortage of physical memory is more
difficult to detect.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Boruch Baum]

On 2018-04-08 10:36, Andreas Schwab wrote:
> It monitors its virtual memory.  But shortage of physical memory is more
> difficult to detect.

Virtual memory would be the relevant metric here.

-- 
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Andreas Schwab]

On Apr 08 2018, Boruch Baum <boruch_baum@gmx.com> wrote:

> On 2018-04-08 10:36, Andreas Schwab wrote:
>> It monitors its virtual memory.  But shortage of physical memory is more
>> difficult to detect.
>
> Virtual memory would be the relevant metric here.

But physical memory will be exhausted long before virtual memory is.
And it may not even be Emacs which exhausted it.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Boruch Baum]

On 2018-04-08 13:24, Andreas Schwab wrote:
> But physical memory will be exhausted long before virtual memory is.
> And it may not even be Emacs which exhausted it.

Before you commit to what you wrote, why not try it yourself first; it's
easy to do, and you will see for yourself whether what you wrote is
true. Start from your working copy of emacs, not a test instance, so
that emacs will have accumulated a nice quantity of data to populate its
various system *-ring and *-history variables, and perform apropos-value
on a common string. Repeat as necessary and see for yourself whether
what you say is true.

-- 
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Andreas Schwab]

On Apr 08 2018, Boruch Baum <boruch_baum@gmx.com> wrote:

> On 2018-04-08 13:24, Andreas Schwab wrote:
>> But physical memory will be exhausted long before virtual memory is.
>> And it may not even be Emacs which exhausted it.
>
> Before you commit to what you wrote, why not try it yourself first;

How is that relevant?  That statement is true for every process.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Boruch Baum]

On 2018-04-08 18:10, Andreas Schwab wrote:
> On Apr 08 2018, Boruch Baum <boruch_baum@gmx.com> wrote:
>
> > On 2018-04-08 13:24, Andreas Schwab wrote:
> >> But physical memory will be exhausted long before virtual memory is.
> >> And it may not even be Emacs which exhausted it.
> >
> > Before you commit to what you wrote, why not try it yourself first;
>
> How is that relevant?  That statement is true for every process.

It's relevant because its reality, and doing so might force you to
confront a reality at odds with your statements.

-- 
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Andreas Schwab]

On Apr 08 2018, Boruch Baum <boruch_baum@gmx.com> wrote:

> It's relevant because its reality, and doing so might force you to
> confront a reality at odds with your statements.

It isn't.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

>> I would like to suggest a optional feature to automatically save
>> buffers and shut down emacs / emacs-server when this condition occurs.
>
> How do you detect that condition?

I don't think there's anything generally useful Emacs can do here to
detect the situation -- Emacs can't know whether the machine is running
out of memory one way or another.  (For instance, it may be in a VM and
the host is running out of memory.)

So I don't think there'll be any progress in this bug report, and I'm
therefore closing it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Eli Zaretskii]

> Date: Sun, 8 Apr 2018 03:22:23 -0400
> From: Boruch Baum <boruch_baum@gmx.com>
> 
> I would like to suggest a optional feature to automatically save
> buffers and shut down emacs / emacs-server when this condition occurs.
> The user could toggle its operation by manipulating a boolean variable.

FWIW, I consider automatic shutdown in this case a misfeature, because
this is likely to shut down a session due to some memory-related bugs
more than anything else.  But if others would like that, I won't block
adding it as an opt-in option.

I do agree with Andreas that normally a system will start paging long
before the virtual memory threshold for that warning is exceeded.

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Boruch Baum]

On 2018-04-08 15:58, Eli Zaretskii wrote:
> FWIW, I consider automatic shutdown in this case a misfeature, because
> this is likely to shut down a session due to some memory-related bugs
> more than anything else.

You would want an emacs bug to bring down your entire system to just
short of a complete freeze (my experience of today / last night)?

>  But if others would like that, I won't block adding it as an opt-in
> option.

> I do agree with Andreas that normally a system will start paging long
> before the virtual memory threshold for that warning is exceeded.

Not my experience today / last night.

-- 
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Eli Zaretskii]

> Date: Sun, 8 Apr 2018 09:21:24 -0400
> From: Boruch Baum <boruch_baum@gmx.com>
> Cc: 31091@debbugs.gnu.org
> 
> On 2018-04-08 15:58, Eli Zaretskii wrote:
> > FWIW, I consider automatic shutdown in this case a misfeature, because
> > this is likely to shut down a session due to some memory-related bugs
> > more than anything else.
> 
> You would want an emacs bug to bring down your entire system to just
> short of a complete freeze (my experience of today / last night)?

You would want to introduce a new non-trivial feature based on a
single data point, which is a bug that most probably was solved many
moons ago.

> > I do agree with Andreas that normally a system will start paging long
> > before the virtual memory threshold for that warning is exceeded.
> 
> Not my experience today / last night.

You are experiencing a particular nasty bug.

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Andreas Schwab]

On Apr 08 2018, Eli Zaretskii <eliz@gnu.org> wrote:

>> Not my experience today / last night.
>
> You are experiencing a particular nasty bug.

The only bug is that apropos-value uses a sub-optimal implementation
that produces a lot of data.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Noam Postavsky]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Apr 08 2018, Eli Zaretskii <eliz@gnu.org> wrote:
>
>>> Not my experience today / last night.
>>
>> You are experiencing a particular nasty bug.
>
> The only bug is that apropos-value uses a sub-optimal implementation
> that produces a lot of data.

No, it's probably Bug#26952, as I mentioned earlier.

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Boruch Baum <boruch_baum@gmx.com>,  31091@debbugs.gnu.org
> Date: Sun, 08 Apr 2018 18:21:33 +0200
> 
> > You are experiencing a particular nasty bug.
> 
> The only bug is that apropos-value uses a sub-optimal implementation
> that produces a lot of data.

No, I meant memory-allocation trouble in Emacs 25.x on some versions
of GNU/Linux.

The same command doesn't get me anywhere near the memory-full
threshold in later versions of Emacs.

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Noam Postavsky]

Boruch Baum <boruch_baum@gmx.com> writes:

> As emacs begins to exhaust memory, it begins to grind all system process's
> performance to a halt, which is clearly undesirable. If the user notices
> in time, either because the user is attentive to the mode line or the
> system slow-down, then the user will have an opportunity to gracefully
> exit and restart emacs, but such is not always the case, and should not
> be presumed to always be the case.

If you set the virtual memory limit to 1 or 2 GB (or something
proportional to the amount of physical memory you have) by doing

    ulimit -Sv $((1000 * 1024))

before you start Emacs, instead of allocating until system slow-down,
Emacs should stop with a message:

    Memory exhausted--use C-x s then exit and restart Emacs

Possibly also a warning like

    Emergency (alloc): Warning: past 75% of memory limit

depending on how quickly you approach the limit.

(yes, I've tried this, see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26952#18)

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Boruch Baum]

On 2018-04-08 09:25, Noam Postavsky wrote:
> If you set the virtual memory limit to 1 or 2 GB (or something
> proportional to the amount of physical memory you have) by doing
>
>     ulimit -Sv $((1000 * 1024))
>
> ...

Yes, I mentioned that alternative in the final paragraph of my initial
email. Good to hear that you can report it works; it looks like that
will have to be my work-around until debian packages emacs26. Thanks.

-- 
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Noam Postavsky]

Boruch Baum <boruch_baum@gmx.com> writes:

> in time, either because the user is attentive to the mode line or the

>> How do you detect that condition?
>
> Emacs already monitors the condition, and currently performs the following:
>
> 1] Adds a memory warning to the mode line;

I'm not sure what mode line warning you are talking about.  But, as
Andreas says, and can be shown by experimenting with 'ulimit', Emacs
measures usage of virtual memory.  Physical memory usage is subject the
allocation algorithms of the OS, and can't be reliably measured.

So I don't think this feature request can be satisfied.

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Boruch Baum]

On 2018-04-08 11:27, Noam Postavsky wrote:
> Boruch Baum <boruch_baum@gmx.com> writes:
> > Emacs already monitors the condition, and currently performs the following:
> >
> > 1] Adds a memory warning to the mode line;
>
> I'm not sure what mode line warning you are talking about.

    mode-line-front-space is a variable defined in ‘bindings.el’.
    Its value is (:eval (if (display-graphic-p) " " "-"))

      This variable may be risky if used as a file-local variable.

    Documentation:
    Mode line construct to put at the front of the mode line.
    By default, this construct is displayed right at the beginning of
    the mode line, except that if there is a memory-full message, it
    is displayed first.

> So I don't think this feature request can be satisfied.

The same logic that currently issues the warning messages would perform
the shutdown if a variable were set.


-- 
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Andreas Schwab]

On Apr 08 2018, Boruch Baum <boruch_baum@gmx.com> wrote:

> The same logic that currently issues the warning messages would perform
> the shutdown if a variable were set.

That logic cannot predict the shortage of a systemwide shared resource.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Noam Postavsky]

Boruch Baum <boruch_baum@gmx.com> writes:

> On 2018-04-08 11:27, Noam Postavsky wrote:
>> Boruch Baum <boruch_baum@gmx.com> writes:
>> > Emacs already monitors the condition, and currently performs the following:
>> >
>> > 1] Adds a memory warning to the mode line;
>>
>> I'm not sure what mode line warning you are talking about.
>

>     By default, this construct is displayed right at the beginning of
>     the mode line, except that if there is a memory-full message, it
>     is displayed first.

Ah, looks like that warning is non-operational in most configurations though:

    static const char *
    decode_mode_spec (struct window *w, register int c, int field_width,
              Lisp_Object *string)
    {
      [...]
        case 'e':
    #if !defined SYSTEM_MALLOC && !defined HYBRID_MALLOC
          {
        if (NILP (Vmemory_full))
          return "";
        else
          return "!MEM FULL! ";
          }
    #else
          return "";
    #endif


>> So I don't think this feature request can be satisfied.
>
> The same logic that currently issues the warning messages would perform
> the shutdown if a variable were set.

Hmm, I guess I'm not sure what advantage an abrubt shutdown has over a
warning.  And as far as I understand, you never saw the warnings
messages, so it wouldn't have helped in your case anyway, since you
didn't hit the virtual memory limit.

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Boruch Baum]

On 2018-04-08 12:30, Noam Postavsky wrote:
> Hmm, I guess I'm not sure what advantage an abrubt shutdown has over a
> warning.

The most useful case would be of a user is running a time-consuming
operation or an asynchronous operation, maybe a long repetitive macro,
for which the current warning would be generated at near the beginning
of the operation and system freeze would happen in mid-operation.

>  And as far as I understand, you never saw the warnings
> messages,

I did see the mini-buffer advice and warning, and ignored it, and I did
see the mode line warning, and ignored it. Added to the long list of
things I've done -- once.

-- 
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Noam Postavsky]

Boruch Baum <boruch_baum@gmx.com> writes:

> On 2018-04-08 12:30, Noam Postavsky wrote:
>> Hmm, I guess I'm not sure what advantage an abrubt shutdown has over a
>> warning.
>
> The most useful case would be of a user is running a time-consuming
> operation or an asynchronous operation, maybe a long repetitive macro,
> for which the current warning would be generated at near the beginning
> of the operation and system freeze would happen in mid-operation.

The final out-of-memory message comes via `signal', so any operation
would be interrupted.  Although the almost-out-of-memory warnings just
use display-warnings, so they don't interrupt anything.

>>  And as far as I understand, you never saw the warnings
>> messages,
>
> I did see the mini-buffer advice and warning, and ignored it, and I did
> see the mode line warning, and ignored it. Added to the long list of
> things I've done -- once.

Hmm, I'm surprised you actually hit the warning.  Are you on a 32 bit
system with about 4GB RAM?  I guess you could hit the 75% warning at 3GB
allocation, and then start swapping soon after, if other processes are
using about 1GB RAM.

bug#31091: [OFFLIST] Re: bug#31091: 25.2: Feature Request: auto exit upon memory exhaustion warning

[Noam Postavsky]

On 8 April 2018 at 14:10, Boruch Baum <boruch_baum@gmx.com> wrote:
> On 2018-04-08 13:26, Noam Postavsky wrote:
>> Hmm, I'm surprised you actually hit the warning.  Are you on a 32 bit
>> system with about 4GB RAM?  I guess you could hit the 75% warning at 3GB
>> allocation, and then start swapping soon after, if other processes are
>> using about 1GB RAM.
>
> 64-bit 4GB RAM

Huh. I have a similar machine, and I never saw the warning until I
started using ulimit. I can't understand how you hit it.