Re: master 5c532fe303: Recommend that the user turn off memory overcommit

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Stefan Monnier]

> +@cindex memory trouble, GNU/Linux
> +  On GNU/Linux systems, the system does not normally report running
> +out of memory to Emacs, and can instead randomly kill processes when
> +they run out of memory.  We recommend that you turn this behavior off,
> +so that Emacs can respond correctly when it runs out of memory, by
> +becoming the super user, editing the file @code{/etc/sysctl.conf} to
> +contain the following lines, and then running the command @code{sysctl
> +-p}:

FWIW, I think this is none of Emacs's business.


        Stefan

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> +@cindex memory trouble, GNU/Linux
>> +  On GNU/Linux systems, the system does not normally report running
>> +out of memory to Emacs, and can instead randomly kill processes when
>> +they run out of memory.  We recommend that you turn this behavior off,
>> +so that Emacs can respond correctly when it runs out of memory, by
>> +becoming the super user, editing the file @code{/etc/sysctl.conf} to
>> +contain the following lines, and then running the command @code{sysctl
>> +-p}:
>
> FWIW, I think this is none of Emacs's business.

+1

Recommending to tweak kernel parameters is way over the top for dealing
with system-wide conditions on a way that supposedly is "correct" for
Emacs specifically.

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Po Lu]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> +@cindex memory trouble, GNU/Linux
>>> +  On GNU/Linux systems, the system does not normally report running
>>> +out of memory to Emacs, and can instead randomly kill processes when
>>> +they run out of memory.  We recommend that you turn this behavior off,
>>> +so that Emacs can respond correctly when it runs out of memory, by
>>> +becoming the super user, editing the file @code{/etc/sysctl.conf} to
>>> +contain the following lines, and then running the command @code{sysctl
>>> +-p}:
>>
>> FWIW, I think this is none of Emacs's business.
>
> +1
>
> Recommending to tweak kernel parameters is way over the top for dealing
> with system-wide conditions on a way that supposedly is "correct" for
> Emacs specifically.

Why? We recommend doing just that in many places when a kernel parameter
interferes with the normal behavior of Emacs, especially in
etc/PROBLEMS, with exec-shield and ASLR.

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Brian Cully]

Po Lu <luangruo@yahoo.com> writes:

> Why? We recommend doing just that in many places when a kernel parameter
> interferes with the normal behavior of Emacs, especially in
> etc/PROBLEMS, with exec-shield and ASLR.

	These features prevent Emacs from building, and I’d argue,
constitute evidence of bugs in Emacs, as it’s operation is at odds with
standard and necessary computer security. The wording in etc/PROBLEMS
echoes this sentiment.

	Your proposal crosses a line into how people should administer
their systems even when Emacs functions as expected within the context
of that system. ie, if I’m on Linux, overcommit is standard and I
presumably know how to deal with it or am ignorant of it. In either
event it is not the place of my text editor to tell me that memory
overcommit is bad. There are countless opinion pieces on the web that
will do that job just fine, should I seek them out.

-bjc

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Po Lu]

Brian Cully <bjc@spork.org> writes:

> 	These features prevent Emacs from building, and I’d argue,
> constitute evidence of bugs in Emacs, as it’s operation is at odds with
> standard and necessary computer security. The wording in etc/PROBLEMS
> echoes this sentiment.

Whatever they might constitute, those changes might be required for
Emacs to build correctly.

> 	Your proposal crosses a line into how people should administer
> their systems even when Emacs functions as expected within the context
> of that system. ie, if I’m on Linux, overcommit is standard and I
> presumably know how to deal with it or am ignorant of it. In either
> event it is not the place of my text editor to tell me that memory
> overcommit is bad. There are countless opinion pieces on the web that
> will do that job just fine, should I seek them out.

None of this says that memory overcommit is "bad", it says that it's
detrimental to the correct functioning of Emacs.  (And "correct" here
means memory_full being called before Emacs is killed by the OOM
killer.)

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Stefan Monnier]

> None of this says that memory overcommit is "bad", it says that it's
> detrimental to the correct functioning of Emacs.  (And "correct" here
> means memory_full being called before Emacs is killed by the OOM
> killer.)

But it has that same detrimental effect on all other applications.
So it's not specific to Emacs, which is why I think it's "none of
Emacs's business".

I disagree with Óscar's "way over the top", tho: it's a fairly minor
question, IMO.


        Stefan

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Eli Zaretskii]

> From: Brian Cully <bjc@spork.org>
> Date: Fri, 08 Apr 2022 09:42:44 -0400
> Cc: Óscar Fuentes <ofv@wanadoo.es>, emacs-devel@gnu.org
> 
> 	Your proposal

It isn't his proposal, it is my proposal.

> crosses a line into how people should administer
> their systems even when Emacs functions as expected within the context
> of that system. ie, if I’m on Linux, overcommit is standard and I
> presumably know how to deal with it or am ignorant of it. In either
> event it is not the place of my text editor to tell me that memory
> overcommit is bad. There are countless opinion pieces on the web that
> will do that job just fine, should I seek them out.

The proposal is to explain how overcommit affects Emacs's handling of
out-of-memory situation, and how to avoid that if desired.  It is up
to each user to decide whether they do or don't want to do that.  our
users are grown-ups, or at least we should treat them as such, so they
should be perfectly capable of making this decision.

Whatever you think about the "usual" practice of over-committing, it
basically disables a very important feature of Emacs, which is
described in the user manual, so there's nothing wrong with explaining
to the users why that happens and how to control this OS behavior.

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Óscar Fuentes]

Po Lu <luangruo@yahoo.com> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>> +@cindex memory trouble, GNU/Linux
>>>> +  On GNU/Linux systems, the system does not normally report running
>>>> +out of memory to Emacs, and can instead randomly kill processes when
>>>> +they run out of memory.  We recommend that you turn this behavior off,
>>>> +so that Emacs can respond correctly when it runs out of memory, by
>>>> +becoming the super user, editing the file @code{/etc/sysctl.conf} to
>>>> +contain the following lines, and then running the command @code{sysctl
>>>> +-p}:
>>>
>>> FWIW, I think this is none of Emacs's business.
>>
>> +1
>>
>> Recommending to tweak kernel parameters is way over the top for dealing
>> with system-wide conditions on a way that supposedly is "correct" for
>> Emacs specifically.
>
> Why?

Because what is (supposedly) good for Emacs can be wrong for some other
application which is at least as important as Emacs for the user.

> We recommend doing just that in many places when a kernel parameter
> interferes with the normal behavior of Emacs, especially in
> etc/PROBLEMS, with exec-shield and ASLR.

Those are recommendations to be applied temporarily just when building
Emacs on certain specific (and possibly rare nowadays) scenarios.

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Po Lu]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Because what is (supposedly) good for Emacs can be wrong for some other
> application which is at least as important as Emacs for the user.

So the user can consult the documentation of that program, and decide
which one he wants better.

Seriously, is there an actual example of a program which works well with
overcommit_memory set to 0, but not with it disabled altogether?

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Óscar Fuentes]

Po Lu <luangruo@yahoo.com> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Because what is (supposedly) good for Emacs can be wrong for some other
>> application which is at least as important as Emacs for the user.
>
> So the user can consult the documentation of that program, and decide
> which one he wants better.
>
> Seriously, is there an actual example of a program which works well with
> overcommit_memory set to 0, but not with it disabled altogether?

Dunno. Why the kernel guys are using that default?

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Po Lu <luangruo@yahoo.com>
> Date: Fri, 08 Apr 2022 08:58:44 -0400
> 
> > +@cindex memory trouble, GNU/Linux
> > +  On GNU/Linux systems, the system does not normally report running
> > +out of memory to Emacs, and can instead randomly kill processes when
> > +they run out of memory.  We recommend that you turn this behavior off,
> > +so that Emacs can respond correctly when it runs out of memory, by
> > +becoming the super user, editing the file @code{/etc/sysctl.conf} to
> > +contain the following lines, and then running the command @code{sysctl
> > +-p}:
> 
> FWIW, I think this is none of Emacs's business.

We describe in that node an Emacs feature that basically doesn't work
on GNU systems.  You are in effect saying that we don't care about
that.

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > +@cindex memory trouble, GNU/Linux
  > > +  On GNU/Linux systems, the system does not normally report running
  > > +out of memory to Emacs, and can instead randomly kill processes when
  > > +they run out of memory.  We recommend that you turn this behavior off,
  > > +so that Emacs can respond correctly when it runs out of memory, by
  > > +becoming the super user, editing the file @code{/etc/sysctl.conf} to
  > > +contain the following lines, and then running the command @code{sysctl
  > > +-p}:

  > FWIW, I think this is none of Emacs's business.

The idea of "Emacs's business" is not a coherent concept
when we're thinking about what to say in the Emacs Manual.
The right question is, is this Emacs users' business?

The Emacs Manual is meant to tell Emacs users what they need to know,
and if this is something important for Emacs users to know about, that
is a fine place to tell them.

I don't have an opinion about what is best to do about this.  However,
if a problem means Emacs is likely crash, and we can't fix the problem
in Emacs itself, giving users advice in the manual is good to
consider.

I wonder if it is possible for Emacs to explicitly ask whether it is
getting close to overcommitting.  Even if that approach can't entirely
prevent the problem of surprise death, it might make that problem
happen much less frequently.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Tim Cross]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > +@cindex memory trouble, GNU/Linux
>   > > +  On GNU/Linux systems, the system does not normally report running
>   > > +out of memory to Emacs, and can instead randomly kill processes when
>   > > +they run out of memory.  We recommend that you turn this behavior off,
>   > > +so that Emacs can respond correctly when it runs out of memory, by
>   > > +becoming the super user, editing the file @code{/etc/sysctl.conf} to
>   > > +contain the following lines, and then running the command @code{sysctl
>   > > +-p}:
>
>   > FWIW, I think this is none of Emacs's business.
>
> The idea of "Emacs's business" is not a coherent concept
> when we're thinking about what to say in the Emacs Manual.
> The right question is, is this Emacs users' business?
>
> The Emacs Manual is meant to tell Emacs users what they need to know,
> and if this is something important for Emacs users to know about, that
> is a fine place to tell them.
>
> I don't have an opinion about what is best to do about this.  However,
> if a problem means Emacs is likely crash, and we can't fix the problem
> in Emacs itself, giving users advice in the manual is good to
> consider.
>
> I wonder if it is possible for Emacs to explicitly ask whether it is
> getting close to overcommitting.  Even if that approach can't entirely
> prevent the problem of surprise death, it might make that problem
> happen much less frequently.

I suspect the problem may be the 'we recommend". I'm not sure "we"
should be recommending modifying default system settings just for the
benefit of one application.

Isn't the whole purpose of random killing processes to free up resources
and prevent core services from failing and crashing the system?
If this is the case, recommending turning it off to allow Emacs to
gracefully handle out of memory situations might not be of any benefit
given it will be more likely for the system to crash in these
situations, preventing any action by Emacs.

Might be better to just state what the situation is and one possible
option to consider if it becomes a frequent issue (i.e. Emacs being
killed unexpectedly).

Is this a very common issue? I've run Emacs on some pretty old systems
with little memory and while I've certainly had processes killed, it has
never killed Emacs.

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Phil Sainty]

I keep seeing the choice of process to kill being described as
"random", but it is not random.

The kernel may or may not choose the best process to kill, but
it does choose it for a reason, and I would suggest that the
described criteria is fairly unlikely to describe an Emacs
process (although it's obviously still possible).

The selection process is described in the article that I linked
to previously in a related thread:

https://www.kernel.org/doc/gorman/html/understand/understand016.html

> 13.3 Selecting a Process
> 
> The function select_bad_process() is responsible for choosing a
> process to kill. It decides by stepping through each running task
> and calculating how suitable it is for killing with the function
> badness(). The badness is calculated as follows, note that the
> square roots are integer approximations calculated with
> int_sqrt();
> 
> badness_for_task = total_vm_for_task /
>   (sqrt(cpu_time_in_seconds) * sqrt(sqrt(cpu_time_in_minutes)))
> 
> This has been chosen to select a process that is using a large
> amount of memory but is not that long lived. Processes which have
> been running a long time are unlikely to be the cause of memory
> shortage so this calculation is likely to select a process that
> uses a lot of memory but has not been running long. If the
> process is a root process or has CAP_SYS_ADMIN capabilities, the
> points are divided by four as it is assumed that root privilege
> processes are well behaved. Similarly, if it has CAP_SYS_RAWIO
> capabilities (access to raw devices) privileges, the points are
> further divided by 4 as it is undesirable to kill a process that
> has direct access to hardware.

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Po Lu]

Phil Sainty <psainty@orcon.net.nz> writes:

> I keep seeing the choice of process to kill being described as
> "random", but it is not random.

It is random, from the perspective of the user, who eventually develops
the sinking feeling that the kernel might not kill IceCat but Emacs the
next time it happens.

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Phil Sainty]

On 2022-04-09 21:08, Po Lu wrote:
> It is random, from the perspective of the user, who eventually
> develops the sinking feeling that the kernel might not kill
> IceCat but Emacs the next time it happens.

It's one thing for the user to mistakenly believe that
something is random, and it's quite another thing for our
documentation to explicitly tell them that is the case.

Describing it as "random" helps nobody and is misleading.
(I think that doing so has already misled some of the
people reading this mailing list.)

The process can still be described (in a less-misleading way),
and references to documentation can be made to inform users.

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Achim Gratz]

Po Lu writes:
> Phil Sainty <psainty@orcon.net.nz> writes:
>> I keep seeing the choice of process to kill being described as
>> "random", but it is not random.
>
> It is random, from the perspective of the user, who eventually develops
> the sinking feeling that the kernel might not kill IceCat but Emacs the
> next time it happens.

Random means different things to different people in various languages
and contexts.  From the user perspective I think I would describe the
behaviour of the OOM killer as "unpredictable".  As Phil said, if you
had the same information the kernel uses, the user could in principle
know what process it is going to select, however that data usually is
not available to the user when the OOM killer triggers.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Eli Zaretskii]

> Date: Sat, 09 Apr 2022 20:42:51 +1200
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: luangruo@yahoo.com, Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel@gnu.org
> 
> I keep seeing the choice of process to kill being described as
> "random", but it is not random.
> 
> The kernel may or may not choose the best process to kill, but
> it does choose it for a reason, and I would suggest that the
> described criteria is fairly unlikely to describe an Emacs
> process (although it's obviously still possible).

I've now removed the "random" and "we recommend" parts from the text.

What about using "ulimit -v" -- is that of any help, if the value is
chosen in accordance with the amount of VM available to the system,
like if the value leaves some reasonable amount to the rest of the
system?  If so, perhaps we should mention that as well?

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Phil Sainty]

On 2022-04-09 21:56, Eli Zaretskii wrote:
> I've now removed the "random" and "we recommend" parts from the text.

That's certainly an improvement.  I'd suggest also including the
URL that I referenced previously, which I think gives quite a
clear explanation, and seemed like a fairly official and likely-
to-be-stable URL.

> What about using "ulimit -v" -- is that of any help

For my part, I don't know the answer.

Re: master 5c532fe303: Recommend that the user turn off memory overcommit

[Eli Zaretskii]

> Date: Sat, 09 Apr 2022 22:28:54 +1200
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: luangruo@yahoo.com, emacs-devel@gnu.org, rms@gnu.org,
>  monnier@iro.umontreal.ca
> 
> On 2022-04-09 21:56, Eli Zaretskii wrote:
> > I've now removed the "random" and "we recommend" parts from the text.
> 
> That's certainly an improvement.  I'd suggest also including the
> URL that I referenced previously, which I think gives quite a
> clear explanation, and seemed like a fairly official and likely-
> to-be-stable URL.

I mentioned the name of the Linux feature so that people could easily
find information about it by searching the net.  I think including the
URL there is too much.