warn-maybe-out-of-memory

warn-maybe-out-of-memory

[Eli Zaretskii]

I wonder if this function should only warn when it is called from
commands invoked by the user, as opposed to from a Lisp program.  The
warning is in find-file-noselect, which AFAIK is widely used in Lisp
programs, where displaying this warning might be inappropriate.

In addition, the function assumes that visiting a file of size N bytes
needs N bytes of memory, which is false: we need more, sometimes much
more.

Re: warn-maybe-out-of-memory

[Eli Zaretskii]

> Date: Thu, 10 Jul 2014 21:23:32 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> I wonder if this function should only warn when it is called from
> commands invoked by the user, as opposed to from a Lisp program.  The
> warning is in find-file-noselect, which AFAIK is widely used in Lisp
> programs, where displaying this warning might be inappropriate.
> 
> In addition, the function assumes that visiting a file of size N bytes
> needs N bytes of memory, which is false: we need more, sometimes much
> more.

Also, is this call to emacs_abort really appropriate?  Or is it some
remnant from debugging this code?

  DEFUN ("memory-info", Fmemory_info, Smemory_info, 0, 0, 0,
	 doc: /* Return a list of (TOTAL-RAM FREE-RAM TOTAL-SWAP FREE-SWAP).
  All values are in Kbytes.  If there is no swap space, last two
  values are zero.  If the system is not supported, return nil.  */)
    (void)
  {
  #ifdef HAVE_LINUX_SYSINFO
    struct sysinfo si;
    uintmax_t units;

    if (sysinfo (&si))
      emacs_abort ();  <<<<<<<<<<<<<<<<<<<<<<

Re: warn-maybe-out-of-memory

[Dmitry Antipov]

On 07/10/2014 10:47 PM, Eli Zaretskii wrote:

>> I wonder if this function should only warn when it is called from
>> commands invoked by the user, as opposed to from a Lisp program.  The
>> warning is in find-file-noselect, which AFAIK is widely used in Lisp
>> programs, where displaying this warning might be inappropriate.

Hm, find-file-noselect also may ask for confirmation to open large file.
If this is undesirable, shouldn't we move all user interaction to top-level
find-file?

>> In addition, the function assumes that visiting a file of size N bytes
>> needs N bytes of memory, which is false: we need more, sometimes much
>> more.

No.  This function assumes that visiting a file of size N bytes needs
at least N bytes of memory, and warns if we have even less than N.

> Also, is this call to emacs_abort really appropriate?  Or is it some
> remnant from debugging this code?
>
>      if (sysinfo (&si))
>        emacs_abort ();  <<<<<<<<<<<<<<<<<<<<<<

Here emacs_abort may be called only if &SI points outside of a process'
address space.  This is possible only if C stack is smashed and so you
have no way to continue anyway.

Dmitry

Re: warn-maybe-out-of-memory

[Eli Zaretskii]

> Date: Fri, 11 Jul 2014 08:42:23 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> CC: emacs-devel@gnu.org
> 
> On 07/10/2014 10:47 PM, Eli Zaretskii wrote:
> 
> >> I wonder if this function should only warn when it is called from
> >> commands invoked by the user, as opposed to from a Lisp program.  The
> >> warning is in find-file-noselect, which AFAIK is widely used in Lisp
> >> programs, where displaying this warning might be inappropriate.
> 
> Hm, find-file-noselect also may ask for confirmation to open large file.
> If this is undesirable, shouldn't we move all user interaction to top-level
> find-file?

Maybe.

> >> In addition, the function assumes that visiting a file of size N bytes
> >> needs N bytes of memory, which is false: we need more, sometimes much
> >> more.
> 
> No.  This function assumes that visiting a file of size N bytes needs
> at least N bytes of memory, and warns if we have even less than N.

But if this warning is to be useful, it should catch the majority of
the cases, otherwise we are just wasting cycles.  With the current
test, many cases of visiting files that cannot be accommodated will go
undetected, thus rendering all this feature useless.

Since it's only a warning that doesn't prevent visiting a file, it is
better to err on the other side of the truth.  E.g., apply some
heuristics as to the average expansion factor, perhaps dependent on
the locale.  And don't forget that even for plain-ASCII files we
allocate the gap in addition to the text itself, so the mere size of
the file is simply _never_ the correct value, it is always an
underestimation.  IOW, this test is always biased towards lower
values.

If we don't do something like that, then I wonder what exactly is the
use case that benefits from this warning?

> > Also, is this call to emacs_abort really appropriate?  Or is it some
> > remnant from debugging this code?
> >
> >      if (sysinfo (&si))
> >        emacs_abort ();  <<<<<<<<<<<<<<<<<<<<<<
> 
> Here emacs_abort may be called only if &SI points outside of a process'
> address space.  This is possible only if C stack is smashed and so you
> have no way to continue anyway.

It is IMO inappropriate for a minor feature like this one to abort
Emacs.  Who knows what the Linux kernel developers will introduce
tomorrow that could possibly fail the function?  It is none of this
feature's business to be the place where C stack smashing is detected.
So my vote is for returning nil and perhaps displaying a warning.

Re: warn-maybe-out-of-memory

[Dmitry Antipov]

On 07/11/2014 10:50 AM, Eli Zaretskii wrote:

> But if this warning is to be useful, it should catch the majority of
> the cases, otherwise we are just wasting cycles.  With the current
> test, many cases of visiting files that cannot be accommodated will go
> undetected, thus rendering all this feature useless.

With this feature, I'm just trying to redirect user to `find-file-literally'
if file looks so large that `find-file-noselect' is likely to run out of
memory.  What other cases do you mean?

> Since it's only a warning that doesn't prevent visiting a file, it is
> better to err on the other side of the truth.  E.g., apply some
> heuristics as to the average expansion factor, perhaps dependent on
> the locale.

IIUC there are no reliable heuristics here.  For example, produce_charset
and compose_text may allocate substantial amount of memory to represent
`composition' and `charset' extra properties, and we need to read and
decode everything to find all places where such an extra properties
should be attached.

> And don't forget that even for plain-ASCII files we
> allocate the gap in addition to the text itself, so the mere size of
> the file is simply _never_ the correct value, it is always an
> underestimation.  IOW, this test is always biased towards lower
> values.

Unless someone still uses a machine with 64K RAM, initial gap size
is very small in comparison with file sizes we're talking about.

> If we don't do something like that, then I wonder what exactly is the
> use case that benefits from this warning?

We don't need to do "something like that", i.e. we don't need a precise
estimation.  OSes are tricky about memory management; overall VM picture
may drastically change when file reading is in progress, etc.  With a lot
of constraints you can't predict, estimate and control, precise estimation
just makes no sense.

> It is IMO inappropriate for a minor feature like this one to abort
> Emacs.  Who knows what the Linux kernel developers will introduce
> tomorrow that could possibly fail the function?  It is none of this
> feature's business to be the place where C stack smashing is detected.
> So my vote is for returning nil and perhaps displaying a warning.

OK.

Dmitry

Re: warn-maybe-out-of-memory

[Eli Zaretskii]

> Date: Fri, 11 Jul 2014 12:43:39 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> CC: emacs-devel@gnu.org
> 
> On 07/11/2014 10:50 AM, Eli Zaretskii wrote:
> 
> > But if this warning is to be useful, it should catch the majority of
> > the cases, otherwise we are just wasting cycles.  With the current
> > test, many cases of visiting files that cannot be accommodated will go
> > undetected, thus rendering all this feature useless.
> 
> With this feature, I'm just trying to redirect user to `find-file-literally'
> if file looks so large that `find-file-noselect' is likely to run out of
> memory.  What other cases do you mean?

I mean the cases where the file size is borderline, near the available
memory, but slightly less than that.

> > Since it's only a warning that doesn't prevent visiting a file, it is
> > better to err on the other side of the truth.  E.g., apply some
> > heuristics as to the average expansion factor, perhaps dependent on
> > the locale.
> 
> IIUC there are no reliable heuristics here.

Heuristics doesn't have to be reliable, just plausible.  Erring on the
"safe side", which in this case means emitting the warning even when
the file is actually not too large, is OK -- it's only a warning.

> For example, produce_charset and compose_text may allocate
> substantial amount of memory to represent `composition' and
> `charset' extra properties

Then by all means add that memory to the estimation, at least in
locales that frequently do that.  Or maybe even always.

> and we need to read and decode everything to find all places where
> such an extra properties should be attached.

That's not the intent, granted.

> > And don't forget that even for plain-ASCII files we
> > allocate the gap in addition to the text itself, so the mere size of
> > the file is simply _never_ the correct value, it is always an
> > underestimation.  IOW, this test is always biased towards lower
> > values.
> 
> Unless someone still uses a machine with 64K RAM, initial gap size
> is very small in comparison with file sizes we're talking about.

It is indeed very small, but it can easily cause the needed memory to
cross the border.  Adding it will catch these borderline cases, which
AFAIU is the purpose of this feature.

> We don't need to do "something like that", i.e. we don't need a precise
> estimation.  OSes are tricky about memory management; overall VM picture
> may drastically change when file reading is in progress, etc.  With a lot
> of constraints you can't predict, estimate and control, precise estimation
> just makes no sense.

Then how does this feature make sense?  It is, according to you,
unpredictable and uncontrollable.

Re: warn-maybe-out-of-memory

[Dmitry Antipov]

On 07/11/2014 01:02 PM, Eli Zaretskii wrote:

> I mean the cases where the file size is borderline, near the available
> memory, but slightly less than that.

We may warn if file size reaches or exceeds, say, 90% of available memory.

> Then how does this feature make sense?  It is, according to you,
> unpredictable and uncontrollable.

This depends on OS and VM pressure. For example, on GNU/Linux if I have
just slightly above 8G free:

$ free
              total       used       free     shared    buffers     cached
Mem:      16127204    7762072    8365132      68248      84396    6401276
-/+ buffers/cache:    1276400   14850804

and asks for 10G, the kernel may shrink page cache by 2G and satisfy
10G mmap request, thus fooling the logic I'm using to issue the warning.
But under some circumstances, this may be not so; in short, I think that
we need to support more OSes and gather more feedback from users.

Dmitry

Re: warn-maybe-out-of-memory

[Eli Zaretskii]

> Date: Fri, 11 Jul 2014 13:43:31 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> CC: emacs-devel@gnu.org
> 
> On 07/11/2014 01:02 PM, Eli Zaretskii wrote:
> 
> > I mean the cases where the file size is borderline, near the available
> > memory, but slightly less than that.
> 
> We may warn if file size reaches or exceeds, say, 90% of available memory.

That's a good start.  But decoding non-ASCII text could expand the
size by a factor of 2 in many cases, so perhaps 50% is a better
threshold, at least in non-English locales?

> > Then how does this feature make sense?  It is, according to you,
> > unpredictable and uncontrollable.
> 
> This depends on OS and VM pressure. For example, on GNU/Linux if I have
> just slightly above 8G free:
> 
> $ free
>               total       used       free     shared    buffers     cached
> Mem:      16127204    7762072    8365132      68248      84396    6401276
> -/+ buffers/cache:    1276400   14850804
> 
> and asks for 10G, the kernel may shrink page cache by 2G and satisfy
> 10G mmap request, thus fooling the logic I'm using to issue the warning.

On Windows, the swap file is enlarged in this situation.

> But under some circumstances, this may be not so; in short, I think that
> we need to support more OSes and gather more feedback from users.

Agreed.

Btw, this feature is all but useless for 32-bit builds on any modern
machine, since the buffer size limitation almost always hits much
sooner.

Re: warn-maybe-out-of-memory

[Eli Zaretskii]

Btw, memory-info should be mentioned in NEWS.

Re: warn-maybe-out-of-memory

[Dmitry Antipov]

On 07/11/2014 02:00 PM, Eli Zaretskii wrote:

> That's a good start.  But decoding non-ASCII text could expand the
> size by a factor of 2 in many cases, so perhaps 50% is a better
> threshold, at least in non-English locales?

Maybe.  But I don't understand your conviction about a meaningful correlation
between locale settings and contents of arbitrary file.  Imagine, say, an
English <-> Japanese translator; she might set the locale to match her origins
but has two working sets of texts in both languages :-).

Dmitry

Re: warn-maybe-out-of-memory

[Eli Zaretskii]

> Date: Fri, 11 Jul 2014 14:34:35 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> CC: emacs-devel@gnu.org
> 
> I don't understand your conviction about a meaningful correlation
> between locale settings and contents of arbitrary file.  Imagine, say, an
> English <-> Japanese translator; she might set the locale to match her origins
> but has two working sets of texts in both languages :-).

It's a heuristic, and as such is imperfect.  If you have a huge
document in Japanese or in pahawh-hmong in an otherwise English
locale, you lose.

Re: warn-maybe-out-of-memory

[Stefan Monnier]

I'm actually wondering what is the use case.  Concretely.

"If file looks so large that `find-file-noselect' is likely to run out of
memory" is not very concrete.  In what kind of situation could this happen?
Especially:
- in which situation is could this be useful given that we
  already have large-file-warning-threshold?
- would using find-file-literally solve the problem?

> This depends on OS and VM pressure. For example, on GNU/Linux if I have
> just slightly above 8G free:

> $ free
>              total       used       free     shared    buffers     cached
> Mem:      16127204    7762072    8365132      68248      84396    6401276
> -/+ buffers/cache:    1276400   14850804

Here's another problem: what kind of "free memory" do you measure?
The above "free" measurement should normally be *very* small.
Your above sample of 8G free typically means one of two things:
- The Linux kernel's heuristics failed to make good use of your memory.
- You have too much memory for what you do ("you wasted your money").
So it's perfectly normal to open a file that's larger than this "free"
amount, since the goal of the kernel's memory management is to keep this
"free" about as low as possible (tho not quite 0, just so we can quickly
respond to new memory requests).


        Stefan

Re: warn-maybe-out-of-memory

[Glenn Morris]

Stefan Monnier wrote:

>> This depends on OS and VM pressure. For example, on GNU/Linux if I have
>> just slightly above 8G free:
>
>> $ free
>>              total       used       free     shared    buffers     cached
>> Mem:      16127204    7762072    8365132      68248      84396    6401276
>> -/+ buffers/cache:    1276400   14850804
>
> Here's another problem: what kind of "free memory" do you measure?

I'd also like to know. If it really is the 8G amount in the example
above, then as you say that seems just plain wrong for Emacs to warn
about. If it subtracts off the cache, that's still wrong if you have any
swap space. The total memory is not wrong, but frankly seems completely
pointless. Because who is opening a file with Emacs that matches the
amount of RAM on their machine? The performance of Emacs is pretty poor
with large files - the default large-file-warning-threshold is 10MB!

Re: warn-maybe-out-of-memory

[Dmitry Antipov]

On 07/12/2014 09:17 PM, Glenn Morris wrote:

>> Here's another problem: what kind of "free memory" do you measure?

"Free" means "immediately available for allocation", i.e. not used by
OS or other processes for any purpose. If an OS should perform some
non-trivial action to get more memory (increase swap space or shrink
some internal data structures), this is not accounted as "free".

> I'd also like to know. If it really is the 8G amount in the example
> above, then as you say that seems just plain wrong for Emacs to warn
> about.

In the example above, I asked for 10G having just 8G free. In that example,
OS was able to shrink page cache by 2GB and satisfy the request; but we
can't assume that OS succeeds with this each time we ask to allocate a lot
of memory.

> Because who is opening a file with Emacs that matches the
> amount of RAM on their machine?

This is not uncommon editing task: as Google reports, users asks this question
after unsuccessful attempts to use their standard/favorite editors. For example:
http://askubuntu.com/questions/28847/text-editor-to-edit-4-3-gb-plain-text-file.

 > The performance of Emacs is pretty poor with large files - the default
 > large-file-warning-threshold is 10MB!

This may be subject to change without notice :-).

Dmitry

Re: warn-maybe-out-of-memory

[Glenn Morris]

Dmitry Antipov wrote:

> In the example above, I asked for 10G having just 8G free.

Then it seems just plain wrong for Emacs to warn about that.
I have 6GB of memory on my machine, but the amount "free" according to
your definition is very close to 0 most of the time.

>> The performance of Emacs is pretty poor with large files - the default
>> large-file-warning-threshold is 10MB!

> This may be subject to change without notice :-).

Well, if you manage to improve Emacs's performance in this regard by 3
orders of magnitude, that would be impressive...

Re: warn-maybe-out-of-memory

[Stefan Monnier]

>>> Here's another problem: what kind of "free memory" do you measure?
> "Free" means "immediately available for allocation", i.e. not used by
> OS or other processes for any purpose. If an OS should perform some
> non-trivial action to get more memory (increase swap space or shrink
> some internal data structures), this is not accounted as "free".

Freeing space from the cache is a trivial action.  The OS wants to keep
as much stuff in the cache as it can, thus minimizing the "free" space,
since "free" here basically means "wasted".

If your "free" includes "free in RAM + free in swap" then it's
marginally more useful ("free in RAM" will usually be close to 0,
whereas "free in swap" should usually be fairly large), but I still
can't think of a frequent enough configuration and situation where this
would be useful enough to justify wasting code on it.

>> The performance of Emacs is pretty poor with large files - the default
>> large-file-warning-threshold is 10MB!
> This may be subject to change without notice :-).

If the user changes it, that's her choice, but notice that it is
currently much smaller than anything your "out of memory" check could
hope to detect, so if these small files are already considered
problematic, trying to catch the "yet larger" case seems difficult
to justify.


        Stefan

Re: warn-maybe-out-of-memory

[Dmitry Antipov]

On 07/15/2014 07:45 AM, Stefan Monnier wrote:

> Freeing space from the cache is a trivial action.

AFAICS on Linux, this is not true - in my tests, huge allocations
may be 20% slower when the kernel should reclaim cache space first.
This may be even slower if an OS has to increase swap space
(according to Eli, this may happen on MS-Windows).

> The OS wants to keep as much stuff in the cache as it can, thus
> minimizing the "free" space, since "free" here basically means "wasted".

This depends on usage patterns. If you interleave I/O and relatively
large allocations, it's fairly unreasonable to fill almost all memory
with cached data just to throw it away very soon.

> If your "free" includes "free in RAM + free in swap" then it's
> marginally more useful ("free in RAM" will usually be close to 0,
> whereas "free in swap" should usually be fairly large), but I still
> can't think of a frequent enough configuration and situation where this
> would be useful enough to justify wasting code on it.

This is not expected to be frequent.

Dmitry

Re: warn-maybe-out-of-memory

[Stefan Monnier]

> AFAICS on Linux, this is not true - in my tests, huge allocations
> may be 20% slower when the kernel should reclaim cache space first.

20% slower is still pretty damn fast compared to the time Emacs will
take to fill that space, so it's really not a reason to take this
measurement into account.

>> The OS wants to keep as much stuff in the cache as it can, thus
>> minimizing the "free" space, since "free" here basically means "wasted".
> This depends on usage patterns.  If you interleave I/O and relatively
> large allocations, it's fairly unreasonable to fill almost all memory
> with cached data just to throw it away very soon.

No, the OS doesn't actively fill the memory with cache data.  It just
waits to throw it away as late as possible.

>> If your "free" includes "free in RAM + free in swap" then it's
>> marginally more useful ("free in RAM" will usually be close to 0,
>> whereas "free in swap" should usually be fairly large), but I still
>> can't think of a frequent enough configuration and situation where this
>> would be useful enough to justify wasting code on it.
> This is not expected to be frequent.

I'm not just talking about "infrequent".  Currently, I can't think of
a single case where this would be useful given the existence of
large-file-threshold.


        Stefan

RE: warn-maybe-out-of-memory

[Drew Adams]

> >> I wonder if this function should only warn when it is called from
> >> commands invoked by the user, as opposed to from a Lisp program.  The
> >> warning is in find-file-noselect, which AFAIK is widely used in Lisp
> >> programs, where displaying this warning might be inappropriate.
> 
> Hm, find-file-noselect also may ask for confirmation to open large file.
> If this is undesirable, shouldn't we move all user interaction to top-level
> find-file?

See bug 8180 about user confirmation interaction in `find-file-no-select':
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8180

I think there was also some other discussion, on this list, of the
question you raise, but I cannot seem to find it.

warn-maybe-out-of-memory

[Eli Zaretskii]

I wonder if this function should only warn when it is called from
commands invoked by the user, as opposed to from a Lisp program.  The
warning is in find-file-noselect, which AFAIK is widely used in Lisp
programs, where displaying this warning might be inappropriate.