Does Emacs return memory to the system on Mac OS X or *BSD?

Does Emacs return memory to the system on Mac OS X or *BSD?

[George Plymale II]

Hi,

So, this question is spurred by a discussion found in this reddit
thread: https://www.reddit.com/r/emacs/comments/5rswpn/freeing_dead_heap/ddae477/
I'm curious as to whether Emacs returns memory to the operating system
when running on Mac OS X or *BSD systems. As is made clear in the
discussion, such things are system-dependent and there is no unified
resource for which systems support this behavior and which ones do
not. If no one here knows about this off the top of their head, which
specific source file(s) can I browse to learn about this? Thanks.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Stefan Monnier]

> I'm curious as to whether Emacs returns memory to the operating system
> when running on Mac OS X or *BSD systems.

This question is largely ill-posed: would you really care if Emacs once in
a blue moon returns 4KB out of a 2GB heap?

The answer depends on a lot of details about the system, the
compile-time options used, and the specific usage pattern.
And in my experience, in the end it never really matters anyway.


        Stefan

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[George Plymale II]

> would you really care if Emacs once in
> a blue moon returns 4KB out of a 2GB heap?

Sorry, I did not know that Emacs would return pages rarely. I figured it
would be somewhat common, if you have multiple, dead GBs of heap laying
around (as in the linked reddit post). Such a thing seems like a token
that you should give some of that back to the OS. But, that is just my
opinion at face value of the situation and I admittedly lack experience
in implementing this sort of thing.

> The answer depends on a lot of details about the system, the
> compile-time options used, and the specific usage pattern.

Well, again, which source file(s) may I poke at to see this sort of
stuff in action myself?

Thanks.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Stefan Monnier]

>> would you really care if Emacs once in
>> a blue moon returns 4KB out of a 2GB heap?
> Sorry, I did not know that Emacs would return pages rarely.

I didn't say that it's what happens.  I just pointed out that if this
happens, then the answer to your question would be a resounding "yes"
even though I'm pretty sure you wouldn't be satisfied with the
actual behavior.


        Stefan

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Stefan Monnier]

> Sorry, I did not know that Emacs would return pages rarely. I figured it
> would be somewhat common, if you have multiple, dead GBs of heap laying
> around (as in the linked reddit post). Such a thing seems like a token
> that you should give some of that back to the OS.

If it's a single 1GB chunk, it'll most likely be returned (tho there are
unlucky build-configurations where that might happen, and I believe such
situation has unwittingly been in common use for Emacs-25.1 under
GNU/Linux, IIUC).

In the reddit thread, the 1.5GB is completely unexplained, so it's hard
to say anything intelligent about it.


        Stefan

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[George Plymale II]

> I didn't say that it's what happens.  I just pointed out that if this
> happens, then the answer to your question would be a resounding "yes"
> even though I'm pretty sure you wouldn't be satisfied with the
> actual behavior.

Ok, thanks for the clarification.

> If it's a single 1GB chunk, it'll most likely be returned (tho there are
> unlucky build-configurations where that might happen, and I believe such
> situation has unwittingly been in common use for Emacs-25.1 under
> GNU/Linux, IIUC).

Thanks for the information; that's good to know.

> In the reddit thread, the 1.5GB is completely unexplained, so it's hard
> to say anything intelligent about it.

Actually, the author stated in a another post,
(https://www.reddit.com/r/emacs/comments/5rswpn/freeing_dead_heap/ddfil8v/)
in reply to the parent of the linked thread, that he is running on Linux
and had been doing some heavy duty web scraping via EWW and had some
PDFs open.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Eli Zaretskii]

> From: George Plymale II <georgie@southernohio.net>
> Date: Wed, 10 May 2017 02:31:09 -0400
> Cc: emacs-devel@gnu.org
> 
> Well, again, which source file(s) may I poke at to see this sort of
> stuff in action myself?

If you are asking about the Emacs sources, then the relevant suspects
are gmalloc.c and ralloc.c, but each one of these is only relevant to
your build if you see the corresponding *.o file in the build
directory.

The other relevant part is the use of mmap for buffer text; look in
src/config.h for the definition of USE_MMAP_FOR_BUFFERS.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[George Plymale II]

Thanks for the specifics. That certainly helps narrow down where I
should look to find what I'm interested in. FYI, this is what I found in
`config.h' for `USE_MMAP_FOR_BUFFERS':

/* Define to use mmap to allocate buffer text. */
/* #undef USE_MMAP_FOR_BUFFERS */

There are no other occurrences of it so I guess it's not defined at all
in my build.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 1336 bytes --]

This may be completely off on the wrong track, but I suspect this really
relates to the underlying C library implementation. I seem to remember
reading (quite some time ago) about limitations on some platforms in the
ability of some C libraries being able to give memory back to the operating
system. My recall is vague as it was a long time ago, but in essence, while
an app can release memory it no longer requires, that memory is not
actually released back to the OS to be used by other apps until the app has
exited. The upshot is that while the app i.e. emacs, may have
freed/released memory, the OS won't see that released memory until after
emacs has exited.

I can say that I use emacs intensively on both Linux and OSX and run for
long periods of time and certainly don't notice any loss of available
memory or emacs growing in footprint size.

On 11 May 2017 at 05:16, George Plymale II <georgie@southernohio.net> wrote:

> Thanks for the specifics. That certainly helps narrow down where I
> should look to find what I'm interested in. FYI, this is what I found in
> `config.h' for `USE_MMAP_FOR_BUFFERS':
>
> /* Define to use mmap to allocate buffer text. */
> /* #undef USE_MMAP_FOR_BUFFERS */
>
> There are no other occurrences of it so I guess it's not defined at all
> in my build.
>
>


-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 1917 bytes --]

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[George Plymale II]

> but in essence, while an app can release memory it no longer requires,
> that memory is not actually released back to the OS to be used by
> other apps until the app has exited. The upshot is that while the app
> i.e. emacs, may have freed/released memory, the OS won't see that
> released memory until after emacs has exited.

Yes, I originally feared that this sort of thing would be the case as I
have seen such from many other apps. I was just wondering if it's
different for Emacs, esp. after reading Eli's initial comment in the
original reddit thread. Anyway, one would probably have to do some more
digging in the places that Eli mentioned to find out for sure (hence my
initial question about that).

> I can say that I use emacs intensively on both Linux and OSX and run
> for long periods of time and certainly don't notice any loss of
> available memory or emacs growing in footprint size.

Yes, I've used Emacs on both as well for long periods of time. It
certainly isn't nearly as bad as many other apps. However, on my main
workstation, I often have many memory-craving processes open and I've
been forced to reboot on occasion due to system malfunctions stemming
from a lack of memory. Ever since the last time that happened, I've
tried to be more vigilant about how much memory things are using,
although I've not seen Emacs exceed 300 MB, which isn't too bad. Still,
I was obviously curious if I can lower that amount (or if Emacs will do
that automatically (which according to Stefan, it likely does do so if
it's a single 1 GB chunk)) in case I accidentally start leaking memory
and don't want to restart it.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Stefan Monnier]

> system. My recall is vague as it was a long time ago, but in essence, while
> an app can release memory it no longer requires, that memory is not
> actually released back to the OS to be used by other apps until the app has
> exited. The upshot is that while the app i.e. emacs, may have
> freed/released memory, the OS won't see that released memory until after
> emacs has exited.

Indeed.  And in practice it makes very little difference: this freed
memory, by its very nature, will not be accessed by the application, so
it will naturally end up swapped to disk if memory pressure requires it.

IOW the only difference is in the fact that it requires more swap space,
which can occasionally be noticeable, e.g. when you don't have swap
space at all.

> I can say that I use emacs intensively on both Linux and OSX and run for
> long periods of time and certainly don't notice any loss of available
> memory or Emacs growing in footprint size.

Since this "unreturned freed memory" is free and hence available for
reuse, it cannot cause excessive *growing* of a process.  It can only
cause a lack of shrinking, which is why it rarely matters.

Excessive growth ("excessive" in the sense that it eats more memory than
what is actually used) is usually caused by things like leaks or
fragmentation, which are different issues.


        Stefan

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Stefan Monnier]

> although I've not seen Emacs exceed 300 MB, which isn't too bad.  Still,
> I was obviously curious if I can lower that amount

IOW, you sometimes see Emacs use an amount of memory you think is too
large, but you have no idea whatsoever why that might be the case (IIUC
you don't even know if it's really too large or if it causes a problem).

There can be *many* different reasons, beside the fact that some freed
memory is not returned to the OS.  More to the point, whether the memory
is returned to the OS or not is usually the least concern on a laptop or
desktop machine.


        Stefan

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[George Plymale II]

> IOW, you sometimes see Emacs use an amount of memory you think is too
> large, but you have no idea whatsoever why that might be the case

Actually, I have monitored which packages, functions, etc. cause Emacs
memory usage to spike up and I've monitored how much of that memory is
marked as "dead" via the memory-usage package on ELPA. So I do have an
idea about why I think some memory seems just a little too large and
which things could be causing that.

> (IIUC you don't even know if it's really too large or if it causes a
> problem)

There's rarely ever a problem. It's not "really too large." My only
point here was to satisfy my own curiosity about whether or not Emacs
gives memory back as Eli originally suggested is possible. It doesn't
hurt to have one process use less memory on my system. But I'm not
trying to complain about said memory usage (which you seem to think that
I'm doing). I was just trying to satisfy a little point of curiosity as
Eli suggested could be done here. Don't read too much into my comment:
"although I've not seen Emacs exceed 300 MB, which isn't too bad", as I
was merely pointing out that I, too, have not had serious memory
problems with Emacs, just a curiosity about the situation.

> There can be *many* different reasons, beside the fact that some freed
> memory is not returned to the OS. 

Ok, I appreciate that there are different reasons as to why memory usage
goes up aside from just not giving stuff back to the OS. I was only
wondering if it is possible on the aforementioned systems, to perhaps
somewhat mitigate the issue if I ever did have a problem.

> More to the point, whether the memory is returned to the OS or not is
> usually the least concern on a laptop or desktop machine.

Ok, I respect that opinion. As previously said, I understand that such a
thing is often not an important factor in real situations. I was just
wondering if it happens or can happen on my platforms.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Stefan Monnier]

> Ok, I appreciate that there are different reasons as to why memory usage
> goes up aside from just not giving stuff back to the OS.

Not returning memory to the OS should basically never cause the memory
use to grow.  It can only cause it to fail to shrink.


        Stefan

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: George Plymale II <georgie@southernohio.net>,  Eli Zaretskii <eliz@gnu.org>,  Emacs developers <emacs-devel@gnu.org>
> Date: Wed, 10 May 2017 20:06:05 -0400
> 
> IOW the only difference is in the fact that it requires more swap space,
> which can occasionally be noticeable, e.g. when you don't have swap
> space at all.

No, it also means that part of the VM address space is in use and
cannot be used by other apps.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Stefan Monnier]

>> IOW the only difference is in the fact that it requires more swap space,
>> which can occasionally be noticeable, e.g. when you don't have swap
>> space at all.
> No, it also means that part of the VM address space is in use and
> cannot be used by other apps.

What do you mean by "VM address space"?  Each process has its own
address space, so Emacs's use of address space shouldn't impact
other processes.  But maybe you just mean the same as what I meant by
"swap" (i.e. the total amount of (virtual) memory that the system can
handle).


        Stefan

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: theophilusx@gmail.com,  georgie@southernohio.net,  emacs-devel@gnu.org
> Date: Thu, 11 May 2017 08:29:31 -0400
> 
> >> IOW the only difference is in the fact that it requires more swap space,
> >> which can occasionally be noticeable, e.g. when you don't have swap
> >> space at all.
> > No, it also means that part of the VM address space is in use and
> > cannot be used by other apps.
> 
> What do you mean by "VM address space"?

That pages mapped into some process's address space diminish the total
amount of VM which the system can allocate, even if those pages are
swapped out of physical memory.  Only unmapping them from the
process's address space will make them available for use by other
processes.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Stefan Monnier]

>> >> IOW the only difference is in the fact that it requires more swap space,
>> >> which can occasionally be noticeable, e.g. when you don't have swap
>> >> space at all.
>> > No, it also means that part of the VM address space is in use and
>> > cannot be used by other apps.
>> What do you mean by "VM address space"?
> That pages mapped into some process's address space diminish the total
> amount of VM which the system can allocate, even if those pages are
> swapped out of physical memory.

I think we're in violent agreement here.


        Stefan

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 11 May 2017 11:39:27 -0400
> Cc: theophilusx@gmail.com, georgie@southernohio.net, emacs-devel@gnu.org
> 
> I think we're in violent agreement here.

As always.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Davis Herring]

> What do you mean by "VM address space"?  Each process has its own
> address space, so Emacs's use of address space shouldn't impact
> other processes.  But maybe you just mean the same as what I meant by
> "swap" (i.e. the total amount of (virtual) memory that the system can
> handle).

Microsoft has long confused "virtual memory" with "swap", which leads to 
the false etymology that you have "real memory" (RAM) and 
"not-quite-real [=virtual] memory" (swap).  Propertly, VM includes all 
possible addresses, and the total amount "allocated" can be larger even 
than RAM+swap (even without overcommit) because of things like 
(non-anonymous) mmap that can be backed by neither RAM nor swap.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

Re: Does Emacs return memory to the system on Mac OS X or *BSD?

[Andreas Politz]

George Plymale II <georgie@southernohio.net> writes:

> Actually, the author stated in a another post, [...] that he is [...]
> doing some heavy duty web scraping via EWW and had some PDFs open.

Opening and displaying many PDFs resp. pages in a relatively short time
can use a lot of memory (I think around 1MB/page is not to far from the
truth.). This is due to the fact that images are kept cached until they
expire via image-cache-eviction-delay, which has a default value of
12min, I believe. Of course, this situation could be improved.

I think this is true for both doc-view and pdf-tools.

-ap