Question about memory usage

Question about memory usage

[Michał Kondraciuk]

Hello,

Originally I wanted to report this to bug-gnu-emacs, but it's so basic 
that I thought I'd ask here instead.

Basically, when I run the sexp below in emacs -Q, Emacs keeps allocating 
a lot of memory. In 10 minutes, it goes from 18 MB to over 200 MB.

(while t
   (with-temp-buffer
     (setq buffer-undo-list nil)
     (insert "a")))


Calling garbage-collect afterwards or even inside the body of the loop 
doesn't help (except the loop obviously runs slower, so after 10 
minutes, Emacs uses ~100 MB of memory).
So I want to know if this behavior is expected for some reason? Does 
Emacs reuse this memory somehow (to make future allocations faster)? I 
tested on newest master and 25.3 and the behavior is the same.

Re: Question about memory usage

[Óscar Fuentes]

Michał Kondraciuk <k.michal@zoho.com> writes:

> Hello,
>
> Originally I wanted to report this to bug-gnu-emacs, but it's so basic
> that I thought I'd ask here instead.
>
> Basically, when I run the sexp below in emacs -Q, Emacs keeps
> allocating a lot of memory. In 10 minutes, it goes from 18 MB to over
> 200 MB.
>
> (while t
>   (with-temp-buffer
>     (setq buffer-undo-list nil)
>     (insert "a")))
>
>
> Calling garbage-collect afterwards or even inside the body of the loop
> doesn't help (except the loop obviously runs slower, so after 10
> minutes, Emacs uses ~100 MB of memory).
> So I want to know if this behavior is expected for some reason? Does
> Emacs reuse this memory somehow (to make future allocations faster)? I
> tested on newest master and 25.3 and the behavior is the same.

Maybe it is related to this (emphasis mine):

 -- User Option: gc-cons-threshold
     The value of this variable is the number of bytes of storage that
     must be allocated for Lisp objects after one garbage collection in
     order to trigger another garbage collection.  You can use the
     result returned by ‘garbage-collect’ to get an information about
     size of the particular object type; space allocated to the contents
     of buffers does not count.  NOTE THAT THE SUBSEQUENT GARBAGE
     COLLECTION DOES NOT HAPPEN IMMEDIATELY WHEN THE THRESHOLD IS
     EXHAUSTED, BUT ONLY THE NEXT TIME THE LISP INTERPRETER IS CALLED.

As your example runs in one call of the interpreter, the gargage
collector is never kicked.

Curiously, if a call to garbage-collect is put inside the loop, memory
usage also grows, although slowly.

As for returning the memory to the OS, I've seen Emacs doing that for
very large objects. The rest is simply marked as free.

Re: Question about memory usage

[Eli Zaretskii]

> From: Michał Kondraciuk <k.michal@zoho.com>
> Date: Mon, 2 Apr 2018 13:57:26 +0200
> 
> Basically, when I run the sexp below in emacs -Q, Emacs keeps allocating 
> a lot of memory. In 10 minutes, it goes from 18 MB to over 200 MB.
> 
> (while t
>    (with-temp-buffer
>      (setq buffer-undo-list nil)
>      (insert "a")))
> 
> 
> Calling garbage-collect afterwards or even inside the body of the loop 
> doesn't help (except the loop obviously runs slower, so after 10 
> minutes, Emacs uses ~100 MB of memory).

What do you mean by "afterwards"?  The while-loop never ends, so
there's no "afterwards" AFAIU.  Am I missing something?

To answer your question: yes, I think this is expected, given that you
set buffer-undo-list to nil (what is the purpose of that, btw?).  If
you either delete that line or change it to set buffer-undo-list to t,
you will see a very different picture, as far as the Emacs memory
footprint is concerned.

> So I want to know if this behavior is expected for some reason? Does 
> Emacs reuse this memory somehow (to make future allocations faster)?

When Emacs ends up requesting more memory from the OS, it usually
doesn't release that memory when it is no longer needed, but keeps it
in the process's address space and reuses it if/when it needs more
memory.  Buffer text allocation and deallocation is treated
differently, in that when a buffer is killed, the memory used for its
text is returned to the OS, which is why deleting the line that sets
buffer-undo-list has the effect it does, I think.

Is there some real-life issue behind this experiment?

Re: Question about memory usage

[Stefan Monnier]

>> (while t
>>    (with-temp-buffer
>>      (setq buffer-undo-list nil)
>>      (insert "a")))
[...]
> To answer your question: yes, I think this is expected, given that you
> set buffer-undo-list to nil (what is the purpose of that, btw?).  If

Hmm... why would setting this var make any significant difference here?


        Stefan

Re: Question about memory usage

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 03 Apr 2018 08:28:46 -0400
> 
> >> (while t
> >>    (with-temp-buffer
> >>      (setq buffer-undo-list nil)
> >>      (insert "a")))
> [...]
> > To answer your question: yes, I think this is expected, given that you
> > set buffer-undo-list to nil (what is the purpose of that, btw?).  If
> 
> Hmm... why would setting this var make any significant difference here?

Because the variable is on/referenced via the stack, I suppose.

Re: Question about memory usage

[Eli Zaretskii]

> Date: Tue, 03 Apr 2018 15:40:27 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date: Tue, 03 Apr 2018 08:28:46 -0400
> > 
> > >> (while t
> > >>    (with-temp-buffer
> > >>      (setq buffer-undo-list nil)
> > >>      (insert "a")))
> > [...]
> > > To answer your question: yes, I think this is expected, given that you
> > > set buffer-undo-list to nil (what is the purpose of that, btw?).  If
> > 
> > Hmm... why would setting this var make any significant difference here?
> 
> Because the variable is on/referenced via the stack, I suppose.

Or maybe I misunderstand what you meant by "make significant
difference"?  What did you mean?

Re: Question about memory usage

[Stefan Monnier]

>> >> (while t
>> >>    (with-temp-buffer
>> >>      (setq buffer-undo-list nil)
>> >>      (insert "a")))
>> [...]
>> > To answer your question: yes, I think this is expected, given that you
>> > set buffer-undo-list to nil (what is the purpose of that, btw?).  If
>> Hmm... why would setting this var make any significant difference here?
> Because the variable is on/referenced via the stack, I suppose.

I don't follow.  At the end of each iteration of the loop, we kill the
temp buffer, so the effect of having set its buffer-undo-list slot or
not should be negligible (or more specifically: this effect should only
exist until the next GC).


        Stefan

Re: Question about memory usage

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 03 Apr 2018 09:13:44 -0400
> 
> >> Hmm... why would setting this var make any significant difference here?
> > Because the variable is on/referenced via the stack, I suppose.
> 
> I don't follow.  At the end of each iteration of the loop, we kill the
> temp buffer, so the effect of having set its buffer-undo-list slot or
> not should be negligible (or more specifically: this effect should only
> exist until the next GC).

As I said: maybe I misunderstand what you are asking.  So let me say
something that perhaps better matches your question: a temporary
buffer has its undo turned off, but setting buffer-undo-list to nil
turns this on again, so inserting a character into the buffer conses
stuff onto the undo-list.

Does that answer the question "why would setting the variable make any
difference"?

Re: Question about memory usage

[Stefan Monnier]

> As I said: maybe I misunderstand what you are asking.  So let me say
> something that perhaps better matches your question: a temporary
> buffer has its undo turned off, but setting buffer-undo-list to nil
> turns this on again, so inserting a character into the buffer conses
> stuff onto the undo-list.

Yes, but that undo-list is local to the buffer, so once we kill the
buffer (at the end of each iteration) this should be reclaimable.


        Stefan

Re: Question about memory usage

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 03 Apr 2018 10:27:56 -0400
> 
> > As I said: maybe I misunderstand what you are asking.  So let me say
> > something that perhaps better matches your question: a temporary
> > buffer has its undo turned off, but setting buffer-undo-list to nil
> > turns this on again, so inserting a character into the buffer conses
> > stuff onto the undo-list.
> 
> Yes, but that undo-list is local to the buffer, so once we kill the
> buffer (at the end of each iteration) this should be reclaimable.

Why do you think it isn't reclaimed?

AFAIU, the issue was to explain why the memory footprint grows,
nothing was said about how much of that memory is free for reuse.  My
point is that activating the undo-list would use more of the "normal"
heap memory, in contrast to using only/mostly buffer text memory
without that.

Re: Question about memory usage

[Michał Kondraciuk]

On 04/03/2018 08:28 AM, Eli Zaretskii wrote:
>> From: Michał Kondraciuk <k.michal@zoho.com>
>> Date: Mon, 2 Apr 2018 13:57:26 +0200
>>
>> Basically, when I run the sexp below in emacs -Q, Emacs keeps allocating
>> a lot of memory. In 10 minutes, it goes from 18 MB to over 200 MB.
>>
>> (while t
>>     (with-temp-buffer
>>       (setq buffer-undo-list nil)
>>       (insert "a")))
>>
>>
>> Calling garbage-collect afterwards or even inside the body of the loop
>> doesn't help (except the loop obviously runs slower, so after 10
>> minutes, Emacs uses ~100 MB of memory).
> 
> What do you mean by "afterwards"?  The while-loop never ends, so
> there's no "afterwards" AFAIU.  Am I missing something?

You're right, I meant after I stop the loop with C-g.

> To answer your question: yes, I think this is expected, given that you
> set buffer-undo-list to nil (what is the purpose of that, btw?). 

No purpose, but some external packages display information in a way 
similar to this sexp, where undo information is also recorded (needlessly):

(with-current-buffer (get-buffer-create "info")
    (let ((inhibit-read-only t))
      (erase-buffer)
      (insert "info"))
    (setq buffer-read-only t)
    (display-buffer (current-buffer)))

I tested it in a loop (but also killed the "info" buffer in each 
iteration) and Emacs also keeps allocating memory, as with 
(with-temp-buffer (setq buffer-undo-list nil) ...).
Obviously normally this isn't a problem because you won't run this code 
in a loop, but if you leave Emacs open for a long time and run this sexp 
as part of some command many times, memory can accumulate (I think, it's 
why I asked if Emacs will finally free this unused memory).

> When Emacs ends up requesting more memory from the OS, it usually
> doesn't release that memory when it is no longer needed, but keeps it
> in the process's address space and reuses it if/when it needs more
> memory.
But shouldn't Emacs reuse the memory from previous loop iteration 
instead of allocating it? Also, if the sexp is modified like this, Emacs 
memory usage is at the same level:


(while t
   (with-temp-buffer
     (insert "a")
     (setq buffer-undo-list (list (cons (point-min) (point-max))))))

Re: Question about memory usage

[Eli Zaretskii]

> From: Michał Kondraciuk <k.michal@zoho.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Tue, 3 Apr 2018 19:57:34 +0200
> 
> > To answer your question: yes, I think this is expected, given that you
> > set buffer-undo-list to nil (what is the purpose of that, btw?). 
> 
> No purpose, but some external packages display information in a way 
> similar to this sexp, where undo information is also recorded (needlessly):

To disable undo, you should bind buffer-undo-list to t, not to nil.
And with-temp-buffer already does that, because temporary buffers have
their undo disabled by default.

> Obviously normally this isn't a problem because you won't run this code 
> in a loop, but if you leave Emacs open for a long time and run this sexp 
> as part of some command many times, memory can accumulate (I think, it's 
> why I asked if Emacs will finally free this unused memory).

In normal usage, the memory footprint of an Emacs session levels out
after some time, and stays approximately fixed, unless you do
something extraordinary.

> > When Emacs ends up requesting more memory from the OS, it usually
> > doesn't release that memory when it is no longer needed, but keeps it
> > in the process's address space and reuses it if/when it needs more
> > memory.
> But shouldn't Emacs reuse the memory from previous loop iteration 
> instead of allocating it?

That depends on the heap fragmentation and the efficiency of the
memory allocating functions to deal with fragmentation.

> Also, if the sexp is modified like this, Emacs 
> memory usage is at the same level:
> 
> 
> (while t
>    (with-temp-buffer
>      (insert "a")
>      (setq buffer-undo-list (list (cons (point-min) (point-max))))))

Did you try not setting buffer-undo-list at all?  What did you see
then?

Re: Question about memory usage

[Michał Kondraciuk]

On 04/03/2018 08:22 PM, Eli Zaretskii wrote:
>> From: Michał Kondraciuk <k.michal@zoho.com>
>> Cc: help-gnu-emacs@gnu.org
>> Date: Tue, 3 Apr 2018 19:57:34 +0200
>>
>>> To answer your question: yes, I think this is expected, given that you
>>> set buffer-undo-list to nil (what is the purpose of that, btw?).
>>
>> No purpose, but some external packages display information in a way
>> similar to this sexp, where undo information is also recorded (needlessly):
> 
> To disable undo, you should bind buffer-undo-list to t, not to nil.
> And with-temp-buffer already does that, because temporary buffers have
> their undo disabled by default.

I know that, I meant it's common to see code like this in some packages, 
especially when the buffer is actually displayed.

> Did you try not setting buffer-undo-list at all?  What did you see
> then?

Emacs behaves as expected, i.e. memory usage is ~20MB all the time.

Re: Question about memory usage

[Stefan Monnier]

>> Yes, but that undo-list is local to the buffer, so once we kill the
>> buffer (at the end of each iteration) this should be reclaimable.
> Why do you think it isn't reclaimed?

I don't know if it's reclaimed or not, but I think it should be
reclaimable.  And if the GC reclaims it, next time around the loop the
same (now free) cons-cell should be reused instead of requesting more
memory from the OS.

So according to how I expect the code to behave, this setting might
cause the memory footprint to be (very slightly) larger but it should not
cause it to keep growing.


        Stefan

Re: Question about memory usage

[Stefan Monnier]

>> But shouldn't Emacs reuse the memory from previous loop iteration 
>> instead of allocating it?
> That depends on the heap fragmentation and the efficiency of the
> memory allocating functions to deal with fragmentation.

The elements of buffer-undo-list in this case are just cons cells (and
immediate integers and nil), IIUC so I don't see why fragmentation would
get in the way: the previous cons-cell of the buffer-undo-list of the
previous iterations, reclaimed by the GC, should be readily reusable.


        Stefan

Re: Question about memory usage

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 03 Apr 2018 17:18:39 -0400
> 
> >> But shouldn't Emacs reuse the memory from previous loop iteration 
> >> instead of allocating it?
> > That depends on the heap fragmentation and the efficiency of the
> > memory allocating functions to deal with fragmentation.
> 
> The elements of buffer-undo-list in this case are just cons cells (and
> immediate integers and nil), IIUC so I don't see why fragmentation would
> get in the way: the previous cons-cell of the buffer-undo-list of the
> previous iterations, reclaimed by the GC, should be readily reusable.

Cons cells are GCed only when enough of them were consed.

Re: Question about memory usage

[Eli Zaretskii]

> From: Michał Kondraciuk <k.michal@zoho.com>
> Date: Tue, 3 Apr 2018 21:16:41 +0200
> 
> > To disable undo, you should bind buffer-undo-list to t, not to nil.
> > And with-temp-buffer already does that, because temporary buffers have
> > their undo disabled by default.
> 
> I know that, I meant it's common to see code like this in some packages, 
> especially when the buffer is actually displayed.
> 
> > Did you try not setting buffer-undo-list at all?  What did you see
> > then?
> 
> Emacs behaves as expected, i.e. memory usage is ~20MB all the time.

OK, so what is your question now?  Are you asking, like Stefan, why
setting buffer-undo-list to nil in this case makes a difference, or
are you asking a more general question (and if the latter, what are
you asking)?

Re: Question about memory usage

[Stefan Monnier]

>> >> But shouldn't Emacs reuse the memory from previous loop iteration 
>> >> instead of allocating it?
>> > That depends on the heap fragmentation and the efficiency of the
>> > memory allocating functions to deal with fragmentation.
>> The elements of buffer-undo-list in this case are just cons cells (and
>> immediate integers and nil), IIUC so I don't see why fragmentation would
>> get in the way: the previous cons-cell of the buffer-undo-list of the
>> previous iterations, reclaimed by the GC, should be readily reusable.
> Cons cells are GCed only when enough of them were consed.

They get GC'd when the GC is run.  Yes, the GC won't be invoked at each
iteration, but that shouldn't affect the long-term trend.  At least
I don't see how it can explain the growth that the OP describes: it can
explain a slightly larger footprint, but not a continued growth.


        Stefan

Re: Question about memory usage

[Michał Kondraciuk]

On 04/04/2018 08:55 AM, Eli Zaretskii wrote:
>>> Did you try not setting buffer-undo-list at all?  What did you see
>>> then?
>>
>> Emacs behaves as expected, i.e. memory usage is ~20MB all the time.
> 
> OK, so what is your question now?  Are you asking, like Stefan, why
> setting buffer-undo-list to nil in this case makes a difference, or
> are you asking a more general question (and if the latter, what are
> you asking)?

I found out what the problem was:

(while t
   (with-temp-buffer
     (setq buffer-undo-list nil)
     (insert "a")
     (print (length undo-auto--undoably-changed-buffers)
	   #'external-debugging-output)))

The variable undo-auto--undoably-changed-buffers was storing a lot of 
killed buffers. There's a timer that periodically clears this variable, 
but it didn't get a chance to run, since Emacs was never idle. Calling 
sit-for inside the loop solves this - not that it matters, because this 
is not "normal" code. Thanks for the responses.

Re: Question about memory usage

[Eli Zaretskii]

> From: Michał Kondraciuk <k.michal@zoho.com>
> Cc: help-gnu-emacs@gnu.org, monnier@iro.umontreal.ca
> Date: Thu, 5 Apr 2018 20:06:34 +0200
> 
> (while t
>    (with-temp-buffer
>      (setq buffer-undo-list nil)
>      (insert "a")
>      (print (length undo-auto--undoably-changed-buffers)
> 	   #'external-debugging-output)))
> 
> The variable undo-auto--undoably-changed-buffers was storing a lot of 
> killed buffers.

Maybe kill-buffer should remove that buffer's reference from the list,
then.

Re: Question about memory usage

[Stefan Monnier]

> I found out what the problem was:
>
> (while t
>   (with-temp-buffer
>     (setq buffer-undo-list nil)
>     (insert "a")
>     (print (length undo-auto--undoably-changed-buffers)
> 	   #'external-debugging-output)))

Aha!

> The variable undo-auto--undoably-changed-buffers was storing a lot of killed
> buffers.

We could/should probably arrange to scan for dead buffers every time we
add a new buffer to it, or from kill-buffer-hook.


        Stefan

Re: Question about memory usage

[Michał Kondraciuk]

[-- Attachment #1: Type: text/plain, Size: 216 bytes --]

On 04/05/2018 08:54 PM, Stefan Monnier wrote:
> We could/should probably arrange to scan for dead buffers every time we
> add a new buffer to it, or from kill-buffer-hook.

Can you take a look at the attached patch?

[-- Attachment #2: remove-killed-buffer-from-changed-list.patch --]
[-- Type: text/x-patch, Size: 687 bytes --]

diff --git a/lisp/simple.el b/lisp/simple.el
index aad8d3b..23f7c4f 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -3014,6 +3014,13 @@ undo-auto--undoably-changed-buffers
 `undo-auto--boundaries' and can be affected by changes to their
 default values.")
 
+(defun undo-auto--remove-buffer-from-changed-list ()
+  "Remove current buffer from list of recently changed ones."
+  (setq undo-auto--undoably-changed-buffers
+        (delq (current-buffer) undo-auto--undoably-changed-buffers)))
+
+(add-hook 'kill-buffer-hook #'undo-auto--remove-buffer-from-changed-list)
+
 (defun undo-auto--add-boundary ()
   "Add an `undo-boundary' in appropriate buffers."
   (undo-auto--boundaries

Re: Question about memory usage

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, help-gnu-emacs@gnu.org
> From: Michał Kondraciuk <k.michal@zoho.com>
> Date: Sat, 7 Apr 2018 15:15:08 +0200
> 
> Can you take a look at the attached patch?
> 
> diff --git a/lisp/simple.el b/lisp/simple.el
> index aad8d3b..23f7c4f 100644
> --- a/lisp/simple.el
> +++ b/lisp/simple.el
> @@ -3014,6 +3014,13 @@ undo-auto--undoably-changed-buffers
>  `undo-auto--boundaries' and can be affected by changes to their
>  default values.")
>  
> +(defun undo-auto--remove-buffer-from-changed-list ()
> +  "Remove current buffer from list of recently changed ones."
> +  (setq undo-auto--undoably-changed-buffers
> +        (delq (current-buffer) undo-auto--undoably-changed-buffers)))
> +
> +(add-hook 'kill-buffer-hook #'undo-auto--remove-buffer-from-changed-list)
> +

Thanks, but I'd be more comfortable with just doing this from
kill-buffer itself.  IMO, hooks are for users and Lisp applications;
using them for internal bookkeeping purposes should be limited to the
cases where there's no better alternative.

Re: Question about memory usage

[Stefan Monnier]

> Thanks, but I'd be more comfortable with just doing this from
> kill-buffer itself.

I installed an alternative patch.


        Stefan

Re: Question about memory usage

[Michał Kondraciuk]

On 04/07/2018 05:22 PM, Stefan Monnier wrote:
>> Thanks, but I'd be more comfortable with just doing this from
>> kill-buffer itself.
> 
> I installed an alternative patch.
> 
> 
>          Stefan

Great, this patch also made the loop run much faster. Thank you.