Any way of dumping strings?

Any way of dumping strings?

[Óscar Fuentes]

I'm trying to diagnose a memory leak. As it is used Emacs requires more
memory, an average of a several tens of MB per hour. Right now it is
using ~1GB and `memory-report' shows:

 207.8 MiB  Vectors
  83.7 MiB  Strings

I terminated the previous session after reaching ~2GB with >700MB
vectors and >300MB strings.

In case the culprit is some package, looking at those strings or vectors
could be revealing.

Is there a method for dumping or displaying those strings or vectors?

Re: Any way of dumping strings?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Wed, 02 Jun 2021 19:03:06 +0200
> 
> Is there a method for dumping or displaying those strings or vectors?

Set a GDB breakpoint in mark_vectorlike, with commands that dump each
vector, then trigger GC?

Re: Any way of dumping strings?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Wed, 02 Jun 2021 19:03:06 +0200
>> 
>> Is there a method for dumping or displaying those strings or vectors?
>
> Set a GDB breakpoint in mark_vectorlike, with commands that dump each
> vector, then trigger GC?

I was hoping for something that could show at least thousands of
instances. Going one by one would be extremely time-consuming and
potentially deceptive.

Does Emacs zero-initialize memory when it is garbage-collected? Even if
it doesn't, maybe it is enough to dump the whole image to a file. Some
part would be gc-ed data, but a comparatively small part among all the
rest of hundreds of MB.

Re: Any way of dumping strings?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Wed, 02 Jun 2021 19:54:20 +0200
> 
> > Set a GDB breakpoint in mark_vectorlike, with commands that dump each
> > vector, then trigger GC?
> 
> I was hoping for something that could show at least thousands of
> instances. Going one by one would be extremely time-consuming and
> potentially deceptive.

That's not what I had in mind.  You can set up breakpoint commands
that dump the contents of the vector being marked, then continue
without stopping.  Then say "set logging on", so that the output goes
to a file, and let Emacs run.  Once GC finishes, you will have a file
with all the vectors in it; then you can examine them at your leisure.

> Does Emacs zero-initialize memory when it is garbage-collected?

You mean, those objects it recycles? it doesn't zero them, but why
would you be interested in the objects that get GCed -- those are not
your problem.  Your problem is those which aren't GCed, and
mark_vector marks each one of them.

> Even if it doesn't, maybe it is enough to dump the whole image to a
> file. Some part would be gc-ed data, but a comparatively small part
> among all the rest of hundreds of MB.

How would you tell the one from the other?

Re: Any way of dumping strings?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Wed, 02 Jun 2021 19:54:20 +0200
>> 
>> > Set a GDB breakpoint in mark_vectorlike, with commands that dump each
>> > vector, then trigger GC?
>> 
>> I was hoping for something that could show at least thousands of
>> instances. Going one by one would be extremely time-consuming and
>> potentially deceptive.
>
> That's not what I had in mind.  You can set up breakpoint commands
> that dump the contents of the vector being marked, then continue
> without stopping.  Then say "set logging on", so that the output goes
> to a file, and let Emacs run.  Once GC finishes, you will have a file
> with all the vectors in it; then you can examine them at your leisure.

That looks quite useful for this purpose, thanks.

Now that I think of it, my impression on the past session was that
performance degraded quite a bit, which could indicate that a very large
number of objects was live (which implies that the leaked objects are
relatively small, right?).

>> Even if it doesn't, maybe it is enough to dump the whole image to a
>> file. Some part would be gc-ed data, but a comparatively small part
>> among all the rest of hundreds of MB.
>
> How would you tell the one from the other?

I wouldn't. I'll just do some crude frequency analysis expecting that
GCed data is noise among the much larger signal of live data.

Re: Any way of dumping strings?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Wed, 02 Jun 2021 20:33:22 +0200
> 
> > That's not what I had in mind.  You can set up breakpoint commands
> > that dump the contents of the vector being marked, then continue
> > without stopping.  Then say "set logging on", so that the output goes
> > to a file, and let Emacs run.  Once GC finishes, you will have a file
> > with all the vectors in it; then you can examine them at your leisure.
> 
> That looks quite useful for this purpose, thanks.
> 
> Now that I think of it, my impression on the past session was that
> performance degraded quite a bit, which could indicate that a very large
> number of objects was live (which implies that the leaked objects are
> relatively small, right?).

I don't know.  Maybe you have a significant number of medium-size
objects as well.  In any case, a large Lisp object in most cases also
means a lot of small ones -- these are the elements of that large
object: a list or a large vector or similar.  Only elements that are
atoms, like integers, don't add to GC's complexity, and such elements
are relatively rare in Emacs.

Re: Any way of dumping strings?

[Stefan Monnier]

> I'm trying to diagnose a memory leak. As it is used Emacs requires more
> memory, an average of a several tens of MB per hour. Right now it is
> using ~1GB and `memory-report' shows:
>
>  207.8 MiB  Vectors
>   83.7 MiB  Strings
>
> I terminated the previous session after reaching ~2GB with >700MB
> vectors and >300MB strings.
>
> In case the culprit is some package, looking at those strings or vectors
> could be revealing.
>
> Is there a method for dumping or displaying those strings or vectors?

Have you tried the new `memory-report` function?


        Stefan

Re: Any way of dumping strings?

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Have you tried the new `memory-report` function?

Yes, I mention that on the message you just quoted :-)

Re: Any way of dumping strings?

[Stefan Monnier]

Óscar Fuentes [2021-06-02 20:55:07] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> Have you tried the new `memory-report` function?
> Yes, I mention that on the message you just quoted :-)

Duh!  So, IIUC the report about the size of specific variables doesn't
point give any hint about where those many strings&vectors are located?
Too bad.  I hope once you find the origin of the problem we can refine
`memory-report` so that it gives info which would have been helpful now.


        Stefan

Re: Any way of dumping strings?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Wed, 02 Jun 2021 15:30:34 -0400
> 
> Óscar Fuentes [2021-06-02 20:55:07] wrote:
> > Stefan Monnier <monnier@iro.umontreal.ca> writes:
> >> Have you tried the new `memory-report` function?
> > Yes, I mention that on the message you just quoted :-)
> 
> Duh!  So, IIUC the report about the size of specific variables doesn't
> point give any hint about where those many strings&vectors are located?
> Too bad.  I hope once you find the origin of the problem we can refine
> `memory-report` so that it gives info which would have been helpful now.

I think a useful extension of memory-report would be optionally to
dump all live objects of a given type to somewhere.  The challenge of
doing that in Lisp is that doing so will generally cons more Lisp
objects, thus skewing both the memory report and possibly the dumped
objects as well.  Patches welcome.