Re: MPS: Loaded pdump

["Gerd Möllmann" <gerd.moellmann@gmail.com>]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: Emacs Devel <emacs-devel@gnu.org>
>> Date: Thu, 09 May 2024 12:52:03 +0200
>> 
>> I feel more and more that the handling of the loaded pdump with MPS as
>> it is now is not sustainable, and would like ask you for your ideas.
>> 
>> What we do now is make the hot part of the dump an ambig root.
>
> Is this specific to native-compilation, or is it not?

It's not specific to native comp.

> If it isn't, why does it come up only now?  We've been running
> MPS-built Emacs for a week at least; if that produces problems, can
> you tell what problems and how can that be reproduced?

As I said, I thought we could get away with the root, and that finding
an alternative would be an optimization to be done later. (I think
admin/igc.org has some ideas regarding the loaded pdump from some time
ago.)

What I wrote are fundamental problems, so they can't be reproduced.

> IOW, I'd like to understand better why we need to make the hot part of
> the dump an ambiguous root before we consider solutions.

I think this mail I sent has the answer:

  Helmut Eller <eller.helmut@gmail.com> writes:

  >> @Helmut: Did we already talk about what the problem with the frame in
  >> the loaded pdump could be? Sorry that I don't remember.
  >
  > I never heard of that before.

  As the famous philosopher Manuel Manousakis says: Katastrophe!
  🙂

  Okay, I think one can understand this best when I try to describe what
  the pdumper does. Let's start with generating a pdump. I'll try to leave
  out as much details as a can.

  When we create a pdump, we start by allocating 3 big memory blocks which
  I'll call H (hot), C (cold), and R (relocs).

  We then traverse the graph of live objects like the old GC, starting
  from known roots. Each newly encountered object is copyied to H or C in
  binary form. C is used for leaf objects like strings and floats, H for
  the rest.

  The copying of objects is done by invoking type-specific functions,
  example dump_float, dump_vector, etc.

  We cannot use memcpy for the copying because we need more information
  when the pdump is loaded, namely relocation information, which goes to
  R. Relocation is necessary because both Emacs' DATA segment as well as
  H/C may end up at different addresses in a new process.

  Relocation info is recorded in S, and tells us where in the copied
  objects Lisp_Objects or pointers are that need patching when loaded.

  At the end, we write H, C, S to one big file.

  Good. Now let's load that file. We mmap the whole file and now have H',
  C', R' in the new process. C' and R'are good to go (In C are leaf
  objects). H' is patches according to the reloc info that is in S'.

  At the end of the relocation H' is ready to use. Some additional setup
  and initalizations, and we are good to go. I won't describe these.

  Thing is that H' now contains real Lisp objects of basically all types.
  Lisp objects contain references, so I make H' an ambig root.

  So far so good, but some Lisp objects contain not only references to
  other Lisp objects but also pointers to malloc'd memory. Not initially,
  in the dump, but during their lifetime.

  And finally we have reached face_cache.

  If initial_frame is an object in H', fix_frame won't be called for it.
  It cannot because the dump is not part of the MPS memory, and is instead
  traced ambigously as part of the big blob H'.

  Does that make any sense?