Re: MPS: Loaded pdump

["Gerd Möllmann" <gerd.moellmann@gmail.com>]

Helmut Eller <eller.helmut@gmail.com> writes:

> On Thu, May 09 2024, Gerd Möllmann wrote:
>
>> I feel more and more that the handling of the loaded pdump with MPS as
>> it is now is not sustainable, and would like ask you for your ideas.
>>
>> What we do now is make the hot part of the dump an ambig root. I don't
>> remember the exact numbers, but I think that's about 18 Mb of root. This
>> has at least these problems, from my POV:
>>
>> - It is very large, and every time MPS scans roots, and that is all the
>>   time, the world is stopped until it has finished. That's not good for
>>   pause times.
>
> Maybe it would be better with the MPS_RM_PROT option.  Do you know which
> of the telemetry events could be used to measure this?

No idea. I've only looked at telemetry to see if it had something
helping me debug things. I didn't see anything obvious doing that at the
time. Maybe you could just add the PROT and see what the difference is?

> I recorded telemetry for the nbody benchmark.  It seems that there are
> 16 GC flips.  From one TraceFlipBegin event to the next TraceFlipEnd
> takes about 32 million cycles (I think the timestamps are cycles, as
> returned by rtdsc).  If we assume the clock frequency is 2.5 GHz that
> would be about 13 milliseconds per GC flip.  But I don't know what the
> events actually mean and whether this includes scanning the pdump.

In "Old Design" it says

  7.4.3. The flip phase

  .phase.flip: The roots (see design.mps.root) are scanned. This has to be
  an atomic action as far as the mutator is concerned, so all threads are
  suspended for the duration.

which probably means that between flips all roots are scanned. Unless
there is something "new" meanwhile.

Does telemetry show something concerning root?

I initially thought we could get away with the root not the least
because Emacs as an interactive program probably doesn't require a big
throughput. When I run it with MPS, what I observe doesn't prove this
immediately wrong at least.

But the horrible workarounds like last for the CUs, and the half a dozen
before that, which are only 90%, and the remaining 90% are still out
there. That's so horrible. And on top of all that, I don't have debug
info for elns ;-).

And right now I rememberd pure space. What is with pure space... Not now.

>> This has of course also consequences:
>>
>> - copying 18 Mb of hot objects + 12 Mb or so of leaf objects to MPS
>>   could be slow. No idea if it is. That could impact startup time (not
>>   important to me at all, but people have different preferences).
>
> It would certainly be interesting to know how long it takes.

I could add a DEFUN that allocates a mixture of objects. I think all
vectors of different sizes probably suffices. I don't think the type of
object makes a difference.

>> - copying the graph requires that the copying functions know the layout
>>   of Lisp objects so that the functions can exchange references in the
>>   old graph to the corresponding ones in the new graph. I'm getting
>>   exhausted already from thinking of writing such functions, and we
>>   don't have C++ templates to help.
>
> Do we need to know the layout or can we get away with just knowing the
> size and the relocation information that the pdump already has?

Does it have the size? I wondered that a while back, considering the
marking in the old GC where I got the impression that there is maybe
more info in the pdump.

>> - AFAIK, but see admin/igc.org, there is no good way of allocating
>>   objects in an old generation, so they will maybe take some time to
>>   wander to an older generation.
>
> There is this ramp allocation pattern but it's not exactly what we
> need.

Yeah ;-)

>> Enough rambling.
>>
>> Ideas, opinions, ...?
>
> Does Open Dylan use MPS in some way to dump/load a large amount of
> state?

No idea. Given that Dylan in a way came out of CL, I always assumed it
had the same image-based development model. But I've never used Dylan
myself. I'll try to find out.