Re: MPS: Loaded pdump

["Gerd Möllmann" <gerd.moellmann@gmail.com>]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>
>>> Ok, I think I'll try to get rid of the pdump root. Maybe I'll regret it,
>>> and maybe I'll fail, but I feel the root is not the right thing to have.
>>> (I'll do in in a branch from the branch in the fork here.)
>>>
>>> Step 1:
>>>
>>> Make the loaded dump traversable by dumping igc_headers.
>>>
>>> How easy or not that is is hard to say. On one hand, there seems to be
>>> dump_object_begin, which looks promising, but I already know one case
>>> (hash tables), where extending that function to write igc_headers might
>>> not be sufficient. And where is one exception there are more. And flags
>>> like pack_objects could interfere, and so on.
>>>
>>> Doenside: igc_header is then even more set in stone. Removing the header
>>> for conses becomes even more work. One would need a new section in the
>>> dump, just for conses.
>>>
>>> Step 1.5:
>>>
>>> When (1) is done, one could make the dump an exact root and use
>>> dflt_scan on it. (That's better than an ambig root, but I don't want the
>>> root because the world is stopped when roots are scanned.)
>>>
>>> Step 2:
>>>
>>> Walk through the dump and make copies of dumped objects in MPS memory.
>>> Record a mapping from dumped -> MPS object. Then walk through the copied
>>> objects and replace references to dumped objects.
>>>
>>> This is a bit hand-wavy. At some point when a dump is loaded, things in
>>> Emacs are set up to refer to objects in the dump. We will have to do our
>>> copy before that and somehow make sure the copy is used instead of the
>>> dump.
>>
>> An update on what's going on:
>>
>> I now have a branch in my fork that builds with and without MPS, where
>> the MPS dump contains igc headers and the non-MPS dump doesn't.
>>
>> Next step will be do ensure that the hot section of the dump is indeed
>> traversable as I want, and fix what's wtong. So we are at step 1.25 or
>> so.
>>
>> Things are a bit slower right now for procrastination reasons. The
>> pdumper is boring as hell :-/. Did I mention that code generation would
>> be a nice thing? Anyway.
>
> I've now pushed someting like step 1.45.
>
> Random notes:
>
> - The dump for MPS now contains the start offsets of igc objects. The
>   existing object_starts relocs cannot be used because they are for Lisp
>   objects only.
>
> - Each igc object in the dump has an igc_header. The function
>   pdumper_visit_object_starts can be used to traverse them in a loaded
>   dump. I chose this interface because it cannot be made sure that igc
>   objects occupy a continguous region in the dump, at least not with
>   surgery on the pdumper.
>
> - Obarrays are probably not yet handled right. I couldn't bring me to do
>   this yet. As you know my fork doesn't have obarrays, but uses CL
>   packages which use hash tables. Any takers?
>
> - There are a number of igc_obj_types of the form IGC_OBJ_DUMPED_xy. I'm
>   not sure what do with these.
>
> - I saw that igc_header::pvec_type is used for something, and shoujld
>   probably tell that was meant as an debugging aid (for the case of
>   hitting IGC_OBJ_FWDs, so that more easily see what type was forwarded.
>   This could also be seen fromt he vectorlike header. I think at some
>   we should remove pvec_type in favor of more hash bits. Whatever, not
>   so important.
>
> - I commented out from .gitignore the .patch files. This is because
>   ignoring .patch files disables some very handy Magit functionality wrt
>   patch handling.
>
> Otherwise this is not yet used. I checked with an MPS and non-MPS
> build (with checking=all).
>
> Taking the next step will be difficult. Copying object from the dump to
> MPS and fixing references from one graph to the other requires writing a
> gazillion functions to do that. At least at the moment that's something
> that exhausts me just be thinking of it.
>
> Whatever. Happy photosynthesizing on a sunny day!

I've transferred more from my fork to GNU, something like step 2.1828.

The dump is still an ambig root, objects are copied to MPS when a dump
is loaded but references in the new graph in MPS are not yet mirrored.
The infrastructure for that is there, but a felt gazillion small
functions need to be written, one for each type :-/.