Re: Preview: portable dumper

[Daniel Colascione <dancol@dancol.org>]

On Mon, Nov 28 2016, Stefan Monnier wrote:
>> I've been working on a portable dumper for GNU Emacs. The attached patch is
>> an early version of this work.
>
> Cool!
>
>> An additional optimization follows, although this part isn't implemented
>> yet: we can define a "preferred load address" for the dump and write
>> relocation information such that if the dump and Emacs end up being where we
>> expect them to be, we don't have to perform any relocations at all.
>
> Oh, so the in-file format is also a valid in-memory format?  Great.
> AFAIK, that's the crucial difference between a "fast .elc loader" and
> your approach.  This should be key to get really good performance
> (e.g. good enough that it doesn't slow down bootstrap noticeably).

Yes. The file is full of valid memory images of Lisp objects --- that's
why I'm guessing it might be a bit more verbose than ELC,
pre-compression. If you guess correctly at dump time what addresses the
Emacs and dump images will have, you can direct mmap it directly and
run.  With a non-PIC executable, you can do a pretty good job of that
and get performance a bit better than unexec, in theory.

If you load a dump and find that you've guessed wrong on the base
addresses, you need to relocate the image, but that can be done cheaply
and gradually, on access.

>> would benefit from compression, which will reduce IO loads: LZ4 reduces the
>> dump size for me from ~12MB to ~4MB. As in the mmap case, we can throw away
>
> Hmm... interesting the "dump one big .elc" approach I played with (which
> kinda works right now, tho with the known performance issues) leads to
> a dumped.elc file of a size a bit below 4MB (which gzip compresses down
> to ~1MB).
>
> I assume your system is using 64bit Lisp_Object, right?  So on a 32bit
> Lisp_Object your dump size is probably closer to 6-7MB.

Right --- I've tested on 64-bit only. I imagine testing 32-bit and
wide-int might reveal some latent bugs.

>
>> functions pdumper.h --- but even so, it's within 100ms or so of an
>> unexeced Emacs.
>
> 0.1s will still lead to a noticeable slow down of bootstrap, but at
> least it should be good enough for the usual end-user who doesn't run
> Emacs in batch mode.

Oh, of course. But that's without any attention paid to
optimization. I'm sure it can go much faster.

>> It's also possible to dump an already-dumped Emacs, so it should be possible
>> for users to have their own dump files.
>
> [ FWIW in theory my dumped.elc approach could also work on an
>   already-dumped Emacs, for use by end-users (it basically just prints the
>   contents of the obarray), tho it would indeed need extra work to weed out
>   problems printing objects like markers and such.  ]

Yeah --- then both approaches get into the problem of how to dump
arbitrary object graphs, which in practice ends up being "dump the whole
heap except for processes, windows, and frames."