Re: Preview: portable dumper

[Daniel Colascione <dancol@dancol.org>]

On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: rms@gnu.org, dancol@dancol.org, emacs-devel@gnu.org,
>> burton.samograd@autodesk.com
>> Date: Tue, 29 Nov 2016 10:03:01 -0800
>> 
>> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> EZ> However, a dumper that we control by manually telling it what to dum and
>> EZ> what not doesn't have to do that. IMO, that's just something that doesn't
>> EZ> need to exist in the brave new unexec-free world. (Note that the "one huge
>> EZ> .elc file" method has this built-in.)
>> 
>> I'd still like to see research into the "one huge .elc file" method, for the
>> obvious benefits you listed.
>
> Stefan and Ken made some good progress.  IMO, we should actively
> continue pursuing that goal.
>
>> As far as I understand, the only difference between the portable
>> dumper, and one-huge-.elc, is efficiency. Given infinite resources,
>> the .elc method is much to be preferred, since everything we need
>> "falls out" from existing design.
>
> Any memory dumper will always be more efficient than loading .elc.
> The challenge is to make the difference smaller than the annoyance
> level of the users.  Since initial testing already produces load times
> below 0.5 sec, I don't see why further optimizations won't deliver
> enough speedup to make it fast enough.

Because you can't possible get away, in the ELC approach, from:

 1) reading every damn byte of the file, and
 2) allocating memory for every object you read.

Efficiency is always going to be at least linear in the size of the
"elc dump" --- super-linear complexity if you want read-circle.

A memory dump is linear only in the amount of data you actually
access. Plus, there's no serialization or deserialization overhead. A
memory dump is also inherently gentler on the underlying system, since
we can use many of the pages unmodified (which makes them shareable and
trivially swappable), while every damn byte of the Emacs heap after
loading ELC files will be process-private dirty storage.

I do not think the portable dumper is something that's so complex as to
not be worth the performance, especially since any lread implementation
that's even close to being acceptably fast is going to be very complex
in its own right.