Re: Preview: portable dumper

[Daniel Colascione <dancol@dancol.org>]

On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: Richard Stallman <rms@gnu.org>
>> CC: dancol@dancol.org, emacs-devel@gnu.org
>> Date: Tue, 29 Nov 2016 11:55:16 -0500
>> 
>> Maintaining the portable dumper will be far simpler than maintaining
>> unexec.  Unexec has to write a file in whatever format the system uses
>> for executables.  If that is ELF format, it is very complicated,
>> and it wasn't designed for this purpose.  It is necessary to copy
>> the material from the temacs executable, and modify it along the way.
>> 
>> All those problems will be absent for the portable dumper, which
>> will make its maintenance much easier.
>
> Easier: yes.  Easy enough: no, not IMO.  Not for most of the Emacs
> contributors we have and will have in the near future.

Who do you imagine maintaining Emacs? CS 101 students? We have to assume
a certain baseline level of skill on the part of our future
colleagues. I don't want to dumb down our implementation for the sake of
people we don't know and who, for all we know, may never
actually appear.

> The number of people aboard who can matter-of-factly hack the Emacs
> internals on the C level is consistently going down, and is already so
> small they can be counted on one hand.  We must make Emacs depend less
> on people from this small and diminishing group, if we want the
> development pace increased or even kept at its current level.

I've discussed before why I think this argument is ill-founded. We
already have examples over the past few years of people who've
familiarized themselves with Emacs internals in order to add a feature
they want. If there's a need, people will familiarize themselves with
the editor.

Please, present some evidence that the reason we've seen a decline in
core contributions due to the complexity of the current codebase. I
don't agree that we'd see more contributors appear if we simplified our
algorithms. On the contrary, making Emacs slower will drive users away.

The solution to the problem you're raising is better internals
documentation, not oversimplification.

> To me,
> that means keep as many features out of low-level C, and limit futzing
> with C-level internals of Lisp objects and the Lisp interpreter to the
> absolute minimum.  IOW, any feature that can have an alternative
> implementation in Lisp should not be done in C, even if the Lisp
> implementation will be slightly slower.

I agree that what *can* be done in Lisp should be done in Lisp. We're
talking about a facility that doesn't fall under this general
guideline. It's the job of the C core to provide to Lisp facilities
necessary to implement higher-level features.  Fast startup is one of
these platform facilities, one very common in other Lisp systems.

>> 		      On top of that, adding Lisp objects will now require
>>   > writing its dumper back-end, so this will be a constant maintenance
>>   > burden of the kind that only a few of us can bear.
>> 
>> Yes, but how often do we make such changes?  Once every few years,
>> I think.
>
> More like once a year, maybe more.

Like what? The last two new types in core have been finalizers and the
module thing, both of which have been trivial Lisp_Misc types.

By the way: it's very easy to add support for new object types
to pdumper. Have you read the code?

There are certain types of object that are especially easy to support:

Do you have a pseudovector with no non-Lisp allocations? Adding support
to pdumper for this object type is one line of code.

Do you have a pseudovector that you want to come back from dumps "dead",
like frames? Adding support to pdumper for this object type is one line
of code.

How many new objects types are we going to add that don't fall under one
of these two categories? You're optimizing for hypotheticals at the
expense of the concrete present.

>
>> And it won't be a big difficulty to do so, if the code of the dumper
>> is clean.
>
> For you and me, and for a few others, sure.  But Emacs cannot be
> developed by a handful of people anymore.  We already fall behind with
> frequency of releases, with the speed of delivering new features, with
> analyzing and fixing bugs reported to the tracker, etc.  If we don't
> attract more core developers, we will never get out of this vicious
> cycle.  And bringing C-level core developers on board is a very slow,
> painful, and unreliable process, because C programmers are a dying
> breed, and Emacs internals are not for the faint at heart.

So move more functionality to Lisp. We've done a good job of doing that
lately. We're talking about a special case. You can't move eval to
Lisp either.

If you want to pursue this path, how about moving redisplay to Lisp? You
know more about redisplay than anyone else around, I think, and
redisplay is a much hairier blob of C code than the portable dumper is.

>> I am sure it will be a lot cleaner than the code of unexec.
>
> I'm sure, too, but that's not the issue.
>
>>   > Making the initial load of preloaded Lisp files (most probably, a
>>   > single large file) fast enough to allow us to dump the dumping phase
>>   > altogether is a much more perspective direction.
>> 
>> I think this has its own difficulty -- the need to represent everything
>> in Lisp syntax.
>
> I don't think I follow: what we preload in loadup already has Lisp
> syntax.

Yes, but the data structures that we create via loadup don't all have
easy-to-digest Lisp representations, and creating them is on the same
level of complexity as the portable dumper.

*This* code exists. Where is the hypothetical sufficiently-fast lread?