Re: Skipping unexec via a big .elc file

[Eli Zaretskii <eliz@gnu.org>]

> Cc: p.stephani2@gmail.com, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 25 Oct 2016 09:14:55 -0700
> 
> >> Everyone who's seriously thought about the unexec problem _understands_
> >> the issue.
> >
> > The important point is that the number of people here who can claim
> > such understanding, enough so to fix the issues, is diminishingly
> > small, and gets smaller every year.
> 
> There's no demand for more yet.

Not true.  Demand for this level of expertise is continuous in Emacs,
and never dwindles, not for the last 25 years that I'm involved.

> There used* to be a lot more (at least 
> per-capita) stonemasons in historical societies than in today's society. 
> That doesn't mean we've forgotten how to cut stones, and if there were a 
> sudden need to do it, more stonemasons would magically appear.

I think your optimism is misplaced.  I'm old enough to have seen
several proficiencies go extinct due to new technology that made them
irrelevant.  When demand for those forgotten proficiencies came up,
people invariably run to the few still around who know how to do that,
they don't learn that themselves (and don't even know how).

> > Why do you think this will have better performance that reading a
> > single .elc file at startup?  It's still mainly file I/O and
> > processing of the file's contents, just like with byte-compiled files.
> 
> Because a portable dumper can do less, on both file I/O and processing 
> of the file's contents. There's no lisp evaluation, no slurping a whole 
> file into memory. Having to read all of Emacs into memory on startup is 
> a burden even on a fast, modern machine like mine.
> 
> ~/edev/trunk/src
> $ sync && sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> 
> ~/edev/trunk/src
> $ time pv < emacs >/dev/null
> 48.6MiB 0:00:00 [ 455MiB/s] 
> [=========================================================>] 100% 
> 
> 
> real	0m0.116s

Which is definitely comparable with my measurements of loading all of
the *.elc files concatenated, which were proclaimed to be "too slow".

> > If we have no reason to believe this portable dumper will be
> > significantly faster, we should IMO investigate the .elc method first,
> > because it's so much simpler, both in its implementation and in future
> > maintenance.  E.g., adding a new kind of Lisp object to Emacs would
> > require corresponding changes in the dumper.
> 
> Adding a new kind of lisp object requires changes throughout core 
> anyway.

The changes in the dumper are _in_addition_ to that.

> > Demand paging in an application, and an application such as Emacs on
> > top of that, makes little sense to me.
> 
> Why? It's conceptually no different from autoload.

The devil is in the details, though.  And there are a lot of details
in this case that are completely unrelated to the concept.  If you
don't get them all right, you get a subtly unstable application that
will crash randomly in hard to reproduce and debug situations.

> > This is the OS business, not
> > ours.  Using mmap as a fast way to read a file, yes, that's done in
> > many applications.  But please lets leave demand paging out of our
> > scope.
> 
> Emacs isn't just an application. It's a Lisp virtual machine

No, it's not.  It's an application with a powerful extension language.

> (FWIW, mmap isn't a particularly fast way of doing bulk file reads. 
> That's why GNU grep removed its mmap support.)

It was an example of a low-level technique that is sufficiently simple
to use, that's all.

> > IMO the less we mess with low-level techniques that no other
> > applications use the better, both because we have very few people who
> > can do that and because doing so runs higher risk of becoming broken
> > by future developments in the platforms we deem important.  The
> > long-term tendency in Emacs development should be to move away from
> > such techniques, not to acquire more of them.
> 
> I'm for anything that delivers meaningful performance advantages.

IME, that way lies madness.  It's the exact opposite of the direction
Emacs should evolve if we want to prevent it from becoming a marginal
package for a few enthusiasts.