Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Ken Raeburn <raeburn@raeburn.org>]

[-- Attachment #1: Type: text/plain, Size: 6615 bytes --]


On Sep 24, 2017, at 09:57, Philipp Stephani <p.stephani2@gmail.com> wrote:
> Ken Raeburn <raeburn@raeburn.org <mailto:raeburn@raeburn.org>> schrieb am Mo., 3. Juli 2017 um 03:44 Uhr:
> 
> On Jul 2, 2017, at 11:46, Philipp Stephani <p.stephani2@gmail.com <mailto:p.stephani2@gmail.com>> wrote:
> 
>> Ken Raeburn <raeburn@raeburn.org <mailto:raeburn@raeburn.org>> schrieb am Mo., 29. Mai 2017 um 11:33 Uhr:
>> 
>> On May 28, 2017, at 08:43, Philipp Stephani <p.stephani2@gmail.com <mailto:p.stephani2@gmail.com>> wrote:
>> 
>>> 
>>> 
>>> Ken Raeburn <raeburn@raeburn.org <mailto:raeburn@raeburn.org>> schrieb am So., 28. Mai 2017 um 13:07 Uhr:
>>> 
>>> On May 21, 2017, at 04:53, Paul Eggert <eggert@cs.ucla.edu <mailto:eggert@cs.ucla.edu>> wrote:
>>> 
>>> > Ken Raeburn wrote:
>>> >> The Guile project has taken this idea pretty far; they’re generating ELF object files with a few special sections for Guile objects, using the standard DWARF sections for debug information, etc.  While it has a certain appeal (making C modules and Lisp files look much more similar, maybe being able to link Lisp and C together into one executable image, letting GDB understand some of your data), switching to a machine-specific format would be a pretty drastic change, when we can currently share the files across machines.
>>> >
>>> > Although it does indeed sound like a big change, I don't see why it would prevent us from sharing the files across machines. Emacs can use standard ELF and DWARF format on any platform if Emacs is doing the loading. And there should be some software-engineering benefit in using the same format that Guile uses.
>>> 
>>> Sorry for the delay in responding.
>>> 
>>> The ELF format has header fields indicating the word size, endianness, machine architecture (though there’s a value for “none”), and OS ABI.  Some fields vary in size or order depending on whether the 32-bit or 64-bit format is in use.  Some other format details (e.g., relocation types, interpretation of certain ranges of values in some fields) are architecture- or OS-dependent; we might not care about many of those details, but relocations are likely needed if we want to play linking games or use DWARF.
>>> 
>>> I think Guile is using whatever the native word size and architecture are.  If we do that for Emacs, they’re not portable between platforms.  Currently it works for me to put my Lisp files, both source and compiled, into ~/elisp and use them from different kinds of machines if my home directory is NFS-mounted.
>>> 
>>> We could instead pick fixed values (say, architecture “none”, little-endian, 32-bit), but then there’s no guarantee that we could use any of the usual GNU tools on them without a bunch of work, or that we’d ever be able to use non-GNU tools to treat them as object files.  Then again, we couldn’t expect to do the latter portably anyway, since some of the platforms don’t even use ELF.
>>> 
>>> 
>>> Is there any significant advantage of using ELF, or could this just use one of the standard binary serialization formats (protobuf, flatbuffer, ...)? 
>> 
>> That’s an interesting idea.  If one of the popular serialization libraries is compatibly licensed, easy to use, and performs well, it may be better than rolling our own.
>> 
>> I've tried this out (with flatbuffers), but I haven't seen significant speed improvements. It might very well be the case that during loading the reader is already fast enough (e.g. for ELC files it doesn't do any decoding), and it's the evaluator that's too slow.
> 
> What’s your test case, and how are you measuring the performance?
> 
> IIRC I've repeatedly loaded one of the biggest .elc files shipped with Emacs and measured the total loading time. I haven't done any detailed profiling, since I was hoping for a significant speed increase that would justify the work.

It’ll depend on what the code in that file is doing.

In the raeburn-startup branch, the last bit of profiling I did — you can see a graph at http://www.mit.edu/~raeburn/emacs.svg <http://www.mit.edu/~raeburn/emacs.svg> and if you haven’t read up on flame graphs (http://www.brendangregg.com/flamegraphs.html <http://www.brendangregg.com/flamegraphs.html>), they provide a nice visualization of the CPU time consumption broken down by what the current call stack looks like — showed nearly 1/3 of the CPU time of a simple run of Emacs in batch mode was spent reading and parsing the saved Lisp environment.  Most of the rest of the CPU time was spent executing the loaded code (lots of fset and setplist calls), but the biggest chunk of that was executing a nested load of international/characters.elc; during that nested load, most of the time was spent in execution (mostly char table processing) and very little in parsing.

So… for the saved Lisp environment file, excluding the nested load, reading and parsing is about 2/3 of the CPU time used; for characters.elc, reading and parsing is a minuscule portion of the CPU time.

Loading a Lisp file internally uses the Lisp “read” routine, which requires an input stream of character values (not byte values) to be supplied; we examine the stream object and dispatch to various bits of code depending on its type (buffer, marker, function, certain special symbols), *for each character*.  Each byte is examined to see if it’s part of a multibyte character.  Each character is considered to see if it’s allowed to be part of a symbol name or string or whatever we’re in the middle of parsing, or if it’s a backslash quoting some other character, etc.

Hence my hopes for a non-text-based format, designed to streamline reading data from files, where we can do things like specify a vector length or string length up front instead of having to consider each character and process character quoting sequences, stuff like that.  E.g., here’s a unibyte string of 47 bytes, so just copy the bytes without considering every one separately.  No human-readable printed form, no escape sequences needed.

Another help might be finding a faster way to load the character data.  I’ve got the branch loading characters.elc at startup because saving and parsing the generated tables was even slower than evaluating the Lisp code to generate them.  Perhaps we can do some processing of them during the build and convert them into some other form that lets us start up faster.

> If people are generally interested in pursuing this further, I'd be happy to put my code into a scratch branch.

I’d be curious to take a look…

Ken

[-- Attachment #2: Type: text/html, Size: 9862 bytes --]