Re: guile and emacs: unexec

[Andy Wingo <wingo@pobox.com>]

Hi Ken,

On Fri 12 Jun 2009 07:02, Ken Raeburn <raeburn@raeburn.org> writes:

> I'm glad to see the emacs-lisp work is progressing.  As it happens, a
> month or so ago I blew some of the dust off my old guile-emacs project
> and started working on it again too.  This flavor of emacs+guile work
> aimed to replace Lisp objects in Emacs with Guile objects at the  lowest
> level (numbers, cons cells; symbols and such become smobs) and  then
> work upwards from there.

Very interesting! To be clear -- the goal would be to represent as much
of Emacs using cheap Guile structures as possible: numbers and cons
cells and such, and represent specific Emacs objects as smobs? That's
probably a good idea.

Symbols however should probably be represented as Guile symbols, not
smobs. I think that you will find that with a more compilation-centric
approach, we will be able to keep more simple datatypes, as we compile
the procedures that operate on those data types to appropriate code.

> I've updated to recent Emacs sources and Guile 1.8.6. I've gotten it
> to a point where it seems to start up fine in tty mode, reads in (and
> does color highlighting of) C files and directories, does some other
> basic stuff. I'm tweaking it now to see if I can get more stuff
> working (like Cocoa support and "make bootstrap") and do more
> extensive testing.

Very neat! That's fantastic that you were able to get it this far, I
didn't know that was possible.

If this is an effort that you want to pay off in the future, though, I
would strongly suggest updating to the 1.9/2.0 series of Guile. The
expressive range of Guile's multilingual facilities is much higher
there, and significantly different from 1.8.

OTOH, the emacs lisp support is not yet up to the level that it is at in
1.8, so perhaps now is not yet the time.

> One really big hiccup I've run into, which I've sort of sidestepped for
> the moment: Guile is not unexec-friendly.
>
> There is a way to build Emacs so it doesn't use unexec, but it then has
> to load a lot of Lisp code at run time, really killing the startup
> performance, and I don't think it's tested all that much (e.g., "make
> bootstrap" doesn't work even without the Guile hacks).  To really make
> this project work, I need to be able to link against Guile (static is
> fine, and probably necessary), do a bunch of Lisp/Scheme processing,
> write out a memory image into a new executable, and later be able to
> run that executable.

It's true that Guile doesn't do unexec currently. It might in the
future -- obviously it will if you implement it of course ;)

But I would ask that you reconsider your approach to making Guile-Emacs
load quickly. There is no a priori reason that loading Lisp code should
be slow. With Guile-compiled elisp, loading a file is just mapping it
into memory -- the same as you have with an image. The loaded code needs
to be run to establish definitions, but that is a very quick operation.

I agree that heap saving could be slightly faster. But I think that
Emacs should be able to load from bytecode within 100 ms or so /with the
current Guile-VM code/ -- and even faster if we do native ahead-of-time
compilation at some point.

> Any record of current threads needs to go away, and be replaced with
> info on the new one-and-only thread in the new process; I'm building
> without thread support for now to get around it.  Any record of stack
> regions to be scanned for SCM objects likewise needs resetting.
> Allocated objects must *not* go away, and must continue to be  processed
> by the garbage collector, so I can't just reinitialize  everything.
> Assigned smob types must remain in effect, and for now  I'm ignoring the
> possibility that some smobs may need some kind of  reinitialization.
> Mutexes... well, I don't know if they need  reinitializing; POSIX is
> kind of unclear on interactions with  unexec. :-)  I expect
> reinitializing them is probably safe, even if  not required in some
> implementations.

This could be complicated if we merge in the BDW-GC branch, to use
libgc. Note that SCM does have unexec, IIRC, we could steal parts of
their implementation

> Is this something that could be useful to anyone outside of Emacs?

Unexec certainly could, to deliver self-contained binaries. But TBH I
think the booting-from-compiled-files option is more maintainable. In
any case this would be a neat hack. Have fun! :)

> P.S.  If anyone wants to take a look at my current work,
> http://www.mit.edu/~raeburn/guilemacs/guile-emacs.tar.bz2
>  has a snapshot from tonight.

Cool! Have you considered using git, and branching from Emacs' git
mirror? That way it is trivial to set up something other people can
comment on, in easily-digestible patch chunks.

Happy hacking,

Andy
-- 
http://wingolog.org/