Emacs Lisp's future (was: Guile emacs thread (again))

[Stefan Monnier <monnier@iro.umontreal.ca>]

> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?
>  - Is anyone running it?  How's it going for you?

Good questions.  I've had the opportunity to think a bit more about
Emacs Lisp and its possible evolution and I'm still not sure what to
think about it.

I see a few different options for Emacs Lisp.

First, of course we can keep on evolving Elisp on its own.  This has
worked OK for the last 30 years, so it's not such a terrible choice.
The main problems I see with that:
- Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
  noticed more often.
- Lack of some features, most notably FFI and concurrency.
- Lack of manpower.

This last point is for me the strongest motivation to try and move to
some other system, where we could use other people's work.

One such option is Guile-Emacs.  This presumably would give us a faster
implementation (at least in theory, their bytecode is significantly
more efficient), would give us an FFI, and would give us more manpower
since we'd be benefiting from the work done on Guile.

Note that while Guile does come with support for threading, it doesn't
immediately let us use concurrency in Guile-Emacs, because of all the
issues of synchronizing access to shared data, with all the existing
Emacs code (both C and Elisp) assuming that this problem doesn't exist.
IOW, language support for concurrency is just a first step on the way to
letting Emacs Lisp use concurrency.

Another detail that needs to be spelled out is the difference between
the language and its implementation.  Guile-Emacs provides 2 languages:
Emacs Lips and Scheme (well, it also provides a few more, but that's
not important).  Many people are thinking "cool, so I'll be able to
write extensions in Scheme", but I'm not sure defining Emacs as "this
editor that comes with N extensions languages" is a good idea.

One of the main reasons for Emacs's enduring success is its large set of
third party packages so obviously we can't drop support for Elisp any
time soon.  And as much as I like Scheme, I'm very much unconvinced that
it's really so much better that it's worth converting packages from
Elisp to Scheme.

So if we go for Guile-Emacs, we'll be stuck with Guile, i.e. we'd
have (old and new) packages that use Elisp, new packages that use
Scheme, maybe yet other new packages that use, say, Javascript (or some
other language support by Guile).  That would make the work of Emacs
(and GNU ELPA) maintenance harder.

And of course, if Guile's own manpower dries up, Emacs would be forced
to keep supporting Guile, which is more work than supporting just Elisp.

So, I think that ideally, we'd want to stick to Elisp, or some
evolution thereof.  Sadly, I don't see how to evolve Elisp into Scheme:
they are closely related languages, but the differences are large enough
that it seems hard to reconcile them.

The only standard language into which Elisp can evolve, AFAICT, is
Common Lisp.  [ Now some readers get disappointed, while some others
become excited.  ]  There are some incompatibilities between the two
languages, but I can imagine working them out over the years, or even
living with them without too much trouble, such that we could use
Common-Lisp libraries in Emacs.

Of course, that's for the language side, but on the implementation side,
I don't really know what Common-Lisp implementation we could re-use
(both GNU implementations are dormant, so there's no manpower for us
tap into).  Still: there are many Common-Lisp implementations out there,
so there's probably one that could work for us.


        Stefan