Re: Emacs Lisp's future

[Robin Templeton <robin@terpri.org>]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The only standard language into which Elisp can evolve, AFAICT, is
> Common Lisp.  [ Now some readers get disappointed, while some others
> become excited.  ]  There are some incompatibilities between the two
> languages, but I can imagine working them out over the years, or even
> living with them without too much trouble, such that we could use
> Common-Lisp libraries in Emacs.
>
> Of course, that's for the language side, but on the implementation side,
> I don't really know what Common-Lisp implementation we could re-use
> (both GNU implementations are dormant, so there's no manpower for us
> tap into).  Still: there are many Common-Lisp implementations out there,
> so there's probably one that could work for us.

There is: Guile-Emacs combined with an improved and extended `cl'
library, plus language extensions needed for Common Lisp compatibility,
many of which have native support in the Guile VM.

I've also considered the possibility of porting Emacs to a Common Lisp
implementation; it does come up in Lisp circles once in a while. I think
that route would entail difficulties that don't apply to Guile-Emacs,
for a few different reasons.

First and most obviously, Guile is also a GNU project, and therefore
shares certain technical and political priorities with Emacs, which
isn't necessarily a given elsewhere.

Consider the Common Lisp implementations that could work for Emacs:
those that are free software and are actively developed with large user
communities. SBCL is one such Lisp. It's a first-rate Lisp
implementation with an excellent compiler, wide adoption in the free
software Lisp community and lots of compatible libraries. But it's not a
good fit for Emacs's needs. First, it's a compiler-only implementation
with a large and complex runtime system including garbage collectors,
native-code compilers, assemblers, etc. that exists largely
independently of the host *nix environment. And as a result, it's
monstrously difficult to port to new platforms. It's also not at all
designed to be an extension language, and the current version of the
manual says: "Calling Lisp functions from C is sometimes possible, but
is extremely hackish and poorly supported as of SBCL 0.7.5."

AFAICT, similar considerations apply to Clozure CL, the only other good
candidate I know of, although it does at least support callbacks from C.

Beyond that, it's true that Common Lisp is more closely related to Elisp
than Scheme is, and Common Lisp and Elisp have a greater degree of
overlap in their semantics and standard libraries. Even so, I believe
that the Guile platform is a superior choice as a foundation for Emacs
and Emacs Lisp.

Common Lisp was the first standardized object-oriented language and was
shaped by decades of experience with traditional Lisp systems, and it
would be beneficial for Elisp to become more compatible with Common
Lisp. Nonetheless, Common Lisp tends to agglutinate logically distinct
features into components which are rather monolithic. That's not a bad
thing in most contexts, but makes things trickier for Emacs and Elisp.
Scheme, on the other hand, is famously designed according to the
principle that

  "Programming languages should be designed not by piling feature on top
   of feature, but by removing the weaknesses and restrictions that make
   additional features appear necessary,"

and this has shaped the design of Guile, due to the primacy of Scheme in
the Guile implementation and community.

A concrete example: Common Lisp and Elisp both include almost identical
facilities for special variables. Guile Scheme doesn't; what it has are
fluids, a library for thread-local dynamic bindings independent of the
core Scheme language, which have a rather different interface. So that
looks like a win for Common Lisp and a point against Guile when it comes
to dynamic binding.

However, once you take into consideration the details of the apparently
slight gap between the languages, the overlap in design begins working
against you. Elisp provides various kinds of context-sensitive
variables, notably buffer-local variables, and thousands of packages
rely on the availability of this facility. But Common Lisp doesn't give
you the tools to implement it. What Common Lisp gives you is something
that has a strong resemblance to the corresponding part of Elisp, but
which isn't actually the same thing. And because it's a fundamental part
of the language, compilers that have been tuned for 20-30 years to
support a single standardized language optimize for a specific special
variable semantics. So, if you want to support buffer-local variables,
you have two basic choices:

(1) You can modify Common Lisp: first figure out exactly how special
variables are implemented in SBCL's compiler and runtime, how to extend
them to support Elisp semantics without breaking any legal Lisp
programs, and how to permit optimization of the use of standard special
variables without breaking the behavior of code using BLVs.

(2) Or you can DIY: implement your own special-variable library that
supports BLVs. Because Common Lisp defines a fixed high-level interface
for special variables, you won't get much help from the host language.
In this case, the apparent similarity between Elisp and Common Lisp
doesn't actually lead to shared functionality, and interaction between
the two languages will be less natural than it could be.

With Guile, the situation is different. When ad hoc extensions to core
functionality are necessary, the situation no worse than with Common
Lisp, and often better because Guile is typically more modular. But ad
hoc extensions often aren't necessary at all, because Guile takes a
fundamentally different approach to language design.

In the case of buffer-local variables, Guile provides delimited
continuations, which are powerful enough to implement both dynamic
binding and BLVs as libraries, without changing the semantics of Scheme
and therefore without disrupting the operation of other programs. Using
delimited continuations and a fluids library, one can implement
context-local bindings in under a hundred lines of code. A
proof-of-concept was presented at GHM 2011; a version compatible with
Elisp BLVs would probably expand to several hundred lines.

Another example: Common Lisp defines a standard package system that may
need extensions or modifications to work well with Elisp, while Guile
provides a module system designed for Scheme but also exposes enough of
the internal functionality to build other namespace systems. Guile-Elisp
makes use of this to compile a Lisp-2 dialect without any special
support from Guile, and it could be used as the basis for an
Elisp-specific module system.

And this pattern repeats over and over in the implementation of Elisp in
the context of a standard Common Lisp system. The small differences turn
out to matter; the family reunion becomes a feud. It's definitely
possible to port Emacs to a Common Lisp implementation; with enough work
and enough time, it could be done and done well. But it won't
necessarily be simple or straightforward, and it's unlikely to be easier
than porting Emacs to the Guile platform, a project which has progressed
from running standalone games of "dunnet.el" to supporting the complete
Emacs system with less than a person-year of work.

Scheme's turn towards minimalism may seem to be something of only
academic appeal, but it affords a freedom of expression that turns out
to be extremely practical. Guile gives you access to the fundamental
abstractions needed to express entire classes of high-level features,
and this makes it uniquely qualified to serve as a platform for multiple
Lisp dialects.

Robin
-- 
Inteligenta persono lernas la lingvon Esperanton rapide kaj facile.
Esperanto estas moderna, kultura lingvo por la mondo. Simpla, fleksebla,
belsona, Esperanto estas la praktika solvo de la problemo de universala
interkompreno. Lernu la interlingvon Esperanton!