Emacs Lisp's future (was: Guile emacs thread (again))

Emacs Lisp's future (was: Guile emacs thread (again))

[Stefan Monnier]

> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?
>  - Is anyone running it?  How's it going for you?

Good questions.  I've had the opportunity to think a bit more about
Emacs Lisp and its possible evolution and I'm still not sure what to
think about it.

I see a few different options for Emacs Lisp.

First, of course we can keep on evolving Elisp on its own.  This has
worked OK for the last 30 years, so it's not such a terrible choice.
The main problems I see with that:
- Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
  noticed more often.
- Lack of some features, most notably FFI and concurrency.
- Lack of manpower.

This last point is for me the strongest motivation to try and move to
some other system, where we could use other people's work.

One such option is Guile-Emacs.  This presumably would give us a faster
implementation (at least in theory, their bytecode is significantly
more efficient), would give us an FFI, and would give us more manpower
since we'd be benefiting from the work done on Guile.

Note that while Guile does come with support for threading, it doesn't
immediately let us use concurrency in Guile-Emacs, because of all the
issues of synchronizing access to shared data, with all the existing
Emacs code (both C and Elisp) assuming that this problem doesn't exist.
IOW, language support for concurrency is just a first step on the way to
letting Emacs Lisp use concurrency.

Another detail that needs to be spelled out is the difference between
the language and its implementation.  Guile-Emacs provides 2 languages:
Emacs Lips and Scheme (well, it also provides a few more, but that's
not important).  Many people are thinking "cool, so I'll be able to
write extensions in Scheme", but I'm not sure defining Emacs as "this
editor that comes with N extensions languages" is a good idea.

One of the main reasons for Emacs's enduring success is its large set of
third party packages so obviously we can't drop support for Elisp any
time soon.  And as much as I like Scheme, I'm very much unconvinced that
it's really so much better that it's worth converting packages from
Elisp to Scheme.

So if we go for Guile-Emacs, we'll be stuck with Guile, i.e. we'd
have (old and new) packages that use Elisp, new packages that use
Scheme, maybe yet other new packages that use, say, Javascript (or some
other language support by Guile).  That would make the work of Emacs
(and GNU ELPA) maintenance harder.

And of course, if Guile's own manpower dries up, Emacs would be forced
to keep supporting Guile, which is more work than supporting just Elisp.

So, I think that ideally, we'd want to stick to Elisp, or some
evolution thereof.  Sadly, I don't see how to evolve Elisp into Scheme:
they are closely related languages, but the differences are large enough
that it seems hard to reconcile them.

The only standard language into which Elisp can evolve, AFAICT, is
Common Lisp.  [ Now some readers get disappointed, while some others
become excited.  ]  There are some incompatibilities between the two
languages, but I can imagine working them out over the years, or even
living with them without too much trouble, such that we could use
Common-Lisp libraries in Emacs.

Of course, that's for the language side, but on the implementation side,
I don't really know what Common-Lisp implementation we could re-use
(both GNU implementations are dormant, so there's no manpower for us
tap into).  Still: there are many Common-Lisp implementations out there,
so there's probably one that could work for us.


        Stefan

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lennart Borgman]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

On Tue, Sep 16, 2014 at 5:50 PM, Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> The main problems I see with that:
> - Elisp is slow and as CPUs aren't getting faster, its slowness makes
> itself
>   noticed more often.
> - Lack of some features, most notably FFI and concurrency.
> - Lack of manpower.
>

Perhaps also the lack of possibility to enhance Emacs with code written in
other languages? I think for example that Javascript will be something most
future programmers will know. Could Guile make it easier to enhance Emacs
with Javascript (as an alternative to Elisp)?

[-- Attachment #2: Type: text/html, Size: 1152 bytes --]

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 16 Sep 2014 11:50:54 -0400
> Cc: emacs-devel@gnu.org
> 
> One such option is Guile-Emacs.  This presumably would give us a faster
> implementation (at least in theory, their bytecode is significantly
> more efficient)

If by "bytecode" you mean the *.go files, then we should also keep in
mind that these are not architecture-independent as *.elc files are.
Not a catastrophe, of course, but something to remember.

> And of course, if Guile's own manpower dries up, Emacs would be forced
> to keep supporting Guile, which is more work than supporting just Elisp.

A data point: judging by "git log", Guile is currently developed and
maintained by about 3 active developers.

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lars Brinkhoff]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> The only standard language into which Elisp can evolve, AFAICT, is
> Common Lisp.  [ Now some readers get disappointed, while some others
> become excited.  ]

Excited.

> [...] we could use Common-Lisp libraries in Emacs.

That's almost possible already, because there is a Common Lisp
implementation (say 90% complete) for Emacs, including a compiler from
CL to Elisp.

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lally Singh]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

So if this a runtime system issue, what about elisp on LLVM?  Let the LLVM
project handle the backend and performance issues, and emacs can maintain
one language frontend.  There are plenty of people working on that, so
emacs can ride that for almost free.

I'm assuming that there are reasons why it doesn't work, as someone
(apparently) did the work some time ago:
https://github.com/boostpro/emacs-llvm-jit  Perhaps it just needs a little
TLC?

[-- Attachment #2: Type: text/html, Size: 577 bytes --]

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Kristian Nygaard Jensen]

 > Of course, that's for the language side, but on the implementation side,
 > I don't really know what Common-Lisp implementation we could re-use
 > (both GNU implementations are dormant, so there's no manpower for us
 > tap into).  Still: there are many Common-Lisp implementations out there,
 > so there's probably one that could work for us.

Embeddable Common-Lisp (http://sourceforge.net/projects/ecls/) seems 
alive, it is lgpl, so there would be no license issue

-- 

Kristian Nygaard Jensen

Emacs Lisp's future (was: Guile emacs thread (again))

[Nic Ferrier]

Stefan Monnier wrote:

> First, of course we can keep on evolving Elisp on its own.  This has
> worked OK for the last 30 years, so it's not such a terrible choice.
> The main problems I see with that:
> - Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
>   noticed more often.
> - Lack of some features, most notably FFI and concurrency.
> - Lack of manpower.
>
> This last point is for me the strongest motivation to try and move to
> some other system, where we could use other people's work.

I don't see that this is going to happen though. Emacs is an unusual
system. Moving the extension language to another community is just going
to cause more arguing along the lines of "this is how X lang does it" vs
"but we're Emacs and don't want to do it like that".

My view is we should improve the contribution process to get more
manpower for elisp. We have been doing that as a community. As a
reminder we have:

- adopted packaging allowing many more people to contribute pure elisp
- accepted a move to the most commonly used support tools (git, etc...)
- started to talk about changing the documentation format to a more
  common format

I see a new spirit of openness and willingness to change in the Emacs
community and it's really great.

I would implore you, my fellow emacs hackers, not to make too hasty a
decision on platform. Guile-Emacs may be cool, but if we can increase
developer diversity in Emacs through git and so on (I for one will be
contributing to the core thanks to this) then we may get all the
advantages of the Guile VM without having to go to Guile.

I'm sure there is more that we could do to get more man and woman
power. I hope that we consider those things as well as techy projects
like switching to Guile's VM.


Nic

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Tom]

Lally Singh <lally.singh <at> gmail.com> writes:

> 
> So if this a runtime system issue, what about elisp on LLVM?  Let the LLVM
project handle the backend and performance issues, and emacs can maintain
one language frontend.  There are plenty of people working on that, so emacs
can ride that for almost free.

In the long run this would be the most practical solution. Choosing
a well supported and widely used VM which gets  tons of developer 
attention, so the VM development would be taken care of, 
and Elisp could be implemented as a frontend.

This approach has the advantage of supporting the existing Elisp code
base and it would also make it possible to use other languages to extend
emacs, because widdely used VMs have frontend implementations for
many modern languages (Python, etc.)

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It is not acceptable to base a GNU package on LLVM.  It is a
non-copylefted competitor to an important GNU package.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Stefan Monnier]

> base and it would also make it possible to use other languages to extend
> Emacs, because widdely used VMs have frontend implementations for
> many modern languages (Python, etc.)

As mentioned in my previous message about Guile-Emacs, I tend to think
this would be a liability rather than a feature.


        Stefan

Re: Emacs Lisp's future

[Tom]

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

> 
> As mentioned in my previous message about Guile-Emacs, I tend to think
> this would be a liability rather than a feature.
> 

I don't see why. It would make it easier for people to write 
configurations for emacs if they can use a language which they already
know and is widely used.

You could still set up a rule that you only accept elisp packages
in the core if it makes core maintenance easier. But why deny 
this feature from the users if a popular VM can easily support it?

It would be a great selling point if you could say "you can use 
your favorite language to configure emacs, you don't have to use
arcane elisp if you don't want to". It could be an additional 
convenience for the users which would make emacs more accesible
than it is today.

Re: Emacs Lisp's future

[Óscar Fuentes]

Tom <adatgyujto@gmail.com> writes:

> Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
>
>> 
>> As mentioned in my previous message about Guile-Emacs, I tend to think
>> this would be a liability rather than a feature.
>> 
>
> I don't see why. It would make it easier for people to write 
> configurations for emacs if they can use a language which they already
> know and is widely used.
>
> You could still set up a rule that you only accept elisp packages
> in the core if it makes core maintenance easier. But why deny 
> this feature from the users if a popular VM can easily support it?
>
> It would be a great selling point if you could say "you can use 
> your favorite language to configure emacs, you don't have to use
> arcane elisp if you don't want to". It could be an additional 
> convenience for the users which would make emacs more accesible
> than it is today.

LLVM is not a VM (Virtual Machine). I know, the name is confusing.

So LLVM will not allow using multiple programming languages to configure
Emacs.

Re: Emacs Lisp's future

[Tom]

Óscar Fuentes <ofv <at> wanadoo.es> writes:
> 
> LLVM is not a VM (Virtual Machine). I know, the name is confusing.
> 
> So LLVM will not allow using multiple programming languages to configure
> Emacs.
> 

I didn't talk about LLVM specifically, I used the word VM in general.
There are VMs where this is possible, the question is: is there a 
sufficiently popular one (considerable developer community, supports
multiple modern languages, etc.) which is GPL compatible?

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lally Singh]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

Would DotGNU work?  Ignore C# and put in a elisp frontend.  The benefit
being that we can watch what the larger efforts are doing in those fronts,
and pick-and-choose the good ideas with little effort on our side.

DotGNU would get a boost of support from emacs, and it's a familiar enough
environment that a lot of programmers would be willing to go in and hack on
it.  They'd often already have transferrable experience on the proprietary
versions of DotGNU, but would be able to vent their innate need to hack on
dotgnu instead of the more proprietary versions.  Emacs has a huge user
footprint, and leveraging it to boost another project would help.
 Especially a project that rides rather close to a very popular proprietary
platform - that's a lot of hackers to harness.

On Wed, Sep 17, 2014 at 7:43 AM, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> It is not acceptable to base a GNU package on LLVM.  It is a
> non-copylefted competitor to an important GNU package.
>
> --
> Dr Richard Stallman
> President, Free Software Foundation
> 51 Franklin St
> Boston MA 02110
> USA
> www.fsf.org  www.gnu.org
> Skype: No way! That's nonfree (freedom-denying) software.
>   Use Ekiga or an ordinary phone call.
>
>

[-- Attachment #2: Type: text/html, Size: 1981 bytes --]

Re: Emacs Lisp's future

[Stefan Monnier]

> Would DotGNU work?

DotGNU is dead.  But there's still LibJIT.  I have no idea in what kind
of shape it is, tho.


        Stefan

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Jorgen Schaefer]

On Tue, 16 Sep 2014 18:03:17 +0200
Lennart Borgman <lennart.borgman@gmail.com> wrote:

> On Tue, Sep 16, 2014 at 5:50 PM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
> 
> > The main problems I see with that:
> > - Elisp is slow and as CPUs aren't getting faster, its slowness
> > makes itself
> >   noticed more often.
> > - Lack of some features, most notably FFI and concurrency.
> > - Lack of manpower.
> >
> 
> Perhaps also the lack of possibility to enhance Emacs with code
> written in other languages? I think for example that Javascript will
> be something most future programmers will know. Could Guile make it
> easier to enhance Emacs with Javascript (as an alternative to Elisp)?

I think the (often-cited, not just here) idea of supporting multiple
languages is a red herring, mostly. Every extension language supported
adds some burden on those who want to understand what their editor
does, not just use pre-packaged code. One of the great things about
Emacs is that, once I know ELisp, I have a good chance of understanding
and modifying any extension I see. And learning Emacs Lisp is not
exactly hard.

But we do not have to be all theoretical here. There is an editor which
supports a dozen extension languages. The paradoxical thing to notice
when you look at vim plugins is that most of them are written in VimL,
including rather complex ones like NERD-tree and fugitive. I'd argue
that VimL is a tiny bit harder to learn and use than ELisp. There are
various reasons for why most plugins are written in it, but I do think
that this is a pretty good indicator that the lack of "common" languages
for extension is not exactly high on the list of problems for an editor.

There are plenty of things in ELisp itself that I'd put much higher on
that list.

- Lack of a common structured datatype. While there's cl-defstruct, the
  support is a bit limited (C-h f does not work well with it), and a
  lot of code simply does not use it, making it seem a bit like a
  red-haired stepchild instead of a core recommended language feature.
  Alists and plists are usually used where modern languages would use
  structured datatypes, or even some hack with cons cells or lists and
  indexed access.
- Hashes are one of those data types that are used all over the place
  in other languages, but you see them rarely in Emacs Lisp, again often
  losing out to alists and plists. This might be related to the
  standard library functions being a bit baroque. (There's some
  third-party hash library somewhere.)
- Speaking of third-party libraries, s.el, dash.el and f.el provide
  things that really ought to be in core Emacs.
- The regex engine is annoying to use. Providing some interface to PCRE
  would be a great step forward, and does not even have to be
  backwards-incompatible.
- There are tons of warts in Emacs Lisp. nth vs. elt for example,
  with their exciting incompatible calling conventions.

One thing I think would benefit Emacs Lisp as a language a lot would be
a standard library cleanup. An effort to go through the libraries that
come with Emacs, separate them into "standard library" and "extensions
that come with Emacs", and then go through the "standard library",
provide sane names when necessary (like setcar is provided for rplaca)
and deprecating the old ones, or simply deprecate all but one version of
functions with overlapping use (nth or elt, pick one). Finally, add
standard libraries for common functionality that is currently lacking
(see above).

The next step would be going through the "extensions that come with
Emacs" and make sure they all use namespace prefixes for anything but
very specific commands meant for users to use with M-x. Only standard
library functions are allowed to be namespace-free.

These things would make Emacs Lisp a lot easier to use and also easier
to learn for new users.

This is all doable, but it needs manpower (#3 on Stefans list). Which
is manpower that would not be doing other cool stuff on Emacs.

Regards,
Jorgen

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lally Singh]

[-- Attachment #1: Type: text/plain, Size: 2045 bytes --]

On Wed, Sep 17, 2014 at 2:24 PM, Jorgen Schaefer <forcer@forcix.cx> wrote:

> On Tue, 16 Sep 2014 18:03:17 +0200
> Lennart Borgman <lennart.borgman@gmail.com> wrote:
> > Perhaps also the lack of possibility to enhance Emacs with code
> > written in other languages? I think for example that Javascript will
> > be something most future programmers will know. Could Guile make it
> > easier to enhance Emacs with Javascript (as an alternative to Elisp)?
>
> I think the (often-cited, not just here) idea of supporting multiple
> languages is a red herring, mostly. Every extension language supported
> adds some burden on those who want to understand what their editor
> does, not just use pre-packaged code. One of the great things about
> Emacs is that, once I know ELisp, I have a good chance of understanding
> and modifying any extension I see. And learning Emacs Lisp is not
> exactly hard.
>

I think a policy of "if written for emacs, do it in elisp" is a good one,
but let's acknowledge the advantage of easy linking/calling into other code
bases that may come with having a multi-language-compatible runtime system.
 I'm sure we've all seen some systems that we'd love to invoke directly
from elisp.


> [snipping some very good points]

One thing I think would benefit Emacs Lisp as a language a lot would be
> a standard library cleanup. An effort to go through the libraries that
> come with Emacs, separate them into "standard library" and "extensions
> that come with Emacs", and then go through the "standard library",
> provide sane names when necessary (like setcar is provided for rplaca)
> and deprecating the old ones, or simply deprecate all but one version of
> functions with overlapping use (nth or elt, pick one). Finally, add
> standard libraries for common functionality that is currently lacking
> (see above).
>

I completely agree that there's plenty of work needed there, but:
 - If staying with elisp, this is a separate discussion
 - If not staying with elisp, these problems can be addressed during
conversion.

[-- Attachment #2: Type: text/html, Size: 2850 bytes --]

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Alexis]

Jorgen Schaefer writes:

> I think the (often-cited, not just here) idea of supporting multiple
> languages is a red herring, mostly. Every extension language supported
> adds some burden on those who want to understand what their editor
> does, not just use pre-packaged code. One of the great things about
> Emacs is that, once I know ELisp, I have a good chance of
> understanding and modifying any extension I see.

+1. It seems to me that much discussion around this issue focuses on
being able to use one's favourite language to extend Emacs, without
considering what it might be like to:

* start using an extension that seems like it might significantly assist
  one's productivity;
* find a show-stopping bug in it; and
* discover that the extension is written in a programming language one
  loves to loathe.

Also, to an (admittedly very limited) extent, it's /already/ possible to
use a number of languages other than ELisp in Emacs, for at least some
things, via org-babel:

http://orgmode.org/manual/Evaluating-code-blocks.html#Evaluating-code-blocks

A list of languages currently supported by org-babel can be found at:

http://orgmode.org/manual/Languages.html#Languages

i would be interested to know what experiences people might have had in
using org-babel as part of an Emacs extension ....

> Speaking of third-party libraries, s.el, dash.el and f.el provide
> things that really ought to be in core Emacs.

Agreed. i've found it quite surprising that simple functions like
-flatten and s-repeat aren't available in ELisp by default.

> The regex engine is annoying to use. Providing some interface to PCRE
> would be a great step forward, and does not even have to be
> backwards-incompatible.

i like this suggestion. i'm rather comfortable with Perl5 REs, and can
find myself frustrated trying to create REs in ELisp. Having said that,
the issue is not usually the syntax of ELisp REs per se (e.g. needing to
escape things like capturing parentheses or the alternatives pipe); it's
needing to escape various things /further/ because REs can only be
specified in the form of a standard ELisp string. On several occasions
i've ended up using a combination of pcre-to-elisp and re-builder to try
to work out if the problem is too few backslashes, too many backslashes,
or both.


Alexis.

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Emilio Lopes]

2014-09-16 18:03 GMT+02:00 Lennart Borgman <lennart.borgman@gmail.com>:
> Perhaps also the lack of possibility to enhance Emacs with code written in
> other languages? I think for example that Javascript will be something most
> future programmers will know. Could Guile make it easier to enhance Emacs
> with Javascript (as an alternative to Elisp)?

"It's nice that the students coming to us already know Java.  We just
have to teach them how to program."

                -- Michael Sperber