guile and emacs and elisp, oh my!

guile and emacs and elisp, oh my!

[Andy Wingo]

Greetings, all!

My name is Andy, and together with Ludovic Courtès I co-maintain Guile.

I meant to put off writing this note until some months in the future,
but as people seem to be interested in the topic, now is as good a time
as any.

The topic? Emacs and Guile. (Cue cheering and derisions!) I'll be
telling some history here, from the perspective of one who wasn't there,
but I believe I have the essentials right.

Emacs and Guile have a long and lovely history, longer in fact than
Guile's existence itself. Hacking Emacs is the core of GNU -- as you
know, Emacs is alive, it's fungible, it's free. The GNU system grew out
of Emacs and the hack-experience that it represents.

(I have a friend who regularly chides me when I have to reboot and shut
down my Emacs -- it's like I'm killing a friend, he says!)

But when it was decided that GNU should be like a Unix, it was a fall
from grace, in a sense. The system no longer had the Emacs nature. Time
passed, and people looked to see how to spread the Emacs nature from
Emacs itself to the rest of the programs in the GNU system.

One possibility would be to extract elisp from emacs and use it as an
extension language. But given that the language would have to be
extracted anyway, the thought was to use a more modern Lisp dialect,
Scheme. Thus Guile was born, as the GNU extension language.

(Its birth coincided unfortunately with the so-called "Tcl wars", but
Guile was not made because of Tcl. Guile was made out of a need that
existed outside the polemic. For example, Neil Jerram -- recently
retired Guile maintainer, may he hack in peace -- mentioned to me once
that he had contacted Richard quite early on, before the "Tcl wars",
discussing the need for a GNU extension language, as Elisp was to
Emacs.)

But I digress! Tom Lord was the one to make the initial Guile, derived
from SCM as you know, and though his time with Guile was too short, we
are privileged to have traces of his code in Guile, and his ethereal
presence here on this list.

Since then, about 15 years ago now, Guile has had many maintainers. It
was stabilized, and documented, and extended, and hacked, and fixed, but
eventually we realized that Guile's interpreter -- which traversed trees
of expressions to produce its effects and values -- was just too slow.
We needed to do some pre-computation on the source; we needed to compile
it to something.

Given that the U is for ubiquitous (*not* Unique, all you salt-n-pepa
fans), we couldn't do native compilation, not directly anyway -- we
needed something portable. So, with the help of a few hackers, I updated
some work that Keisuke Nishida did on writing a VM and associated
compiler for Guile, and that's what we're going to release hopefully
next month as Guile 2.0.

We've been trying hard to maintain backwards compatibility for Guile,
and we've done well, I think. The move from interpreted to compiled code
has all of the eval-when pitfalls that you all know, but most extensions
just compile fine with 2.0, and practically all correct Scheme code -- a
number of errors were not caught with 1.8 and before -- practically all
of that code runs fine without modification.

If you really want the whole story, I'd advise you to see my
presentation at November's GNU Hackers Meeting in Sweden, which can be
viewed or downloaded here:

  http://wingolog.org/archives/2010/04/02/recent-developments-in-guile



Hi! Still with me? Cool :)

I've said all of this to give some credibility to what I'm going to say
now.

Guile can implement Emacs Lisp better than Emacs can.

We can compile Emacs Lisp to Guile's VM, and make C shims to present a
suitably compatible C interface to the rest of Emacs' C primitives.

No one will notice! Except that after a switch, Emacs would be faster,
more powerful, and have the ability to access all of Guile's facilities
-- the Scheme language, other languages implemented for Guile
(Javascript, Lua, ...), a proper ffi, dynamically loadable libraries, a
module system, the numeric tower (rationals, bignums, etc), Guile's
existing libraries, delimited continuations (!), fast bytevector access,
native threads, etc.

On a language level, I am convinced that not only can Guile implement
Emacs Lisp, but it can provide a coherent story for tight interaction
between e.g. minor modes in Javascript and Scheme and Elisp, etc. More
details will come later, though feel free to bring up specific points.

There have been some suggestions on the list that "Guile people want to
rewrite Emacs in Scheme". Though Scheme does have some facilities to
offer, personally I do not believe in rewriting software. I've failed
enough times, thank-you-very-much -- and if we could retrofit a compiler
into Guile in two years via correctness-preserving transformations,
surely we can replace the implementation of Elisp without causing
problems.

My goal is to make it the obvious decision for the Emacs maintainers to
say, "yes, let's switch to Guile's implementation of elisp, because it
will make Emacs better and have no drawbacks". We're not there yet --
and that's why I wanted to put off this mail -- but we'll get there.

I guess that's all I have to say, for now -- my bandwidth is not with
Emacs right now, as there are enough things to take care of in Guile
itself. But, I'll be responding to this thread, and lurking on the
list, so do get in contact if you have a concern.

In the meantime, happy hacking!

Andy
-- 
http://wingolog.org/

Re: guile and emacs and elisp, oh my!

[christian.lynbech]

>>>>> "Andy" == Andy Wingo <wingo@pobox.com> writes:

Andy> ... We're not there yet -- and that's why I wanted to put off this
Andy> mail -- but we'll get there.

This sounds exciting. 

I do hope that I managed to make it clear that my own Guile experiences
was rather dated. I did not mean to dis Guile in any way. My personal
language preferences aside, I think this will provide a significant
boost for emacs.


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: guile and emacs and elisp, oh my!

[joakim]

Andy Wingo <wingo@pobox.com> writes:

> Greetings, all!
>
> My name is Andy, and together with Ludovic Courtès I co-maintain Guile.
>
> I meant to put off writing this note until some months in the future,
> but as people seem to be interested in the topic, now is as good a time
> as any.
>
> The topic? Emacs and Guile. (Cue cheering and derisions!) I'll be
> telling some history here, from the perspective of one who wasn't there,
> but I believe I have the essentials right.
>
> Emacs and Guile have a long and lovely history, longer in fact than
> Guile's existence itself. Hacking Emacs is the core of GNU -- as you
> know, Emacs is alive, it's fungible, it's free. The GNU system grew out
> of Emacs and the hack-experience that it represents.
>
> (I have a friend who regularly chides me when I have to reboot and shut
> down my Emacs -- it's like I'm killing a friend, he says!)
>
> But when it was decided that GNU should be like a Unix, it was a fall
> from grace, in a sense. The system no longer had the Emacs nature. Time
> passed, and people looked to see how to spread the Emacs nature from
> Emacs itself to the rest of the programs in the GNU system.
>
> One possibility would be to extract elisp from emacs and use it as an
> extension language. But given that the language would have to be
> extracted anyway, the thought was to use a more modern Lisp dialect,
> Scheme. Thus Guile was born, as the GNU extension language.
>
> (Its birth coincided unfortunately with the so-called "Tcl wars", but
> Guile was not made because of Tcl. Guile was made out of a need that
> existed outside the polemic. For example, Neil Jerram -- recently
> retired Guile maintainer, may he hack in peace -- mentioned to me once
> that he had contacted Richard quite early on, before the "Tcl wars",
> discussing the need for a GNU extension language, as Elisp was to
> Emacs.)
>
> But I digress! Tom Lord was the one to make the initial Guile, derived
> from SCM as you know, and though his time with Guile was too short, we
> are privileged to have traces of his code in Guile, and his ethereal
> presence here on this list.
>
> Since then, about 15 years ago now, Guile has had many maintainers. It
> was stabilized, and documented, and extended, and hacked, and fixed, but
> eventually we realized that Guile's interpreter -- which traversed trees
> of expressions to produce its effects and values -- was just too slow.
> We needed to do some pre-computation on the source; we needed to compile
> it to something.
>
> Given that the U is for ubiquitous (*not* Unique, all you salt-n-pepa
> fans), we couldn't do native compilation, not directly anyway -- we
> needed something portable. So, with the help of a few hackers, I updated
> some work that Keisuke Nishida did on writing a VM and associated
> compiler for Guile, and that's what we're going to release hopefully
> next month as Guile 2.0.
>
> We've been trying hard to maintain backwards compatibility for Guile,
> and we've done well, I think. The move from interpreted to compiled code
> has all of the eval-when pitfalls that you all know, but most extensions
> just compile fine with 2.0, and practically all correct Scheme code -- a
> number of errors were not caught with 1.8 and before -- practically all
> of that code runs fine without modification.
>
> If you really want the whole story, I'd advise you to see my
> presentation at November's GNU Hackers Meeting in Sweden, which can be
> viewed or downloaded here:
>
>   http://wingolog.org/archives/2010/04/02/recent-developments-in-guile
>
>
>
> Hi! Still with me? Cool :)
>
> I've said all of this to give some credibility to what I'm going to say
> now.
>
> Guile can implement Emacs Lisp better than Emacs can.
>
> We can compile Emacs Lisp to Guile's VM, and make C shims to present a
> suitably compatible C interface to the rest of Emacs' C primitives.
>
> No one will notice! Except that after a switch, Emacs would be faster,
> more powerful, and have the ability to access all of Guile's facilities
> -- the Scheme language, other languages implemented for Guile
> (Javascript, Lua, ...), a proper ffi, dynamically loadable libraries, a
> module system, the numeric tower (rationals, bignums, etc), Guile's
> existing libraries, delimited continuations (!), fast bytevector access,
> native threads, etc.
>
> On a language level, I am convinced that not only can Guile implement
> Emacs Lisp, but it can provide a coherent story for tight interaction
> between e.g. minor modes in Javascript and Scheme and Elisp, etc. More
> details will come later, though feel free to bring up specific points.
>
> There have been some suggestions on the list that "Guile people want to
> rewrite Emacs in Scheme". Though Scheme does have some facilities to
> offer, personally I do not believe in rewriting software. I've failed
> enough times, thank-you-very-much -- and if we could retrofit a compiler
> into Guile in two years via correctness-preserving transformations,
> surely we can replace the implementation of Elisp without causing
> problems.
>
> My goal is to make it the obvious decision for the Emacs maintainers to
> say, "yes, let's switch to Guile's implementation of elisp, because it
> will make Emacs better and have no drawbacks". We're not there yet --
> and that's why I wanted to put off this mail -- but we'll get there.
>
> I guess that's all I have to say, for now -- my bandwidth is not with
> Emacs right now, as there are enough things to take care of in Guile
> itself. But, I'll be responding to this thread, and lurking on the
> list, so do get in contact if you have a concern.

I find this very interesting. Do you have some form of plan how to
integrate this form of Guile and Emacs? Can it be done incrementally,
that is, can the current elisp system and the guile system coexist in
the same Emacs? If so, more hackers could help out with the integration,
maybe.


>
> In the meantime, happy hacking!
>
> Andy
-- 
Joakim Verona

Re: guile and emacs and elisp, oh my!

[Andy Wingo]

Hi,

On Thu 15 Apr 2010 10:53, joakim@verona.se writes:

> Do you have some form of plan how to integrate this form of Guile and
> Emacs? Can it be done incrementally, that is, can the current elisp
> system and the guile system coexist in the same Emacs? If so, more
> hackers could help out with the integration, maybe.

It's a bit vague, currently; we need to finish the Elisp compiler, and
then check on the state Ken Raeburn's Guilemacs branch, and try to
figure out how to do the C shims; lather, rinse, repeat. There has been
one summer of code student last year, Daniel Kraft, and hopefully I'll
have one this year as well.

But Guile isn't quite ready yet. I feel like with a refactoring like
this, once we're ready it will take one person a few months to get
something workable. It has to fit in the head of one person I think.

But we'll see!

Andy
-- 
http://wingolog.org/

Re: guile and emacs and elisp, oh my!

[Ken Raeburn]

On Apr 15, 2010, at 04:53, joakim@verona.se wrote:
> I find this very interesting. Do you have some form of plan how to
> integrate this form of Guile and Emacs? Can it be done incrementally,
> that is, can the current elisp system and the guile system coexist in
> the same Emacs? If so, more hackers could help out with the integration,
> maybe.

I think I've written up some email in the past (you may have to search back several months) about this.  My project -- changing Emacs to use Guile object types (at least the simplest ones) and Guile's allocator and GC -- can be done somewhat incrementally.  E.g., make sure Emacs uses accessor functions or macros for all uses of type X outside of its own GC system (those changes can be done upstream), and passes around Lisp_Object values rather than pointers to struct Lisp_Foo (likewise), then redefine type X's implementation using more appropriate Guile types, then work on type Y... currently strings, symbols, vectors, and hashes are on my list; individual "misc" and "vectorlike" types can come later, maybe.

I have no intention of getting rid of elisp from emacs, most certainly.  As for eventually replacing the entire Lisp implementation in Emacs with the Guile one, well, I'm hoping that, like Andy suggested, it'll become an obvious win to do so, but if we wind up for a while with two distinct language interpreters (Emacs elisp and Guile VM) but a shared object system and working communication between languages, while Andy madly tries to fix whatever performance problem we might have run into with Guile's elisp implementation :-), I'd be okay with that.

I've recently updated my code to be based on a recent development version of Emacs, though I haven't published the updated version yet and there appears to be some kind of GC problem or something at the moment.  I also haven't got it fully updated for guile 1.9.x (and the upcoming 2.0), which has a different GC system anyways.  (The new GC system is Boehm's GC library.  There was an effort to make Emacs use it some time back, by Dave Love, and I think some of those changes will be needed for Emacs-with-Guile-2.0.  So updating that work might also be helpful?)

But, now that we've got bzr upstream and branching should be relatively easy, perhaps I should clean it up a little and push it all into a branch on savannah, if the maintainers are interested?  If there's really interest in working on it, I'd love to get some help....

Ken

Re: guile and emacs and elisp, oh my!

[joakim]

Ken Raeburn <raeburn@raeburn.org> writes:

> On Apr 15, 2010, at 04:53, joakim@verona.se wrote:
>> I find this very interesting. Do you have some form of plan how to
>> integrate this form of Guile and Emacs? Can it be done incrementally,
>> that is, can the current elisp system and the guile system coexist in
>> the same Emacs? If so, more hackers could help out with the integration,
>> maybe.
>
> I think I've written up some email in the past (you may have to search back several months) about this.  My project -- changing Emacs to use Guile object types (at least the simplest ones) and Guile's allocator and GC -- can be done somewhat incrementally.  E.g., make sure Emacs uses accessor functions or macros for all uses of type X outside of its own GC system (those changes can be done upstream), and passes around Lisp_Object values rather than pointers to struct Lisp_Foo (likewise), then redefine type X's implementation using more appropriate Guile types, then work on type Y... currently strings, symbols, vectors, and hashes are on my list; individual "misc" and "vectorlike" types can come later, maybe.
>
> I have no intention of getting rid of elisp from emacs, most certainly.  As for eventually replacing the entire Lisp implementation in Emacs with the Guile one, well, I'm hoping that, like Andy suggested, it'll become an obvious win to do so, but if we wind up for a while with two distinct language interpreters (Emacs elisp and Guile VM) but a shared object system and working communication between languages, while Andy madly tries to fix whatever performance problem we might have run into with Guile's elisp implementation :-), I'd be okay with that.
>
> I've recently updated my code to be based on a recent development version of Emacs, though I haven't published the updated version yet and there appears to be some kind of GC problem or something at the moment.  I also haven't got it fully updated for guile 1.9.x (and the upcoming 2.0), which has a different GC system anyways.  (The new GC system is Boehm's GC library.  There was an effort to make Emacs use it some time back, by Dave Love, and I think some of those changes will be needed for Emacs-with-Guile-2.0.  So updating that work might also be helpful?)
>
> But, now that we've got bzr upstream and branching should be relatively easy, perhaps I should clean it up a little and push it all into a branch on savannah, if the maintainers are interested?  If there's really interest in working on it, I'd love to get some help....

I would love to have a look at it! Please push a branch.

> Ken
-- 
Joakim Verona

Re: guile and emacs and elisp, oh my!

[Ken Raeburn]

On Apr 15, 2010, at 19:54, I wrote:
> I've recently updated my code to be based on a recent development version of Emacs, though I haven't published the updated version yet and there appears to be some kind of GC problem or something at the moment.

For now (until/unless I hear from the Emacs maintainers that I should set up a branch on savannah), I've got it at git://raeburn.scripts.mit.edu/guile-emacs.git where branches "new-master-2" and "new-master-3" are a couple attempts at rebasing onto the new bzr history (one just before the bidi merge, one more recent) instead of the old cvs->git mirror's version of history.  Until/unless it moves to savannah, I reserve the right to rebase the branches in this repository again in the future at my convenience...

Ken

Re: guile and emacs and elisp, oh my!

[Ken Raeburn]

On Apr 16, 2010, at 13:05, Ken Raeburn wrote:
> For now (until/unless I hear from the Emacs maintainers that I should set up a branch on savannah), I've got it at git://raeburn.scripts.mit.edu/guile-emacs.git where branches "new-master-2" and "new-master-3" are a couple attempts at rebasing onto the new bzr history (one just before the bidi merge, one more recent)

Sorry, I remembered this wrong: One is shortly after the bidi merge, earlier this month, and the other is from back in August, when I sent the email I was thinking of earlier (though it was to the guile-devel list, actually: http://www.mail-archive.com/guile-devel@gnu.org/msg04255.html).  I had previously updated a branch to reflect the August rebasing using the cvs->git mirror; this rebases it again to use the new bzr->git mirror (which was harder than it ought to have been).

Also, some assorted (but partly outdated) notes: http://www.mit.edu/~raeburn/guilemacs/

Ken

Re: guile and emacs and elisp, oh my!

[Stefan Monnier]

> For now (until/unless I hear from the Emacs maintainers that I should
> set up a branch on savannah),

If you want to push a branch to sftp://bzr.sv.gnu.org/srv/bzr/emacs/guile,
that's perfectly fine by me.


        Stefan

Re: guile and emacs and elisp, oh my!

[Tom Tromey]

Andy> My name is Andy, and together with Ludovic Courtès I co-maintain Guile.

Hi.  Thanks for writing this.

I'm somewhat hesitant to write this response, since although I'm a
longtime Emacs-lover and random contributor, I'm not an Emacs developer
by any stretch.  So it isn't clear that I have standing to try to push a
particular big agenda.  Nevertheless, I care a lot about Emacs and I
found that my view on this topic has not been well-represented.

Andy> Guile can implement Emacs Lisp better than Emacs can.

Andy> No one will notice! Except that after a switch, Emacs would be faster,
Andy> more powerful, and have the ability to access all of Guile's facilities
Andy> -- the Scheme language, other languages implemented for Guile
Andy> (Javascript, Lua, ...), a proper ffi, dynamically loadable libraries, a
Andy> module system, the numeric tower (rationals, bignums, etc), Guile's
Andy> existing libraries, delimited continuations (!), fast bytevector access,
Andy> native threads, etc.

From my perspective, many of these are either not interesting, or are
outright misfeatures.  Let me explain.

First, people have often discussed the desirability of one runtime to
rule them all.  This has even been attempted -- Gnome floated a
proposal, there is Parrot, even the JVM and .NET are getting in on it.

I think, contrariwise, that not only is this not desirable, it is
actively bad.  I think it is beneficial to a program like Emacs to have
a single extension language.  A single language makes both reuse and
debugging simpler.  Multiple languages equals chaos.

Furthermore, I think that secondary implementations of popular languages
(why even consider the unpopular ones?) are destined to always remain
second-class citizens.  There is plenty of evidence to support both
these assertions: the wretched state of Gnome scripting (brought on,
IMO, by its tower-of-babel approach), and the lessons of Jython, Iron
Python, et al.

For the other points:

* FFI, and dynamically loadable libraries.  This feature has come up and
  been rejected several times for Emacs.  Anyhow, as with all of these
  features, we don't need Guile to implement it -- and in fact that is
  an extremely roundabout way to add a simple feature.

* A module system.  I've seen this idea explicitly rejected by RMS.

* The numeric tower.  Ok, this one is interesting!  But switching to
  Guile doesn't actually push this into the one place where it is
  needed: buffer sizes.  In fact, this is a sort of general problem with
  the rebase -- from what I understand, it doesn't deeply touch the bits
  that matter.

* Continuations.  If you thought threads were going to wreak havoc, wait
  until people call/cc from inside redisplay.

* Native threads.  There's an Emacs patch already.


My experience with Emacs as a user and occasional developer is that
elisp is fast enough for most tasks you'd actually want to write in
elisp.  Sure, maybe parsing could use some low-level love (something
better than parse-partial-sexp), or maybe regexps could use a speedup,
but even those are marginal.  Even the GC doesn't seem so bad nowadays.

Instead, I see three areas for improvement.

1. Platform integration.  Some progress has been made here (dbus comes
   to mind).  My pet missing feature here is notification area support,
   but there are others (e.g., embedding).

2. Display model.  If you try to write, say, a decent presentation
   program in Emacs (let alone something truly complex like a browser),
   you quickly run into holes.

3. Blocking behavior.  I run nearly everything in Emacs, and anything
   that blocks is at least an annoyance, and can even wreak havoc
   (causing an irc timeout is no fun).  Emacs has decent-ish tools for
   handling this problem already, except for situations like Gnus --
   which is why I started working on threads.

So, what I would propose is that instead of putting a lot of energy into
rebasing Emacs on a different VM, put that same energy into providing
features that affect users.  If elisp doesn't quite suit, I propose
simply accepting that elisp is its own language, and not be constantly
looking at Scheme or CL as the promised land: just evolve elisp in situ,
say by merging lexbind or adding threads or whatever else there that is
both concrete and incremental.

Tom

Re: guile and emacs and elisp, oh my!

[christian.lynbech]

I think there are a number of benefits to rebasing Emacs Lisp to a more
commonly used platform, even if this will not be immediately visible to
the casual user, some of the more obvious (as already hashed out in the
other discussions):

   - If elisp is fast enough today, it is to some extent because *lots*
     of functionality has moved to the C world. A faster implementation
     (with a FFI) would allow to take more of the internals back and hence
     allow gerater flexbility and more room for improvement.

   - Developer resources are limited, teaming up with an existing
     platform will provide more hands on the internals and hence free up
     time for the emacs specific work.

   - Having access to a more widely used language will enable non-emacs
     libraries to be imported into emacs or emacs libraries to be used
     elsewhere. For instance the buffer abstraction is one that probaly
     is useful in a number of applications.

I am not able to say if standardizing on a single language platform
across the GNU project makes sense or not. It probably will be very
difficult to decide on one particular language but having a common
platform with multiple languages sitting on top at least will enable a
considerable amount of work to be shared among application development.

It will take time and energy to change the language machinery of Emacs
but we should not forget that a significant portion of that effort has
already been spent and that one also should not underestimate the effort
needed to make the needed high-quality extensions to Emacs Lisp on the
existing implementation and to maintain them afterwards across the vast
score of platforms on which Emacs runs today.


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: guile and emacs and elisp, oh my!

[David Engster]

Tom Tromey <tromey@redhat.com> writes:
> I think it is beneficial to a program like Emacs to have a single
> extension language.  A single language makes both reuse and debugging
> simpler.  Multiple languages equals chaos.

I tend to agree. The priority should be to make Emacs Lisp more
powerful, for example by adding stuff from the CL library to the
language core.

> My experience with Emacs as a user and occasional developer is that
> elisp is fast enough for most tasks you'd actually want to write in
> elisp.

With Emacs now integrating IDE features like CEDET, the demands on the
speed of Emacs Lisp are increasing. The Semantic analyzer really pushes
what currently can be done in Emacs Lisp in a reasonable time, and it
would obviously benefit from a faster implementation. I guess that it
will become inevitable to implement some of the database handling into a
native C library of some sorts, but the more can be done in Emacs Lisp,
the better.

> 2. Display model.  If you try to write, say, a decent presentation
>    program in Emacs (let alone something truly complex like a browser),
>    you quickly run into holes.

I agree. Some time ago, I tried to implement a graphical diff, similar
to Kompare (see
http://www.caffeinated.me.uk/kompare/screenshot.png). People might argue
that stuff like this is just eye candy, but in my opinion a graphical
presentation like this can be really useful to get a quick overview of
the changes. I used a terrible hack to dynamically generate bitmap
images in the middle of the two buffers, but it was just too slow and
'flickery'. I think it would be great if Emacs could support graphical
primitives like this.

-David

Re: guile and emacs and elisp, oh my!

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> Tom Tromey <tromey@redhat.com> writes:
>> I think it is beneficial to a program like Emacs to have a single
>> extension language.  A single language makes both reuse and debugging
>> simpler.  Multiple languages equals chaos.
>
> I tend to agree. The priority should be to make Emacs Lisp more
> powerful, for example by adding stuff from the CL library to the
> language core.

The cl library is an incredibly complex hack.  It makes debugging and
figuring out things quite harder.

It makes more sense to switch to a system which deals with such
complexity in more straightforward ways.

So I tend to see your wish of "make Emacs Lisp more powerful instead" as
more expensive and painful in the long run.

-- 
David Kastrup

Re: guile and emacs and elisp, oh my!

[Lennart Borgman]

On Wed, Apr 21, 2010 at 11:49 AM, David Kastrup <dak@gnu.org> wrote:
>>
>> I tend to agree. The priority should be to make Emacs Lisp more
>> powerful, for example by adding stuff from the CL library to the
>> language core.
>
> The cl library is an incredibly complex hack.  It makes debugging and
> figuring out things quite harder.


I think it is very hard to find the description of slot functions etc
defined by the cl library. I have tried to make it easier in my
patched version of Emacs. Perhaps those patches are interesting to
include in Emacs?

Re: guile and emacs and elisp, oh my!

[David Engster]

David Kastrup <dak@gnu.org> writes:
> David Engster <deng@randomsample.de> writes:
>
>> Tom Tromey <tromey@redhat.com> writes:
>>> I think it is beneficial to a program like Emacs to have a single
>>> extension language.  A single language makes both reuse and debugging
>>> simpler.  Multiple languages equals chaos.
>>
>> I tend to agree. The priority should be to make Emacs Lisp more
>> powerful, for example by adding stuff from the CL library to the
>> language core.
>
> The cl library is an incredibly complex hack.  It makes debugging and
> figuring out things quite harder.
>
> It makes more sense to switch to a system which deals with such
> complexity in more straightforward ways.
>
> So I tend to see your wish of "make Emacs Lisp more powerful instead" as
> more expensive and painful in the long run.

With "stuff from the CL library" I'm not talking about features like
'lexical-let' or even the whole thing. What I'd really like to see, at
least in the short term, are things like 'remove-if', 'delete-if',
'every', 'notany', 'count', 'loop', etc. . I know that the
implementation of those often uses CL keywords, which is why Richard
didn't want them in Emacs Lisp, at least not in the current form.

-David

Re: guile and emacs and elisp, oh my!

[David Kastrup]

David Engster <deng@randomsample.de> writes:

> David Kastrup <dak@gnu.org> writes:
>> David Engster <deng@randomsample.de> writes:
>>
>>> Tom Tromey <tromey@redhat.com> writes:
>>>> I think it is beneficial to a program like Emacs to have a single
>>>> extension language.  A single language makes both reuse and debugging
>>>> simpler.  Multiple languages equals chaos.
>>>
>>> I tend to agree. The priority should be to make Emacs Lisp more
>>> powerful, for example by adding stuff from the CL library to the
>>> language core.
>>
>> The cl library is an incredibly complex hack.  It makes debugging and
>> figuring out things quite harder.
>>
>> It makes more sense to switch to a system which deals with such
>> complexity in more straightforward ways.
>>
>> So I tend to see your wish of "make Emacs Lisp more powerful instead" as
>> more expensive and painful in the long run.
>
> With "stuff from the CL library" I'm not talking about features like
> 'lexical-let' or even the whole thing. What I'd really like to see, at
> least in the short term, are things like 'remove-if', 'delete-if',
> 'every', 'notany', 'count', 'loop', etc.

But most of the stuff working with predicates is a rather bad idea to
use extensively without proper and efficient closures.

Putting that stuff into Emacs without a better integration into the
engine than the CL hackery is going to encourage programming styles that
lead to inefficient solutions.

-- 
David Kastrup

Re: guile and emacs and elisp, oh my!

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2896 bytes --]

Hello Tom,

(My first post on this list, so I’m a bit impressed.  ;-))

I think this discussion should be framed in the larger context of the
GNU System.  Guile came into existence to “spread the Emacs nature” to
the rest of GNU, to paraphrase Andy[*].

The GNU System as I view it aims to liberate users in two ways:

  1. Legally, by using software licenses that guarantee the 4 freedoms.

  2. Technically, by providing extension points allowing users to easily
     adapt the system to their own needs.

The latter is paramount to the design of 3 key components of GNU:
the Hurd, Guile, and Emacs.

Guile is meant to be a powerful interface between users and the GNU
System, more powerful than a Bourne shell, and more moldable than a C
API.  It aims to be a substrate allowing system components to be glued
together to let new applications emerge.

Users should be able to take Emacs, GnuTLS, and Mailutils and have Gnus
benefit from what each component has to offer.  They should be able to
write GTK+/GNUstep GUIs that present Org-Mode files in a nice way, or
render them as PDF via GNU PDF.

All these GNU libraries, programs, and Emacs modes should be able to be
tightly integrated together, and this is where Guile comes in.

Part of this is already possible, but the interface between Emacs and
the rest of GNU remains disappointingly poor.

A few remarks:

Tom Tromey <tromey@redhat.com> writes:

> First, people have often discussed the desirability of one runtime to
> rule them all.  This has even been attempted -- Gnome floated a
> proposal, there is Parrot, even the JVM and .NET are getting in on it.

Guile isn’t aiming at support for tens of languages like Parrot.  Guile
is a Scheme implementation, first and foremost.

It has an elisp compiler and integrates well with elisp because elisp is
primordial as an extension language for GNU.  It also comes with a
compiler for ECMAScript, which turns out to have many features
comparable to Scheme and a surface syntax that some people prefer.

(It also has a Brainfuck front-end.  Imagine yourself extending Emacs in
Brainfuck, thank Guile!)

> Multiple languages equals chaos.

I don’t think it’s that clear cut.  Vim (heh...) supports several
extension languages, PLT Scheme implements multiple languages.

Emacs-on-Guile as I imagine it would be mainly about elisp and (to a
lesser extent) Scheme, though.

> * FFI, and dynamically loadable libraries.  This feature has come up and
>   been rejected several times for Emacs.

In practice D-BUS can be thought of as an FFI, not to mention the use of
special-purpose external programs such as ‘gnutls-cli’.

Thanks,
Ludo’.

[*] I recommend looking at or rather listening to Andy’s talk at
    Göteborg’s GHM, entitled “Guile, GNU, and You”:
    http://www.gnu.org/ghm/2009/ .

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: guile and emacs and elisp, oh my!

[Andy Wingo]

Hi Tom,

On Wed 21 Apr 2010 00:36, Tom Tromey <tromey@redhat.com> writes:

> Andy> My name is Andy, and together with Ludovic Courtès I co-maintain Guile.
>
> Hi.  Thanks for writing this.

Thank you for writing this thoughtful reply!

> People have often discussed the desirability of one runtime to rule
> them all. This has even been attempted -- Gnome floated a proposal,
> there is Parrot, even the JVM and .NET are getting in on it.

Heh, perhaps we have oversold Guile a bit. It's true that Guile isn't
particularly well suited for implementing Haskell right now; nor perhaps
will it ever be.

But Elisp has a special place in Guile. We have thought a lot about how
to implement it well, and how to integrate it well with Scheme. We have
great support for threadsafe dynamic binding, good nil/null
interoperability, and if something needs changing in Guile to support
Elisp better, we'll do it.

So while it's true that we're currently getting a number of proposals to
implement other languages, and some code, Guile's first language will
still be Scheme, and the "other language" which it will best support
will be Elisp.

> A single language makes both reuse and debugging simpler. Multiple
> languages equals chaos.

You're certainly right that support for multiple languages complicates
reuse and debugging. I'm not sure I'd go so far as "chaos", though;
we'll see once we have a more real proposal.

Some other points:

> * The numeric tower.  Ok, this one is interesting!  But switching to
>   Guile doesn't actually push this into the one place where it is
>   needed: buffer sizes.  In fact, this is a sort of general problem with
>   the rebase -- from what I understand, it doesn't deeply touch the bits
>   that matter.

Heh, I think that's the essence of a refactor -- small,
correctness-preserving patches that eventually make shallow the deeper
changes. With an Emacs-on-Guile, one could make the buffer size a SCM
value, which would normally be a fixnum but could be a bignum. This
approach has advantages and disadvantages, and they would be considered
as part of that change, not as part of an Emacs-on-Guile change.

> * Continuations.  If you thought threads were going to wreak havoc, wait
>   until people call/cc from inside redisplay.

We have continuation barriers. Most likely we would, by default,
prohibit nonlocal reentry to Elisp functions.

> My experience with Emacs as a user and occasional developer is that
> elisp is fast enough for most tasks you'd actually want to write in
> elisp.  Sure, maybe parsing could use some low-level love (something
> better than parse-partial-sexp), or maybe regexps could use a speedup,
> but even those are marginal.  Even the GC doesn't seem so bad
> nowadays.

I would humbly submit that you have, over time, restricted the range of
programs that you would write in Elisp, to that set of programs that
Emacs' implementation of Elisp can deal with.

There are many programs that you just can't write in Elisp, linguistic
expressiveness aside: the implementation just isn't fast enough. It
shouldn't be that hard to thread my inbox ;-)


That said, I think that your concerns about what a Guile-implemented
Elisp would do to Emacs are important ones, and that our implementation
should take them into account. I still think that Guile and Emacs would
be a beautiful future, so we'll keep working on it; but our work should
not stall your own. Lexbind, native-threads, and emacs-on-guile do not
conflict in any important way, IMO.

Happy hacking,

Andy
-- 
http://wingolog.org/

Re: guile and emacs and elisp, oh my!

[Karl Fogel]

Andy Wingo <wingo@pobox.com> writes:
>I would humbly submit that you have, over time, restricted the range of
>programs that you would write in Elisp, to that set of programs that
>Emacs' implementation of Elisp can deal with.

What Andy said.

>There are many programs that you just can't write in Elisp, linguistic
>expressiveness aside: the implementation just isn't fast enough. It
>shouldn't be that hard to thread my inbox ;-)

What Andy said, again :-).

Re: guile and emacs and elisp, oh my!

[Tom Tromey]

>>>>> "Karl" == Karl Fogel <kfogel@red-bean.com> writes:

Karl> Andy Wingo <wingo@pobox.com> writes:
>> I would humbly submit that you have, over time, restricted the range of
>> programs that you would write in Elisp, to that set of programs that
>> Emacs' implementation of Elisp can deal with.

Karl> What Andy said.

>> There are many programs that you just can't write in Elisp, linguistic
>> expressiveness aside: the implementation just isn't fast enough. It
>> shouldn't be that hard to thread my inbox ;-)

Karl> What Andy said, again :-).

Do you know for certain that the performance problem is caused by the
elisp implementation, and not by some bad algorithm in whatever mail
program you are using?

My experience profiling Emacs is that the profiles have a long tail, and
the actual interpreter, while slow (it was #3 in the profiles as I
recall), would not result in an enormous gain even if it disappeared
completely.  I think this was also the result of the elisp JIT
experiment.

In the particular case I looked at, regular expressions were the #1 slow
point, followed by the GC; and even the GC was only 10%.  (My test was
heavy on regex use; though FWIW I suspect this is very common in elisp
code.)

I'm interested to understand how Guile will be faster-enough to make
this better.

I can see how this could be done via a very deep rewrite (e.g., if the
Emacs core was itself in lisp, you could do all the fun tricks that
typical CL implementations do), but I'm curious to know what other
clever approaches there are.

Tom

Re: guile and emacs and elisp, oh my!

[Thomas Lord]

On Thu, 2010-04-22 at 11:35 +0200, Andy Wingo wrote:
> good nil/null
> interoperability

May I ask: what do you have and why do you consider
it to be good?

-t

Re: guile and emacs and elisp, oh my!

[Andy Wingo]

On Fri 23 Apr 2010 04:00, Thomas Lord <lord@emf.net> writes:

> On Thu, 2010-04-22 at 11:35 +0200, Andy Wingo wrote:
>> good nil/null
>> interoperability
>
> May I ask: what do you have and why do you consider
> it to be good?

We've just started docs on this one, but here's the initial thread:

  http://thread.gmane.org/gmane.lisp.guile.devel/10232

Here's the docs from master (this link will probably vanish in a few
weeks):

  http://hydra.nixos.org/build/367041/download/2/guile.html/Nil.html#Nil

It's a little different since you last asked me about this; I took your
feedback into account, I think, but do let me know if we're missing
something.

I think it's good because Scheme or Elisp lists and booleans can be
passed back and forth, and traversing those lists or testing values for
truth Just Works(tm). Of course, there are a couple of pitfalls; these
are noted in the docs there.

Cheers,

Andy
-- 
http://wingolog.org/

Re: guile and emacs and elisp, oh my!

[Ludovic Courtès]

Andy Wingo <wingo@pobox.com> writes:

> Here's the docs from master (this link will probably vanish in a few
> weeks):

You can use this one instead:

  http://hydra.nixos.org/job/gnu/guile-master/manual/latest/download/2/guile.html/Nil.html#Nil

Thanks,
Ludo’.

Re: guile and emacs and elisp, oh my!

[christian.lynbech]

>>>>> "Andy" == Andy Wingo <wingo@pobox.com> writes:

Andy> I think it's good because Scheme or Elisp lists and booleans can be
Andy> passed back and forth, and traversing those lists or testing values for
Andy> truth Just Works(tm). Of course, there are a couple of pitfalls; these
Andy> are noted in the docs there.

I may have missed it in the doc, but what will

        (car nil)

return?


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: guile and emacs and elisp, oh my!

[Andy Wingo]

On Fri 23 Apr 2010 12:19, <christian.lynbech@tieto.com> writes:

>>>>>> "Andy" == Andy Wingo <wingo@pobox.com> writes:
>
> Andy> I think it's good because Scheme or Elisp lists and booleans can be
> Andy> passed back and forth, and traversing those lists or testing values for
> Andy> truth Just Works(tm). Of course, there are a couple of pitfalls; these
> Andy> are noted in the docs there.
>
> I may have missed it in the doc, but what will
>
>         (car nil)
>
> return?

In Scheme, this is an error. In Elisp, it returns nil. They are actually
two different operators, that just happen to share the same name.

Andy
-- 
http://wingolog.org/

Re: guile and emacs and elisp, oh my!

[christian.lynbech]

>>>>> "Andy" == Andy Wingo <wingo@pobox.com> writes:

Andy> In Scheme, this is an error. In Elisp, it returns nil. They are actually
Andy> two different operators, that just happen to share the same name.

Out of curiosity, how is scheme and elisp separated? Ie. what should one
do to work in the elisp world?


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: guile and emacs and elisp, oh my!

[Andy Wingo]

On Fri 23 Apr 2010 13:31, <christian.lynbech@tieto.com> writes:

>>>>>> "Andy" == Andy Wingo <wingo@pobox.com> writes:
>
> Andy> In Scheme, this is an error. In Elisp, it returns nil. They are actually
> Andy> two different operators, that just happen to share the same name.
>
> Out of curiosity, how is scheme and elisp separated? Ie. what should one
> do to work in the elisp world?

Currently, the only supported ways of switching Guile to other languages
are documented here:

  http://hydra.nixos.org/build/367041/download/2/guile.html/Using-Other-Languages.html#Using-Other-Languages

There are other ways, but we have not documented them yet.

Andy
-- 
http://wingolog.org/

Re: guile and emacs and elisp, oh my!

[Thomas Lord]

So, because this thread is a large confusing 
tangle of issues I'll mention that *this* message
is about the treatment of elisp nil vs. Scheme () and #f
in Guile.

Andy, thanks.  I do think that what you have there in
Guile's elisp integration plans is
plausibly "the best that can be done".  If the 
requirement is to have elisp and Scheme unified
within a single programming environment, I doubt anyone
could do much better.  This is not to say I think
the result is particularly satisfying.

And, I don't want to dwell on the topic.   Last time I
mentioned something off-hand it led to several 10s of 
messages of dubious appropriateness for this list (and 
who the hell am I to comment on this list at all).
But as XCDD semi-famously put it: "There's something 
*wrong* on the Internet," so I'll briefly say my piece
here and hope not to be too disruptive.  (Just disruptive
enough? :-)   (Everyone should please think thrice 
before replying and remember that this subthread
ultimately is likely to have very little impact or
importance so it isn't essential to comment here unless
you really, really have something useful to say.)

The reason I asked about what you had in mind for Guile
re nil v. () v. #f is because I think the issue is a 
quite fine, compact illustration of what I understand
to be the root causes of what Tom Tromey described
as the "chaos" that results in trying to build 
multi-lingual solutions that somehow (and then a miracle
happens) unify legacy languages into a coherent 
programming language.

Sure, Schemers ought to be encouraged to write
"(null? x)" rather than "(eq? x '())" but ... 
what about "(eq? x y)"?  

More damingly, perhaps, what distinction does 
Guile plan between the Scheme symbol, "'nil", and
Emacs the Emacs lisp symbol, "nil"?  Do emacs
lisp and Guile not even agree on what is and what 
is not a symbol?

And: since Emacs lisp and Guile can't agree about
equality (whichever flavor) - how do I write a 
hash table library useful to both environments?

If, absent the historic liabilities, someone proposed
a programming environment with multiple syntaxes, sure,
but all sharing a common data representation - with
(), #f, and Emacs lisp nil .... wouldn't you think they
were kind of insane?   The multiple syntaxes part - fine.
But the triad of (), #f, and nil?  It would seem, in my view, 
to be a (mis-)feature designed mainly to make programming
in that environment harder.

So there is a non-trivial and hard to measure but also
hard to ignore cost to trying to duct tape these two languages
together.   It makes a mess.  It makes something that nobody
in their right mind would come close to designing from 
first principles without the historical baggage or the
economic incentive of tying in to the Emacs legacy.

So there is a cost benefit analysis to be done and, to 
be clear, I'm sure nobody can objectively claim a right
answer here - we're in the realm of pure opinion.

I say screw it. 

Let Emacs lisp live and the current implementation
thrive and incrementally evolve.  Tromey said all that
better than I do.

And also note that the corpus of Emacs lisp code is
finite and also not *that* huge -- so on a friendly
fork (partially) screw that corpus of code and make a
pure Scheme version.  Technically it's a much simpler task
and leads to a much more coherent programming environment.

Now there's competition there (presumably friendly).  Maybe
the benefits of Scheme aren't good enough and Emacs Lisp 
should prevail.  Maybe the other way around.  Who knows?
But this hybrid approach is such a muddle I don't see how people
aren't going to spend 10+ years spitting in that gutter 
(I was going to say "pissing down that drain" but decided
not to be so crude) and not getting very far.

Go for it, I say.  Fork the M.F.P. in a friendly way and
float a pure Scheme version that is anemic in extension 
packages.  Invest the programming language smarts not in
putting elisp and Scheme together in a run-time but rather
in assisting the semi-manual translation of elisp to the Scheme
environment.   Get out a solid Scheme Emacs first and *then*
worry about how to import all of that legacy elisp code.
The route you're going is pure politics-driven-engineering.
The friendly fork / Scheme-first route -- I'd bet an entire 
$0.25 you *could* get somewhere worth going in 18-24 months.
The route you are going?  I think that *at best* you yield a 
temporary and tiny improvement in elisp performance - more likely
you produce nothing that ever sticks.  I'm not trying to be
harsh - just (hopefully) helpfully honest.

-t


On Fri, 2010-04-23 at 10:28 +0200, Andy Wingo wrote:
> On Fri 23 Apr 2010 04:00, Thomas Lord <lord@emf.net> writes:
> 
> > On Thu, 2010-04-22 at 11:35 +0200, Andy Wingo wrote:
> >> good nil/null
> >> interoperability
> >
> > May I ask: what do you have and why do you consider
> > it to be good?
> 
> We've just started docs on this one, but here's the initial thread:
> 
>   http://thread.gmane.org/gmane.lisp.guile.devel/10232
> 
> Here's the docs from master (this link will probably vanish in a few
> weeks):
> 
>   http://hydra.nixos.org/build/367041/download/2/guile.html/Nil.html#Nil
> 
> It's a little different since you last asked me about this; I took your
> feedback into account, I think, but do let me know if we're missing
> something.
> 
> I think it's good because Scheme or Elisp lists and booleans can be
> passed back and forth, and traversing those lists or testing values for
> truth Just Works(tm). Of course, there are a couple of pitfalls; these
> are noted in the docs there.
> 
> Cheers,
> 
> Andy

Re: guile and emacs and elisp, oh my!

[Andy Wingo]

Hi Tom,

Thanks for your mail, it brought up some things I hadn't thought about.

On Sat 24 Apr 2010 00:43, Thomas Lord <lord@emf.net> writes:

> Sure, Schemers ought to be encouraged to write
> "(null? x)" rather than "(eq? x '())" but ... 
> what about "(eq? x y)"?  

To me, `(null? x)' checks nullity. `(eq? x '())' is ambiguous -- it
could be a check for nullity, or it could be a check for identity. `(eq?
x y)' to me is unambiguous though, it is a check for identity.

I would be interested to know if you had a case in mind in which the
introduction of nil to Scheme let to incorrect results via `(eq? x y)',
beyond those documented in Nil.html.

> More damingly, perhaps, what distinction does 
> Guile plan between the Scheme symbol, "'nil", and
> Emacs the Emacs lisp symbol, "nil"?  Do emacs
> lisp and Guile not even agree on what is and what 
> is not a symbol?

Irritating! I did not even realize that nil was a symbol.

Given that there appear to be few Elisp primitives that operate on
symbols, I imagine the way this will go will be for nil to not be a
symbol to Scheme, but we'll add some shims to elisp to have it be a
symbol, and have the elisp reader read 'nil as nil. There are only about
50 instances of 'nil in the emacs source, and some of them are ''nil, so
hopefully this will not cause too much of a problem.

To my (naively optimistic?) tastes, we're not at chaos yet; but this is
certainly a step in that direction.

> And: since Emacs lisp and Guile can't agree about
> equality (whichever flavor) - how do I write a 
> hash table library useful to both environments?

Scheme will probably get some new operators out of all of this: nil-car,
nil-cdr, nil-equal?, nil-hash, nil-assoc, etc. By default, a hash table
implemented in Scheme would treat nil and '() as distinct, though you
could implement one with nil-hash and nil-assoc and you get an
elisp-style hash table. Elisp hash tables use Elisp hash and assoc,
which are bound to nil-hash and nil-assoc or their equivalents.

> If, absent the historic liabilities, someone proposed
> a programming environment with multiple syntaxes, sure,
> but all sharing a common data representation - with
> (), #f, and Emacs lisp nil .... wouldn't you think they
> were kind of insane?   The multiple syntaxes part - fine.
> But the triad of (), #f, and nil?  It would seem, in my view, 
> to be a (mis-)feature designed mainly to make programming
> in that environment harder.

I am inclined to agree, though I would probably state things
differently: the Elisp code that is out there is liability, yes, but
wealth also.

As you note, we're always making these economic decisions, and with
regards to unknown (and often unquantifiable) costs and benefits -- well
you never really know until you step into that unknown, right? (And the
reasons that lead one to make those steps often go by slippery names
like "faith" or "instinct" or such things.)

Anyway, rambling here at the end. I respect your desire for a new lovely
Scheme Emacs; but I think I'm going to try for a future world that
includes Elisp and Elisp programmers. Maybe over time it all stays in
Elisp; that's fine. Maybe people find they like Scheme better, and
incrementally rewrite things in that language. That's cool too, and
personally I hope things go that way. But we'll see!

Happy hacking,

Andy
-- 
http://wingolog.org/

Re: guile and emacs and elisp, oh my!

[Thomas Lord]

On Sat, 2010-04-24 at 13:05 +0200, Andy Wingo wrote:
> Hi Tom,

Hello.

> Thanks for your mail, it brought up some things I hadn't thought about.

Glad to help.



> On Sat 24 Apr 2010 00:43, Thomas Lord <lord@emf.net> writes:

> > Sure, Schemers ought to be encouraged to write
> > "(null? x)" rather than "(eq? x '())" but ... 
> > what about "(eq? x y)"?  

> To me, `(null? x)' checks nullity. `(eq? x '())' is ambiguous -- it
> could be a check for nullity, or it could be a check for identity. `(eq?
> x y)' to me is unambiguous though, it is a check for identity.

> I would be interested to know if you had a case in mind in which the
> introduction of nil to Scheme let to incorrect results via `(eq? x y)',
> beyond those documented in Nil.html.

An example is easier with elisp EQUAL and Scheme EQUAL?:
Because they give different results:

   (equal? #nil #f) => #f
   (equal nil f) => t

the two languages can not even reliably agree
when two keys in an equality-based association
are the same.


> > More damingly, perhaps, what distinction does 
> > Guile plan between the Scheme symbol, "'nil", and
> > Emacs the Emacs lisp symbol, "nil"?  Do emacs
> > lisp and Guile not even agree on what is and what 
> > is not a symbol?

> Irritating! I did not even realize that nil was a symbol.

   (symbolp nil) =>  t
   (symbolp t) => t

Per old lisp tradition, they are symbols bound permanently
to themselves:

   (symbol-value nil) => nil
   (symbol-value t) => t
   (let ((t 3)) t) =>  ERROR: attempt to set a constant symbol


> Given that there appear to be few Elisp primitives that operate on
> symbols, I imagine the way this will go will be for nil to not be a
> symbol to Scheme, but we'll add some shims to elisp to have it be a
> symbol, and have the elisp reader read 'nil as nil. There are only about
> 50 instances of 'nil in the emacs source, and some of them are ''nil, so
> hopefully this will not cause too much of a problem.

And I guess you will have to do the same for #t?

Suppose that Scheme's STRING->SYMBOL is made available
(directly or indirectly) to the Emacs lisp side.  Then:

    ,L elisp
    ;; for some symbol bound to sym:
    ;; 
    (eq sym (string->symbol (symbol-name sym)))
    => t     in all cases where sym is not nil or t
    => nil   otherwise

If Emacs lisp's INTERN is made available directly or
indirectly to Scheme:

    (eq? sym (intern (symbol->string sym)))
    => #t   in all cases where sym is not 'nil or 't
    => #f   otherwise

I find it hard to think through what code that might break
or what kinds of things it will make hard to write.  I find
strange, random exceptions to simple rules hard to think 
through in general.   That's why I describe the result here
as the kind of programming environment that nobody would
ever design on purpose - an environment that seems to have features
whose sole (apparent) purpose is to make programming harder.

Is upwards compatibility with elisp really worth that much?



> To my (naively optimistic?) tastes, we're not at chaos yet; but this is
> certainly a step in that direction.

Back in yesteryear, I was pretty optimistic about
joining Scheme and Emacs lisp, too.

Things were different, back then.  R5 was not yet
released.   R4RS and IEEE were the latest standards
and were the *first* to require that Scheme '() not 
be a false value.

Indeed, R3RS *required* '() to be false and stated:

    (eqv? '() #f) => *unspecified*

So, at first, we thought to just diverge from that
newfangled R4RS noise and Guile would be a Scheme
in which:

    (eq? '() #f) => #t

Additionally, all symbols would have a dynamic
binding slot (in effect, every symbol would be
a "fluid" in today's Guile terminology).

I don't recall that we had clearly thought through
the problem of '() (elisp's nil) not being a symbol.

Pressure grew to distinguish '() and #f and at 
some point, a proposal for #nil (very much like the
one you have now) was put forward.   At first I
thought that might work but it nagged at me.  The more
I thought about it, the less I believed that it or any
other "solution" could work.




> > And: since Emacs lisp and Guile can't agree about
> > equality (whichever flavor) - how do I write a 
> > hash table library useful to both environments?

> Scheme will probably get some new operators out of all of this: nil-car,
> nil-cdr, nil-equal?, nil-hash, nil-assoc, etc. By default, a hash table
> implemented in Scheme would treat nil and '() as distinct, though you
> could implement one with nil-hash and nil-assoc and you get an
> elisp-style hash table. Elisp hash tables use Elisp hash and assoc,
> which are bound to nil-hash and nil-assoc or their equivalents.

So much for re-using code written for other 
Scheme implementations.   And, again: Scheme is 
already a touch suspect for having three distinct
equality predicates instead of the more customary
two.  Now you propose to add a fourth.

It's also unfortunate that this will make a mess
out of using the FFI.   When an extension function
written in C uses an equality predicate or a 
checks to see if something is a symbol: what should
be its policy - Scheme rules or Emacs lisp rules or
some kind of dynamically configrable option?


> > If, absent the historic liabilities, someone proposed
> > a programming environment with multiple syntaxes, sure,
> > but all sharing a common data representation - with
> > (), #f, and Emacs lisp nil .... wouldn't you think they
> > were kind of insane?   The multiple syntaxes part - fine.
> > But the triad of (), #f, and nil?  It would seem, in my view, 
> > to be a (mis-)feature designed mainly to make programming
> > in that environment harder.

> I am inclined to agree, though I would probably state things
> differently: the Elisp code that is out there is liability, yes, but
> wealth also.

Perhaps.  But that "wealth" can be preserved in other ways.
For example, by writing tools that assist in the semi-manual
translation of Emacs lisp to Scheme.



> As you note, we're always making these economic decisions, and with
> regards to unknown (and often unquantifiable) costs and benefits -- well
> you never really know until you step into that unknown, right? (And the
> reasons that lead one to make those steps often go by slippery names
> like "faith" or "instinct" or such things.)

Sure.  And I'm not one to stop you from following your 
dreams.

I think that there are some objective things to contemplate:
Disjointness of types, truth and falsity, uniformly represented lists, 
and universal equivalence and structural equivalence predicates
are *central* to many (perhaps most) idioms of lisp programming,
in every dialect, for as long as there has been lisp.  Those
are the basic aspects of values in general.

>From first principles you can correctly deduce that Emacs lisp
and Scheme must always disagree about disjointedness, the
representation of lists, equivalence, and structural equivalence.
It is trivial to get them to agree about falsity by introducing
#nil, but doing so makes the disjointedness, list representation,
and equivalence predicates even more problematic.   It's a "now you
have two problems" kind of situation.

>From historic reality you can also see that the decision to 
add #nil *greatly limits the prospects for Guile and Emacs in
the future*.   In particular, there will be a long-term 
barrier on re-using any third party Scheme code, FFI-extension
to Scheme, or new and better Scheme implementation.   The 
proposal here is to exit the isolated and lonely world of 
Emacs Lisp and start afresh in an isolated and lonely world
of Guile+Emacs Lisp.

You can have your dreams and, heck, you might be right.
After all, the problems of three little lisp values 
don't amount to a hill of beans in this crazy world.
But as for me, I'm not telling you get on that plane.
I don't think the Scheme Underground resistance movement
benefits.  And I'm pretty sure you're not really in love
with the whole plan. If you take that plane, if you ask me, one day
you'll regret it.  Maybe not today, maybe not tomorrow, but
soon and for the rest of your life.



> Anyway, rambling here at the end.

It's good to talk it through.

>  I respect your desire for a new lovely
> Scheme Emacs; but I think I'm going to try for a future world that
> includes Elisp and Elisp programmers. Maybe over time it all stays in
> Elisp; that's fine. Maybe people find they like Scheme better, and
> incrementally rewrite things in that language. That's cool too, and
> personally I hope things go that way. But we'll see!

I suppose we shall.

-t



> Happy hacking,
> 
> Andy

Re: guile and emacs and elisp, oh my!

[Richard Stallman]

    Sure, Schemers ought to be encouraged to write
    "(null? x)" rather than "(eq? x '())" but ... 
    what about "(eq? x y)"?  

My original plan was to have two execution modes,
a Lisp mode in which () and Scheme's false object are equal,
and a Scheme mode in which they were not equal.
I had a plan for nil too, but I don't remember what it was.

    And: since Emacs lisp and Guile can't agree about
    equality (whichever flavor) - how do I write a 
    hash table library useful to both environments?

It should hash all the kinds of nil the same.
Then, as regards the comparison, it can depend on
the execution mode as inherited from the caller.

Re: guile and emacs and elisp, oh my!

[Andy Wingo]

Hi,

On Sun 25 Apr 2010 18:54, Richard Stallman <rms@gnu.org> writes:

>     Sure, Schemers ought to be encouraged to write
>     "(null? x)" rather than "(eq? x '())" but ... 
>     what about "(eq? x y)"?  
>
> My original plan was to have two execution modes,
> a Lisp mode in which () and Scheme's false object are equal,
> and a Scheme mode in which they were not equal.

That is something like what Guile does, except the mode can be
determined statically -- it's the language that the code is written in.
If it's elisp, they are equal, if it's scheme, they're not.

>     And: since Emacs lisp and Guile can't agree about
>     equality (whichever flavor) - how do I write a 
>     hash table library useful to both environments?
>
> It should hash all the kinds of nil the same.
> Then, as regards the comparison, it can depend on
> the execution mode as inherited from the caller.

Hashing nil, null, and false to the same value is probably a good idea.
Then if you use an elisp accessor, you get elisp semantics, or scheme
semantics with a scheme accessor.

Andy
-- 
http://wingolog.org/