ELisp?

ELisp?

[Noah Lavine]

Hello all,

Could anyone tell me the status of the ELisp implementation since July
21st (the last update)?

I'm especially interested now because of a thread on emacs-devel -
http://lists.gnu.org/archive/html/emacs-devel/2011-09/msg00753.html.
People were talking about adding multithreading to Emacs. Some people
said that moving Emacs to Guile might be the best solution, but others
were pessimistic about that ever happening. It looks like the Emacs
people would really like to use Guile, and we would certainly like
them to. Also, every time Emacs implements a feature that Guile
already has, it's wasted effort when we eventually integrate them.

It seems like Emacs-Guile integration is a project that is on the
verge of happening, and just needs some sort of push to get it over
the hump. I hope that this summer's SoC work can be the impetus.

Noah

Re: ELisp?

[joakim]

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Hello all,
>
> Could anyone tell me the status of the ELisp implementation since July
> 21st (the last update)?
>
> I'm especially interested now because of a thread on emacs-devel -
> http://lists.gnu.org/archive/html/emacs-devel/2011-09/msg00753.html.People were talking about adding multithreading to Emacs. Some people
> said that moving Emacs to Guile might be the best solution, but others
> were pessimistic about that ever happening. It looks like the Emacs
> people would really like to use Guile, and we would certainly like
> them to. Also, every time Emacs implements a feature that Guile
> already has, it's wasted effort when we eventually integrate them.
>
> It seems like Emacs-Guile integration is a project that is on the
> verge of happening, and just needs some sort of push to get it over
> the hump. I hope that this summer's SoC work can be the impetus.
>
> Noah
>
>

I'm an Emacs developer(not the most prolific but I have contributed
ImageMagick core integration and Webkit integration for Emacs 25)

I'm now trying to learn non-trivial Guile usage so as to be useful in
the future merge of Guile and Emacs.

Currently, for me coming from the Emacs side is that I have no idea how
integration would happen. I've studied the various attempts but haven't
come to any conclusion.

My naive approach would be to make an Emacs Guile branch on
Savannah. The branch would unify incrementally the 3 different existing
approaches:

  - Nishidas patches to run Guile in Emacs
  - Ken Raeburns patches to replace the Emacs Lisp engine
  - Templetons implementation of Elisp in Guile

This branch would initially use Nishidas strategy of just linking Guile
and providing some simple interoperability functions. The version linked
would be Templetons Elisp Guile version. So, there would be 2 lisp
engines in Emacs for a while. When things work well the old Emacs lisp
engine would be ripped out.
  
As I'm coming from the Emacs side of things my interest is in getting
Guile in the Emacs core efficiently. My motivation is somewhat selfish
because I don't want to re-implement all the interesting features Guile
has in Emacs, such as dynamic FFI etc..

Of course there are many details to work out. A random example could
be that we would like a cookie to indicate that eval-buffer should use
guile rather than elisp for a buffer.

The advantage of the above approach is that it is incremental. We gain
exposure in the Emacs developer community. Since Emacs 24 is entering
pretest its a good time to start to think of features for Emacs 25.



-- 
Joakim Verona

Re: ELisp?

[Noah Lavine]

That makes sense. I was hoping we could skip the phase where we have
two lisp interpreters around, if Guile's ELisp implementation was good
enough, but maybe that will not be possible. Emacs already does
something similar to this for the programming language modes, where it
has another interpreter as a subprocess, so this doesn't sound so bad
technically.

One question I have is, how much of Emacs' C code can Guile use right
now? I guess Ken Raeburn's patches are the ones that address that, but
I don't know what is left to do.

(I also have the same question about ELisp - what of the language is
left to implement? Is it just adding more builtins?)

I don't know if this would work, but the way I was imagining it
happening is really just starting up Guile and loading Emacs source
files, one by one. If we have a complete ELisp implementation with all
of Emacs' C code, then in theory Emacs would start. I don't know what
would actually happen though.

A branch on Savannah sounds like a really good idea though. Then at
least there would be a central thing for people to look at and hack
on, and more people might get involved. Right now I suspect that most
people don't know what's happening or where to look. If you set up a
branch, I for one would love to check it out and perhaps contribute
some patches. Do you need permission from the Emacs maintainers to do
that, or anything else?

Noah

On Sun, Oct 9, 2011 at 9:37 AM,  <joakim@verona.se> wrote:
> Noah Lavine <noah.b.lavine@gmail.com> writes:
>
>> Hello all,
>>
>> Could anyone tell me the status of the ELisp implementation since July
>> 21st (the last update)?
>>
>> I'm especially interested now because of a thread on emacs-devel -
>> http://lists.gnu.org/archive/html/emacs-devel/2011-09/msg00753.html.People were talking about adding multithreading to Emacs. Some people
>> said that moving Emacs to Guile might be the best solution, but others
>> were pessimistic about that ever happening. It looks like the Emacs
>> people would really like to use Guile, and we would certainly like
>> them to. Also, every time Emacs implements a feature that Guile
>> already has, it's wasted effort when we eventually integrate them.
>>
>> It seems like Emacs-Guile integration is a project that is on the
>> verge of happening, and just needs some sort of push to get it over
>> the hump. I hope that this summer's SoC work can be the impetus.
>>
>> Noah
>>
>>
>
> I'm an Emacs developer(not the most prolific but I have contributed
> ImageMagick core integration and Webkit integration for Emacs 25)
>
> I'm now trying to learn non-trivial Guile usage so as to be useful in
> the future merge of Guile and Emacs.
>
> Currently, for me coming from the Emacs side is that I have no idea how
> integration would happen. I've studied the various attempts but haven't
> come to any conclusion.
>
> My naive approach would be to make an Emacs Guile branch on
> Savannah. The branch would unify incrementally the 3 different existing
> approaches:
>
>  - Nishidas patches to run Guile in Emacs
>  - Ken Raeburns patches to replace the Emacs Lisp engine
>  - Templetons implementation of Elisp in Guile
>
> This branch would initially use Nishidas strategy of just linking Guile
> and providing some simple interoperability functions. The version linked
> would be Templetons Elisp Guile version. So, there would be 2 lisp
> engines in Emacs for a while. When things work well the old Emacs lisp
> engine would be ripped out.
>
> As I'm coming from the Emacs side of things my interest is in getting
> Guile in the Emacs core efficiently. My motivation is somewhat selfish
> because I don't want to re-implement all the interesting features Guile
> has in Emacs, such as dynamic FFI etc..
>
> Of course there are many details to work out. A random example could
> be that we would like a cookie to indicate that eval-buffer should use
> guile rather than elisp for a buffer.
>
> The advantage of the above approach is that it is incremental. We gain
> exposure in the Emacs developer community. Since Emacs 24 is entering
> pretest its a good time to start to think of features for Emacs 25.
>
>
>
> --
> Joakim Verona
>
>
>

Re: ELisp?

[Ken Raeburn]

On Oct 9, 2011, at 15:22, Noah Lavine wrote:
> 
> One question I have is, how much of Emacs' C code can Guile use right
> now? I guess Ken Raeburn's patches are the ones that address that, but
> I don't know what is left to do.

With the stuff I've done, not much execution happens with Guile's engine, and I'm not sure Guile can actually invoke anything from the Elisp side directly.  I did add an "evaluate as Lisp" Scheme function and vice versa at one point (I *think* it was even in the public repository), but it's a pretty minimal level of support.  It was enough to throw together a simple and crude "interactive Guile mode" similar to the interactive Emacs Lisp mode -- ielm -- that lets you type and evaluate expressions in-process in a buffer.  So when you hit return, Emacs Lisp code looked at the s-expression and passed it off to be evaluated as Scheme, though if I recall correctly, strings were passed back and forth between the two environments because symbols are (were) still too different.

> (I also have the same question about ELisp - what of the language is
> left to implement? Is it just adding more builtins?)

In addition to "what of the language is left to implement", I'd suggest that the work remaining should include:

* conformance tests
  ** lots of the Emacs-specific extensions like buffer-local variables and defaults, int-only variables, aliases, etc
  ** interactions between them (e.g., let-bindings plus local variables when switching buffers)
  ** stack overflow protection
  ** out-of-memory handling
  ** error/signal trapping
  ** timeouts, event processing
  ** errors storing to special kinds of variables
  ** weak hashes (make sure excess references aren't kept around due to handling of symbol function slots, object properties, etc, and that GC doesn't take things it shouldn't)
  ** defadvice
  ** debugger? debug-on-entry? debug-on-quit?
* performance tests
  ** high-level - e.g., byte-compile, hanoi, font-lock, hairy regexp search and replace
  ** specific functionality - global/dynamic/local bindings, symbol or text properties
  ** startup time
  ** both speed and memory use; include tests on memory-limited configurations
  ** multiple architectures
     *** including at least one without Scheme->native compilation
  ** multiple OSes
     *** GNU/Linux
     *** Windows (preferably native, not mingw/cygwin)
     *** at least one non-GNU-libc one
     *** the more the merrier
* tests of Lisp/Scheme interactions
  ** expected nil/#f/()/#t/t handling in both Lisp and Scheme
  ** symbol value vs symbol function slot
  ** inter-language calls
  ** interactions of Lisp features with Guile threads (if Emacs Lisp doesn't get threads first)
     *** buffer-local or dynamic bindings and thread creation
     *** is the current buffer a per-thread concept?

Those would be a lot more convincing than "I typed in a bunch of Lisp commands and read my email, and it all looked okay", though of course we want to try those too.  This is a major and fundamental change we're proposing for Emacs, and not all of the Emacs developers are convinced it's a good thing.  Having tests like these (and a lot more for the full Guile-Emacs implementation) may help convince people that we've done our homework.

We don't necessarily have to look better on all the performance numbers.  Though, if we don't look better on most of them, maybe it's a sign that we've got more work to do first.


> A branch on Savannah sounds like a really good idea though. Then at
> least there would be a central thing for people to look at and hack
> on, and more people might get involved. Right now I suspect that most
> people don't know what's happening or where to look. If you set up a
> branch, I for one would love to check it out and perhaps contribute
> some patches. Do you need permission from the Emacs maintainers to do
> that, or anything else?

If there's a specific plan for moving forward, and someone in particular planning to work on it and coordinate, a branch on savannah may be a good idea.  The plan has got to cover dealing with changes happening in the main Emacs development in the meantime (frequent merges or rare, big merges?), stuff like that.  If it's just, "I'll hack on it for a few months and see how it goes, and then see if someone else wants to take over", I'm not sure it'd be worthwhile.

Wherever the work gets done, it's important for everyone to have signed assignment or disclaimer paperwork covering both Emacs and Guile, unless they're going to be working very clearly on just one side or the other.

Ken

Re: ELisp?

[joakim]

Ken Raeburn <raeburn@raeburn.org> writes:

> On Oct 9, 2011, at 15:22, Noah Lavine wrote:
>> 
>> One question I have is, how much of Emacs' C code can Guile use right
>> now? I guess Ken Raeburn's patches are the ones that address that, but
>> I don't know what is left to do.
>
> With the stuff I've done, not much execution happens with Guile's engine, and I'm not sure Guile can actually invoke anything from the Elisp side directly.  I did add an "evaluate as Lisp" Scheme function and vice versa at one point (I *think* it was even in the public repository), but it's a pretty minimal level of support.  It was enough to throw together a simple and crude "interactive Guile mode" similar to the interactive Emacs Lisp mode -- ielm -- that lets you type and evaluate expressions in-process in a buffer.  So when you hit return, Emacs Lisp code looked at the s-expression and passed it off to be evaluated as Scheme, though if I recall correctly, strings were passed back and forth between the two environments because symbols are (were) still too different.

Where is your public repository? I would be interested to have a look.

A String interface is an okay start.

>
>> (I also have the same question about ELisp - what of the language is
>> left to implement? Is it just adding more builtins?)
>
> In addition to "what of the language is left to implement", I'd suggest that the work remaining should include:
>
> * conformance tests
>   ** lots of the Emacs-specific extensions like buffer-local variables and defaults, int-only variables, aliases, etc
>   ** interactions between them (e.g., let-bindings plus local variables when switching buffers)
>   ** stack overflow protection
>   ** out-of-memory handling
>   ** error/signal trapping
>   ** timeouts, event processing
>   ** errors storing to special kinds of variables
>   ** weak hashes (make sure excess references aren't kept around due to handling of symbol function slots, object properties, etc, and that GC doesn't take things it shouldn't)
>   ** defadvice
>   ** debugger? debug-on-entry? debug-on-quit?
> * performance tests
>   ** high-level - e.g., byte-compile, hanoi, font-lock, hairy regexp search and replace
>   ** specific functionality - global/dynamic/local bindings, symbol or text properties
>   ** startup time
>   ** both speed and memory use; include tests on memory-limited configurations
>   ** multiple architectures
>      *** including at least one without Scheme->native compilation
>   ** multiple OSes
>      *** GNU/Linux
>      *** Windows (preferably native, not mingw/cygwin)
>      *** at least one non-GNU-libc one
>      *** the more the merrier
> * tests of Lisp/Scheme interactions
>   ** expected nil/#f/()/#t/t handling in both Lisp and Scheme
>   ** symbol value vs symbol function slot
>   ** inter-language calls
>   ** interactions of Lisp features with Guile threads (if Emacs Lisp doesn't get threads first)
>      *** buffer-local or dynamic bindings and thread creation
>      *** is the current buffer a per-thread concept?
>
> Those would be a lot more convincing than "I typed in a bunch of Lisp commands and read my email, and it all looked okay", though of course we want to try those too.  This is a major and fundamental change we're proposing for Emacs, and not all of the Emacs developers are convinced it's a good thing.  Having tests like these (and a lot more for the full Guile-Emacs implementation) may help convince people that we've done our homework.
>
> We don't necessarily have to look better on all the performance numbers.  Though, if we don't look better on most of them, maybe it's a sign that we've got more work to do first.
>
>
>> A branch on Savannah sounds like a really good idea though. Then at
>> least there would be a central thing for people to look at and hack
>> on, and more people might get involved. Right now I suspect that most
>> people don't know what's happening or where to look. If you set up a
>> branch, I for one would love to check it out and perhaps contribute
>> some patches. Do you need permission from the Emacs maintainers to do
>> that, or anything else?
>
> If there's a specific plan for moving forward, and someone in particular planning to work on it and coordinate, a branch on savannah may be a good idea.  The plan has got to cover dealing with changes happening in the main Emacs development in the meantime (frequent merges or rare, big merges?), stuff like that.  If it's just, "I'll hack on it for a few months and see how it goes, and then see if someone else wants to take over", I'm not sure it'd be worthwhile.

Okay, so a Savannah branch can wait.

Anyway, I think a successful approach must be incremental. Elisp and
Guile can co-exist for the time being.

As a concrete example, one could use guiles libffi integration to offer
the same feature to Emacs. I could personally use it in my xwidgets
emacs branch to call dynamically into webkit rather than having to make
a c wrapper for each and every function.

So, mainline Emacs would offer optional core guile integration for
advanced features. That would be no worse than embedding guile in any
other application and would offer benefits in its own right.

> Wherever the work gets done, it's important for everyone to have signed assignment or disclaimer paperwork covering both Emacs and Guile, unless they're going to be working very clearly on just one side or the other.

I have papers for Emacs at least, and no issue with signing more papers.

>
> Ken
>

-- 
Joakim Verona

Re: ELisp?

[Ken Raeburn]

On Nov 11, 2011, at 04:46, joakim@verona.se wrote:
> Where is your public repository? I would be interested to have a look.

Try git://raeburn.scripts.mit.edu/guile-emacs.git.  But it's about two years old. :-(
One of these days....

> As a concrete example, one could use guiles libffi integration to offer
> the same feature to Emacs. I could personally use it in my xwidgets
> emacs branch to call dynamically into webkit rather than having to make
> a c wrapper for each and every function.

Hmm... this touches on a political issue I'd been avoiding thinking about.  Namely, adding Guile to Emacs, with Guile's new FFI support, would make dynamically loading new executable code into Emacs easy, technically, including non-GPL code written specifically to extend Emacs.  There's been a lot of resistance to that in the past.  See for example http://lists.gnu.org/archive/html/emacs-devel/2003-07/msg00403.html .

On the technical side (ignoring the political/legal angles), I wonder if it would be quicker to drop FFI support into Emacs directly, using an interface based on the Guile one, and use that for now, until the Emacs+Guile work is far enough along to merge.  I suspect your xwidgets code would be ready for integration much sooner than that. :-)

Ken

Re: ELisp?

[joakim]

Ken Raeburn <raeburn@raeburn.org> writes:

> On Nov 11, 2011, at 04:46, joakim@verona.se wrote:
>> Where is your public repository? I would be interested to have a look.
>
> Try git://raeburn.scripts.mit.edu/guile-emacs.git.  But it's about two years old. :-(
> One of these days....
>
>> As a concrete example, one could use guiles libffi integration to offer
>> the same feature to Emacs. I could personally use it in my xwidgets
>> emacs branch to call dynamically into webkit rather than having to make
>> a c wrapper for each and every function.
>
> Hmm... this touches on a political issue I'd been avoiding thinking about.  Namely, adding Guile to Emacs, with Guile's new FFI support, would make dynamically loading new executable code into Emacs easy, technically, including non-GPL code written specifically to extend Emacs.  There's been a lot of resistance to that in the past.  See for example http://lists.gnu.org/archive/html/emacs-devel/2003-07/msg00403.html .

Yes, this seems to have been resolved if the GCC scheme for identifying
GPL libraries is used. Basically GPL libraries expose a symbol declaring GPL compliance.

>
> On the technical side (ignoring the political/legal angles), I wonder if it would be quicker to drop FFI support into Emacs directly, using an interface based on the Guile one, and use that for now, until the Emacs+Guile work is far enough along to merge.  I suspect your xwidgets code would be ready for integration much sooner than that. :-)

Quicker, yes, but less interesting. I suppose I'm looking for an excuse
to do this :)

> Ken

-- 
Joakim Verona

Re: ELisp?

[Ken Raeburn]

On Nov 12, 2011, at 10:00, joakim@verona.se wrote:
>> Hmm... this touches on a political issue I'd been avoiding thinking about.  Namely, adding Guile to Emacs, with Guile's new FFI support, would make dynamically loading new executable code into Emacs easy, technically, including non-GPL code written specifically to extend Emacs.  There's been a lot of resistance to that in the past.  See for example http://lists.gnu.org/archive/html/emacs-devel/2003-07/msg00403.html .
> 
> Yes, this seems to have been resolved if the GCC scheme for identifying
> GPL libraries is used. Basically GPL libraries expose a symbol declaring GPL compliance.

The Linux kernel has a method for dealing with this issue too.  As I understand it, a module declares its license terms, and if they're not GPL, you get a smaller subset of kernel functionality you can access from the module, though you can still load it.

But the Emacs FFI case is different.  Those other modules would have to be written specifically *for* GCC or Linux and licensed appropriately, and with an Emacs FFI we'd want to be able to load, for example, MIT-licensed widget libraries that have other uses not specific to Emacs.  We can't expect all other libraries anyone might want to use to tweak their sources to declare GPL compatibility.

Then there's the question of someday doing static compilation of Guile code to machine code.  Can I load a .o or .so of my own Lisp code without it being GPL-licensed or at least GPL-compatible?

*sigh*  I don't know, I'd rather let RMS and the lawyers worry about it.

>> On the technical side (ignoring the political/legal angles), I wonder if it would be quicker to drop FFI support into Emacs directly, using an interface based on the Guile one, and use that for now, until the Emacs+Guile work is far enough along to merge.  I suspect your xwidgets code would be ready for integration much sooner than that. :-)
> 
> Quicker, yes, but less interesting. I suppose I'm looking for an excuse
> to do this :)

Ah, that works for me.  My excuse was wanting to do things with Emacs while Gnus was collecting data from mail and news servers, using Guile's thread support rather than rewriting all of Gnus to be callback-driven. :-)

Ken

Re: ELisp?

[Noah Lavine]

> Hmm... this touches on a political issue I'd been avoiding thinking about.  Namely, adding Guile to Emacs, with Guile's new FFI support, would make dynamically loading new executable code into Emacs easy, technically, including non-GPL code written specifically to extend Emacs.  There's been a lot of resistance to that in the past.  See for example http://lists.gnu.org/archive/html/emacs-devel/2003-07/msg00403.html .

I haven't thought about the rest of this yet, but I believe what will
happen is probably what GCC did to solve this issue. Specifically,
there is no problem loading dynamic libraries that define a symbol
"this_compiles_with_gplv3", or whatever the specific symbol name is.
And checking for such a symbol could certainly be done with Guile's
FFI.

Of course, whether Emacs wants to be able to load *other* dynamic
libraries is a different question. But Emacs would have to face this
issue with any FFI, and I think Emacs probably wants an FFI (based on
following emacs-devel and thinking about what would make things easier
for Emacs), so they can't avoid these questions anyway. So at least
Guile isn't making things worse.

Noah