GSoC: Emacs Lisp support for GNU Guile

GSoC: Emacs Lisp support for GNU Guile

[Daniel Kraft]

Hi all,

as already discussed briefly with the Guile guys behind the new VM 
thing, I got the idea to implement Emacs Lisp as supported language for 
the Guile VM system.

Below is a proposal for a GSoC project I submitted to GNU as mentoring 
organization;  Karl Berry replied that it sounds interesting but they 
needed to find a mentor / mentors for this project; what do you (Emacs 
and Guile people) think about it?

Thank you very much for your ideas!

Daniel



Your name:
Daniel Kraft


Your email address:
d@domob.eu


Package name and title of the project:
GNU Guile -- implement Emacs Lisp support on top of new VM framework.


Summary:

Guile's new VM/compiler framework allows easy extension and addition of the
supported programming languages, and I would like to utilize it and add 
support
for Emacs Lisp to Guile using this new infrastructure.


Benefits and Deliverables:

Originally, Guile as the GNU scripting language was designed as a language
based on the Scheme lisp dialect; however, as a prime example for use of a
scripting language (and with a huge existing base of scripts) Emacs Lisp is
another popular lisp based scripting language.  With the new capabilities of
Guile, it will be possible to add support of this language to Guile and thus
allow it (and all programs using Guile as scripting engine) to be 
programmed not
only in Scheme but also Emacs lisp; thus, it will also be possible to 
utilize
all the existing elisp code for creating even more nifty and useful scripts.

My project's work will be the basic implementation of Emacs Lisp for Guile,
which of course includes a comprehensive test-suite for making sure it works
(and to catch future regressions easily) as well as documentation, so the
overall result will be of the best possible use to the community.  For the
course of the project, it may also be necessary (or say, "a side effect") to
work on extensions and bug-fixes to the core Guile system as needed, which
will in those cases hopefully be another benefit to Guile.

Finally, I hope to get to know the Guile community and become a part of 
it in
general, not only for Summer of Code because it is a very interesting 
project
in general to me; and the new VM work of course adds a lot more reasons for
being interested as well...


Plan / Timeline:

Until the summer (when I still have to spent most time for my studies) I 
want
to use the time I can find to get familiar with the community and codebase a
little bit (maybe working on small bugs and such) as well as thinking about
and discussing the plans for the project with my mentor.

When I have holidays (end of June) I will be able to work full time on the
project; my plan is to get a mainly working implementation during July 
(maybe
missing some features, but already usable) so that I have time to put on
the finishing touches, test it, write documentation and all that in August.


Communication:

Depends on the requests from my mentor / the community, but I think the
public development mailing lists and private email should be fine; other 
ideas
are instant messaging / IRC.


Qualification:

I'm an undergraduate student with subjects Mathematics and Physics at the
University of Graz, Austria; however, I've been interested in computer
science and programming in my spare time for already about ten years and
have experience with some variety of languages, including C, C++, Fortran
as well as functional and logic languages (Haskell, Scheme and Prolog) and
things like bash scripting or Makefiles.  I was employed half a year before
starting my studies as programmer and have a registered small company in
Austria for IT services I do (but very little of course) in parallel to
my work for university.  I participated two times for Austria in the
International Olympiad in Informatics and got a basic understanding
of algorithms and data structures in the course of this.

Regarding my experiences with Free Software, I'm a user of GNU/Linux and
only free software for my system for three years now, and were also
involved in some projects in the past in addition to developing some own
small projects released under GPL.  Projects I work(ed) with include
FreeWRL, the Mozilla Foundation and currently GCC.

Especially interesting for this project I propose might be that I 
already have
some experience with implementing parsers / compilers, as I implemented 
already
some of my own scripting languages as well as the work on the Fortran 
front-end
of GCC.

Re: GSoC: Emacs Lisp support for GNU Guile

[Clinton Ebadi]

Daniel Kraft <d@domob.eu> writes:

> Hi all,
>
> as already discussed briefly with the Guile guys behind the new VM
> thing, I got the idea to implement Emacs Lisp as supported language
> for the Guile VM system.
>
> Below is a proposal for a GSoC project I submitted to GNU as mentoring
> organization;  Karl Berry replied that it sounds interesting but they
> needed to find a mentor / mentors for this project; what do you (Emacs
> and Guile people) think about it?

This is an excellent plan!

There is already a working elisp->scheme translator for the interpeter
in lang/elisp. I've poked about in it and it would be fairly
straightfoward to port it to the VM by retargetting it to GHIL and
replacing the @fop and @bind operators in the interpeter with a bit of
GHIL implementing the same behavior.

As a general implemention order I suggest:

 - Replace @fop and @bind with a GHIL fragment implementing the same
   behavior
 - Integrate the elisp->scheme translator into the VM compiler framework
 - Retarget the elisp translator from Scheme to GHIL

If there is time left (the above tasks should take maybe a month,
perhaps more) then a few more tasks would be useful as far as cleaning
up the implementation of Guile:

 - Remove handling of NIL in Scheme. This special casing is, in my
   opinion, distressingly unclean. This would require adding a new
   ELISP-IF form that accepts #f/nil as false values
 - Reimplement the basic elisp list operators in elisp/ghil rather than
   using the existing Scheme implementations
 - [Potentially] Remove value/function slots from Guile's symbol type
   and implement the Lisp-2ness of elisp in another way
 - [Assuming the above is done; if not ((fref SYMBOL) ...)  works
   already] Implement a convenient way to call elisp functions from
   Scheme. Something like (funcall SYMBOL ...)

And then use any remaining time to fill in any useful bits
lang/elisp/primitives/*.


-- 
<captain_krunk> ntk is currently using "telnet fyodor 25" to send email

Re: GSoC: Emacs Lisp support for GNU Guile

[Neil Jerram]

Daniel Kraft <d@domob.eu> writes:

> Hi all,
>
> as already discussed briefly with the Guile guys behind the new VM
> thing, I got the idea to implement Emacs Lisp as supported language
> for the Guile VM system.
>
> Below is a proposal for a GSoC project I submitted to GNU as mentoring
> organization;  Karl Berry replied that it sounds interesting but they
> needed to find a mentor / mentors for this project; what do you (Emacs
> and Guile people) think about it?

Well I guess I could mentor this...  On the other hand, if anyone else
is very keen to, I'd be happy to let them!  (Simply because of time
pressure.)

Having worked on Guile/Elisp on and off for some time, I think it
would really help if you had a concrete goal in mind - i.e. something
more concrete than implementing Emacs Lisp in general on the VM.

I don't want to preempt your own ideas, but this could take the form
of

- identifying some existing (non-trivial) Emacs Lisp library or
  application

- thinking of some cool extra thing that could be done with that, if
  it was combined with Guile (and maybe other guile libraries, such as
  guile-gnome)

- then the concrete goal would be to get Elisp/VM working well enough
  to run that existing Elisp code and do the cool extra thing.

If you followed this kind of approach, note that it would also need
work - in addition to the Elisp/VM translation - to implement whatever
Emacs primitives the Elisp code relies on.  That would make the
project as a whole less pure-language-translation, but (in my view)
more real and useful.

What do you think?

Regards,

    Neil

Re: GSoC: Emacs Lisp support for GNU Guile

[Neil Jerram]

This email is going to be about Guile details that the Emacs crowd
probably won't be interested in, so I've removed emacs-devel.

Clinton Ebadi <clinton@unknownlamer.org> writes:

> There is already a working elisp->scheme translator for the interpeter
> in lang/elisp. I've poked about in it and it would be fairly
> straightfoward to port it to the VM by retargetting it to GHIL and
> replacing the @fop and @bind operators in the interpeter with a bit of
> GHIL implementing the same behavior.
>
> As a general implemention order I suggest:
>
>  - Replace @fop and @bind with a GHIL fragment implementing the same
>    behavior
>  - Integrate the elisp->scheme translator into the VM compiler framework
>  - Retarget the elisp translator from Scheme to GHIL

Agreed with all that.  Also I think there is still some support in
Guile's reader code for Elisp stuff such as [...] vectors.  That
should probably be removed and replaced by a proper Elisp parser
written in Scheme.

> If there is time left (the above tasks should take maybe a month,
> perhaps more) then a few more tasks would be useful as far as cleaning
> up the implementation of Guile:
>
>  - Remove handling of NIL in Scheme. This special casing is, in my
>    opinion, distressingly unclean. This would require adding a new
>    ELISP-IF form that accepts #f/nil as false values

I'm not clear what you mean here.  Can you expand on this?

>  - Reimplement the basic elisp list operators in elisp/ghil rather than
>    using the existing Scheme implementations

Sounds right, but do you have a specific reason?

>  - [Potentially] Remove value/function slots from Guile's symbol type
>    and implement the Lisp-2ness of elisp in another way

That sounds too dull for GSoC!  More a matter of API continuity than
of anything technically interesting.  Or am I missing something?

>  - [Assuming the above is done; if not ((fref SYMBOL) ...)  works
>    already] Implement a convenient way to call elisp functions from
>    Scheme. Something like (funcall SYMBOL ...)

I thought that was already possible... yes, using elisp-function,
use-elisp-file or use-elisp-library.

Regards,
        Neil

Re: GSoC: Emacs Lisp support for GNU Guile

[Andy Wingo]

Hi Daniel,

On Tue 31 Mar 2009 12:44, Daniel Kraft <d@domob.eu> writes:

> as already discussed briefly with the Guile guys behind the new VM
> thing, I got the idea to implement Emacs Lisp as supported language for
> the Guile VM system.

This sounds great! I'd love to assist. As the fellow who's hacked most
on Guile's VM, I can offer "mentorship" if you like, on the Guile
side -- though I am not as knowledgable about Emacs as I'd like to be.

I'll direct replies to further Guile-type details to the Guile list, but
in summary:

> Originally, Guile as the GNU scripting language was designed as a language
> based on the Scheme lisp dialect; however, as a prime example for use of a
> scripting language (and with a huge existing base of scripts) Emacs Lisp is
> another popular lisp based scripting language.  With the new capabilities of
> Guile, it will be possible to add support of this language to Guile and thus
> allow it (and all programs using Guile as scripting engine) to be
> programmed not
> only in Scheme but also Emacs lisp; thus, it will also be possible to
> utilize
> all the existing elisp code for creating even more nifty and useful
> scripts.

YMMV, but I think that elisp is most useful within the context of Emacs
itself -- its data types, its libraries, its runtime. So after getting
Emacs Lisp's semantics to compile, perhaps as a "if time allows" thing,
defining a Guile implementation of emacs/src/lisp.h would allow Guile to
slip into Emacs with minimal modification of C sources.

Or perhaps it's better to limit this SOC to just the Scheme side of
things? Dunno.

In any case, I think with "world enough and time", Guile can do whatever
it takes to provide a faster, better Elisp implementation -- including
adding Elisp-specific ops to its VM, if necessary.

> Communication:
>
> Depends on the requests from my mentor / the community, but I think the
> public development mailing lists and private email should be fine; other
> ideas
> are instant messaging / IRC.

I'm often on #guile, with CET as my timezone. Between there and the
mailing lists we should be good, avoiding private email.

I hope this works out!

Andy
-- 
http://wingolog.org/

Re: GSoC: Emacs Lisp support for GNU Guile

[Andy Wingo]

Hi Clinton,

On Tue 31 Mar 2009 13:28, Clinton Ebadi <clinton@unknownlamer.org> writes:

> This is an excellent plan!

Agreed!

> There is already a working elisp->scheme translator for the interpeter
> in lang/elisp.

Yes, that would be a great starting point.

> replacing the @fop and @bind operators in the interpeter with a bit of
> GHIL implementing the same behavior.

We can add ops to the VM if needed.

>  - Remove handling of NIL in Scheme.

I used to think as you do, but Neil convinced me otherwise. For me, %nil
can stay.

>  - Reimplement the basic elisp list operators in elisp/ghil rather than
>    using the existing Scheme implementations

This could be interesting but doesn't seem necessary

>  - [Potentially] Remove value/function slots from Guile's symbol type
>    and implement the Lisp-2ness of elisp in another way

Dunno.

>  - [Assuming the above is done; if not ((fref SYMBOL) ...)  works
>    already] Implement a convenient way to call elisp functions from
>    Scheme. Something like (funcall SYMBOL ...)

As Neil mentions, this seems possible already -- though not with VM
code.

Cheers,

Andy
-- 
http://wingolog.org/

Re: GSoC: Emacs Lisp support for GNU Guile

[Andy Wingo]

Hi Neil :)

Though I was not the person to whom the question was addressed, a
comment :)

On Tue 31 Mar 2009 15:23, Neil Jerram <neil@ossau.uklinux.net> writes:

> If you followed this kind of approach, note that it would also need
> work - in addition to the Elisp/VM translation - to implement whatever
> Emacs primitives the Elisp code relies on.  That would make the
> project as a whole less pure-language-translation, but (in my view)
> more real and useful.

I keep thinking that it should be possible to write some kind of C shim
so that Guile could implement the Emacs C "API". That way we keep the
existing C code working, we keep the fine-tuned implementations and
semantics, and we can move on from "re-implementing" to "implementing"
-- e.g. adding to Emacs instead of just reproducing it. 

Happy hacking,

Andy
-- 
http://wingolog.org/

Re: GSoC: Emacs Lisp support for GNU Guile

[Neil Jerram]

Andy Wingo <wingo@pobox.com> writes:

> Hi Daniel,
>
> On Tue 31 Mar 2009 12:44, Daniel Kraft <d@domob.eu> writes:
>
>> as already discussed briefly with the Guile guys behind the new VM
>> thing, I got the idea to implement Emacs Lisp as supported language for
>> the Guile VM system.
>
> This sounds great! I'd love to assist. As the fellow who's hacked most
> on Guile's VM, I can offer "mentorship" if you like, on the Guile
> side

Thanks, Andy, that's fantastic.  I have little doubt that you would be
a more available mentor than me.  But I'll still follow and chip in!

> YMMV, but I think that elisp is most useful within the context of Emacs
> itself -- its data types, its libraries, its runtime.

I agree, but I think there are some boundaries that could be drawn.
E.g. choosing to use a different GUI approach (or widget set) than
Emacs's own one, but still using all the rest of the runtime and
libraries.

> So after getting
> Emacs Lisp's semantics to compile, perhaps as a "if time allows" thing,
> defining a Guile implementation of emacs/src/lisp.h would allow Guile to
> slip into Emacs with minimal modification of C sources.

I started on that once.  It rapidly started to feel hopeless - but
that was because of the code management aspect - not having Git in
those days, and not having a prior agreement with the Emacs people
that this would be a supported exercise (in some sense).

Now that we have nice code management tools, it should be a lot more
promising!

Regards,
        Neil

Re: GSoC: Emacs Lisp support for GNU Guile

[Mike Gran]

On Tue, 2009-03-31 at 16:15 -0700, Andy Wingo wrote:

Andy et al-

> I keep thinking that it should be possible to write some kind of C shim
> so that Guile could implement the Emacs C "API". That way we keep the
> existing C code working, we keep the fine-tuned implementations and
> semantics, and we can move on from "re-implementing" to "implementing"
> -- e.g. adding to Emacs instead of just reproducing it. 

FWIW, In the context of a GSoC project, it would be fairly easy to
create a testbed for Emacs Lisp in Scheme by shimming GNU Zile.  Zile
is a minimalist Emacs clone with a Lisp interpreter that, at first
glance, would be simple to modify for Guile.

-Mike Gran

Re: GSoC: Emacs Lisp support for GNU Guile

[Stefan Monnier]

> as already discussed briefly with the Guile guys behind the new VM thing,
> I got the idea to implement Emacs Lisp as supported language for the Guile
> VM system.

I won't have time to mentor it, but I'd like to point out some relevant
directions in Emacs's future: as some of you know, other than hacking on
Emacs and its weakly typed C code and dynamically typed Lisp code, my
day job revolves around static typing and type theory, which makes for
some frustrations.  But while thinking of how to integrate Miles's
lexical scoping branch, it occurred to me that it would be a good
opportunity to try and combine both lives: with modern type systems (as
seen in Coq, Agda, TSCB, and friends), we can easily accomodate any
valid Lisp code.  While the plan is not fully fledged out yet, I've been
talking to Richard and Chong about it and we'll probably end up using
a variant of Agda's type system in Emacs-24: the code should be 100%
backward compatible, except that to take advantage of the newer features
(like closures, unboxed values, tail recursion, ...), you might need to
add a few simple type annotations at key places, like:

   (defun append (l1 l2)
     (declare (type _∷_ <↓> x ⊛ sequence Γ))
     (if (null l1)
         (withtype l2 ∃ \i -> sequence (i ⊕ (1 + max bal)))
       (cons (car l1) (append (cdr l1) l2))))

As I said, it's not fully fledged out, so take the above example with
a grain of salt.  I didn't really want to talk about it in public yet,
but figured that maybe now would be the right time after all, especially
since Guile people might be interested in it as well.


        Stefan

Re: GSoC: Emacs Lisp support for GNU Guile

[Daniel Kraft]

Andy Wingo wrote:
> Hi Daniel,
> 
> On Tue 31 Mar 2009 12:44, Daniel Kraft <d@domob.eu> writes:
> 
>> as already discussed briefly with the Guile guys behind the new VM
>> thing, I got the idea to implement Emacs Lisp as supported language for
>> the Guile VM system.
> 
> This sounds great! I'd love to assist. As the fellow who's hacked most
> on Guile's VM, I can offer "mentorship" if you like, on the Guile
> side -- though I am not as knowledgable about Emacs as I'd like to be.

Thank you all for the quick replies and offers for mentoring!  I'd be 
really glad to work on this project and hopefully to bring the elisp 
support to guile, which really seems something valuable to the community!

And it seems mentoring won't be any problem...  I'll mention this to the 
GNU guy involved, so hopefully my proposal can finally be accepted.

>> Originally, Guile as the GNU scripting language was designed as a language
>> based on the Scheme lisp dialect; however, as a prime example for use of a
>> scripting language (and with a huge existing base of scripts) Emacs Lisp is
>> another popular lisp based scripting language.  With the new capabilities of
>> Guile, it will be possible to add support of this language to Guile and thus
>> allow it (and all programs using Guile as scripting engine) to be
>> programmed not
>> only in Scheme but also Emacs lisp; thus, it will also be possible to
>> utilize
>> all the existing elisp code for creating even more nifty and useful
>> scripts.
> 
> YMMV, but I think that elisp is most useful within the context of Emacs
> itself -- its data types, its libraries, its runtime. So after getting
> Emacs Lisp's semantics to compile, perhaps as a "if time allows" thing,
> defining a Guile implementation of emacs/src/lisp.h would allow Guile to
> slip into Emacs with minimal modification of C sources.

So far I've only thought about elisp as "yet another language" but 
you're of course right, it probably consists mainly (or at least 
importantly) of its bindings to Emacs.  While I've no particular 
experience there (at the moment), the idea to use Zile as test-bed for 
the SoC project and simply for getting started sounds reasonable to me.

>> Communication:
>>
>> Depends on the requests from my mentor / the community, but I think the
>> public development mailing lists and private email should be fine; other
>> ideas
>> are instant messaging / IRC.
> 
> I'm often on #guile, with CET as my timezone. Between there and the
> mailing lists we should be good, avoiding private email.

Cool, I'll try to drop in some time near (when I've got some time spare, 
which is unfortunatelly a little spare during the week at the moment)! 
Timezone is CEST for me, too :D

Yours,
Daniel

-- 
Done:  Arc-Bar-Cav-Ran-Rog-Sam-Tou-Val-Wiz
To go: Hea-Kni-Mon-Pri

Re: GSoC: Emacs Lisp support for GNU Guile

[Andy Wingo]

Hi Stefan,

On Wed 01 Apr 2009 06:31, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>    (defun append (l1 l2)
>      (declare (type _∷_ <↓> x ⊛ sequence Γ))

Another tack would be something like Typed Scheme, from the PLT folks. I
assume you've seen it? It makes particular sense in the context of
modules, where interprocedural calls within a typed module have no type
checking.

Probably not something for Daniel, however ;-)

Cheers,

Andy
-- 
http://wingolog.org/

Re: GSoC: Emacs Lisp support for GNU Guile

[Chong Yidong]

Andy Wingo <wingo@pobox.com> writes:

> On Wed 01 Apr 2009 06:31, Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>    (defun append (l1 l2)
>>      (declare (type _∷_ <↓> x ⊛ sequence Γ))
>
> Another tack would be something like Typed Scheme, from the PLT folks. I
> assume you've seen it? It makes particular sense in the context of
> modules, where interprocedural calls within a typed module have no type
> checking.

You might want to check the date of Stefan's message.

Re: GSoC: Emacs Lisp support for GNU Guile

[Andy Wingo]

On Fri 03 Apr 2009 13:31, Chong Yidong <cyd@stupidchicken.com> writes:

> Andy Wingo <wingo@pobox.com> writes:
>
>> On Wed 01 Apr 2009 06:31, Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>    (defun append (l1 l2)
>>>      (declare (type _∷_ <↓> x ⊛ sequence Γ))
>>
>> Another tack would be something like Typed Scheme, from the PLT folks. I
>> assume you've seen it? It makes particular sense in the context of
>> modules, where interprocedural calls within a typed module have no type
>> checking.
>
> You might want to check the date of Stefan's message.

I considered that possibility. People who use these type systems are
crazy every day of the year though, so as I don't know Stefan, I had to
give him the benefit of the doubt :)

Andy
-- 
http://wingolog.org/

Re: GSoC: Emacs Lisp support for GNU Guile

[Richard M Stallman]

     - Remove handling of NIL in Scheme. This special casing is, in my
       opinion, distressingly unclean. This would require adding a new
       ELISP-IF form that accepts #f/nil as false values

We carefully designed ways to handle nil, #f and () in Guile
so as to be able to intermix Lisp and Scheme programs.
That was 14 years ago, and I don't recall what we decided.  But I think
the idea was that #f and () would be eq when viewed
from Lisp, but not eq when viewed from Scheme.

I don't know whether this design was actually implemented.

We should not make changes in this based on a rigid idea that
something is "unclean".  To make the two languages interoperate
properly on the same data requires careful design, and it's
possible that something less than totally elegant is the only
way to make it work at all.

Re: GSoC: Emacs Lisp support for GNU Guile

[Ludovic Courtès]

Hey,

Andy Wingo <wingo@pobox.com> writes:

> On Wed 01 Apr 2009 06:31, Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>    (defun append (l1 l2)
>>      (declare (type _∷_ <↓> x ⊛ sequence Γ))
>
> Another tack would be something like Typed Scheme, from the PLT folks.

Bigloo has syntactic support for static typing, which it also uses to
optimize code:

  http://www-sop.inria.fr/mimosa/fp/Bigloo/doc/bigloo-25.html#Explicit-typing

(I'm not fond of this syntax, though.)

Thanks,
Ludo'.

Re: GSoC: Emacs Lisp support for GNU Guile

[Samuel Bronson]

On Fri, Apr 3, 2009 at 4:31 PM, Chong Yidong <cyd@stupidchicken.com> wrote:
> Andy Wingo <wingo@pobox.com> writes:
>
>> On Wed 01 Apr 2009 06:31, Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>    (defun append (l1 l2)
>>>      (declare (type _∷_ <↓> x ⊛ sequence Γ))
>>
>> Another tack would be something like Typed Scheme, from the PLT folks. I
>> assume you've seen it? It makes particular sense in the context of
>> modules, where interprocedural calls within a typed module have no type
>> checking.
>
> You might want to check the date of Stefan's message.

... And notice how the syntax in that message isn't even close to valid Agda!

Re: GSoC: Emacs Lisp support for GNU Guile

[Stefan Monnier]

> ... And notice how the syntax in that message isn't even close to valid Agda!

That is unfair: I copied the type annotations from random places in the
Agda library (and then edited them to make them more interesting).


        Stefan


PS: Of course, any sequence of chars (especially funny Unicode symbols)
is potentially valid Agda code (as long as you provide the appropriate
context).