A Modest Proposal

A Modest Proposal

[Noah Lavine]

Hello Guile and Clisp developers,

I'm writing to talk about vague big-picture ideas, but please bear
with me for a minute, because I think this could be useful.

I noticed in the recent GNU Summer of Code applications (I'm a mentor
for Guile) that CLisp wants to become embeddable, and embed into Emacs
as a first test case. That would make Clisp an embeddable Lisp
implementation with a bytecode-interpreting virtual machine, compiler,
interpreter, debugger, etc. A very cool thing. Clisp developers might
not be aware that that is exactly what GNU Guile is - a
bytecode-interpreting virtual machine with a compiler, interpreter,
debugger, and collection of useful tools. Guile is already embeddable,
because that was one of Guile's original goals, but the difference is
not so big. Guile also has a summer of code projects to embed itself
into Emacs, in fact.

It seems to me that it is time for Guile and CLisp to consider working
together, because if we don't work together then I think we're both
going to do the same work twice, separately.

This depends greatly on what CLisp's goals are as a project, and I do
not know those. Maybe you have very different goals than Guile, in
which case we might not gain anything by working together. But I do
have a feeling that we are both evolving towards the same place, and
if so, I think it would be nice to cooperate some along the way.

Noah

Re: A Modest Proposal

[Pascal J. Bourguignon]

Noah Lavine <noah.b.lavine@gmail.com> writes:

> Hello Guile and Clisp developers,
>
> I'm writing to talk about vague big-picture ideas, but please bear
> with me for a minute, because I think this could be useful.
>
> I noticed in the recent GNU Summer of Code applications (I'm a mentor
> for Guile) that CLisp wants to become embeddable, and embed into Emacs
> as a first test case. That would make Clisp an embeddable Lisp
> implementation with a bytecode-interpreting virtual machine, compiler,
> interpreter, debugger, etc. A very cool thing. Clisp developers might
> not be aware that that is exactly what GNU Guile is - a
> bytecode-interpreting virtual machine with a compiler, interpreter,
> debugger, and collection of useful tools. Guile is already embeddable,
> because that was one of Guile's original goals, but the difference is
> not so big. Guile also has a summer of code projects to embed itself
> into Emacs, in fact.
>
> It seems to me that it is time for Guile and CLisp to consider working
> together, because if we don't work together then I think we're both
> going to do the same work twice, separately.
>
> This depends greatly on what CLisp's goals are as a project, and I do
> not know those. Maybe you have very different goals than Guile, in
> which case we might not gain anything by working together. But I do
> have a feeling that we are both evolving towards the same place, and
> if so, I think it would be nice to cooperate some along the way.


I think we should first compare the virtual machines.


If no obvious impossibility is observed, then perhaps modifying the
compiler of clisp to generate guile VM code would be an easy path to
obtain a CL implementation running on guile VM.  (This would disable the
interpreter in clisp, since it is implemented in C).

Or, vice-versa, if the VM of clisp has advantages (but if guile has
continuations, this would be lacking in the clisp VM).



In general, there may be a need for a very good lisp virtual machine to
run and integrate lisp code in general (CL, various schemes, and other
sorts of lisp-like languages, we could include perhaps implementations
of python,  ruby,  smalltalk,  javascript, etc).  From well afar, it
looks like the JVM is not good enough for lisp (ABCL, Clojure, seem to
have some difficulties to implement some basic lisp features on the
JVM).


This VGLVM would:

- be embeddable in applications,

- include a FFI to native code,

- be more performant than the JVM,

- be natively multithreaded,

- have a good real-time multithreaded garbage collector,

- possibly have a JIT and/or a retargetting compiler,

- allow easy (FFI-less) communication between the different languages
  targetting it (for example, when we run pseudo, we can call CL
  function from scheme and scheme functions from CL)..



-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.


------------------------------------------------------------------------------
Forrester Wave Report - Recovery time is now measured in hours and minutes
not days. Key insights are discussed in the 2010 Forrester Wave Report as
part of an in-depth evaluation of disaster recovery service providers.
Forrester found the best-in-class provider in terms of services and vision.
Read this report now!  http://p.sf.net/sfu/ibm-webcastpromo
_______________________________________________
clisp-devel mailing list
clisp-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clisp-devel

Re: A Modest Proposal

[Noah Lavine]

Hello,

> I think we should first compare the virtual machines.
>
>
> If no obvious impossibility is observed, then perhaps modifying the
> compiler of clisp to generate guile VM code would be an easy path to
> obtain a CL implementation running on guile VM.  (This would disable the
> interpreter in clisp, since it is implemented in C).
>
> Or, vice-versa, if the VM of clisp has advantages (but if guile has
> continuations, this would be lacking in the clisp VM).
>
>
>
> In general, there may be a need for a very good lisp virtual machine to
> run and integrate lisp code in general (CL, various schemes, and other
> sorts of lisp-like languages, we could include perhaps implementations
> of python,  ruby,  smalltalk,  javascript, etc).  From well afar, it
> looks like the JVM is not good enough for lisp (ABCL, Clojure, seem to
> have some difficulties to implement some basic lisp features on the
> JVM).

I think what you're saying may be good, but I'm very hesitant to move
ahead without lots of people agreeing. I would be very afraid that we
would have friction as two groups of people who hadn't known each
other previously tried to work together. For instance, what if some of
us want to move the VM in one direction and others in another
direction? Or what if the groups have different standards for when we
make changes to the VM?

On the other hand, there could be significant advantages. I see two
big ones: first, that interoperability means that when one of us
scratches an itch, all of us benefit, and second, that users of
embedded interpreters would have to learn less. If the GNU project has
only one embedded interpreter, then once you know how to use that, you
can modify all GNU programs.

So I think it might be good in the end if we all worked together, but
I think it is a rather big step. What do other people think?

Finally, to answer a few or your concrete points:

> This VGLVM would:
>
> - be embeddable in applications,
>
> - include a FFI to native code,
>
> - be more performant than the JVM,
>
> - be natively multithreaded,
>
> - have a good real-time multithreaded garbage collector,
>
> - possibly have a JIT and/or a retargetting compiler,
>
> - allow easy (FFI-less) communication between the different languages
>  targetting it (for example, when we run pseudo, we can call CL
>  function from scheme and scheme functions from CL)..

Guile already has 1, 2, and 4, and we are planning both parts of 6 for
the near future. I think 7 would come automatically for languages that
shared a VM. 3 and 5 I don't know anything about.

It's also worth noting that Guile is already set up to compile
multiple languages to its VM, so if we decide to go that route, some
of the infrastructure is already there.

Noah

Re: A Modest Proposal

[William ML Leslie]

On 14 April 2011 01:23, Pascal J. Bourguignon <pjb@informatimago.com> wrote:
> In general, there may be a need for a very good lisp virtual machine to
> run and integrate lisp code in general (CL, various schemes, and other
> sorts of lisp-like languages, we could include perhaps implementations
> of python,  ruby,  smalltalk,  javascript, etc).  From well afar, it
> looks like the JVM is not good enough for lisp (ABCL, Clojure, seem to
> have some difficulties to implement some basic lisp features on the
> JVM).

The problem of language integration has become common enough that we
already have the opportunity to make some observations about what did
work, and what didn't.  A concerted approach to working with distinct
object spaces / models would be a mammoth task, it is entirely a
different undertaking to integrating two particular languages.

The particular issue I find is that the mechanisms for defining how
two object spaces interact don't compose well.  If you work in a Java
environment, for example, you may know how to interact with or create
Java objects from JRuby, Kawa, Jython, E and Clojure, but that doesn't
let you interact between any of these languages in a sane way.  Quick,
what is the process for reifying my Jython or Kawa function as a
callable in JRuby?

This common-denominator approach applies equally well outside these
safe bytecode VM environments - there are all sorts of mechanisms for
generating FFI code for specific VMs given a C or C++ interface.  For
the most part, they make you feel like you are writing C or C++.

Another thing worth thinking about involves promoting idiomatic usage
despite differences in languages.  If you've ever implemented a
language inside a VM that was not designed for it, you will know what
I mean when I ask questions like:

0. Objects in javascript are maps from string keys to any value.  If
they are to be cast as hashes or dictionaries when passed to a
language that allows non-string keys, and the language adds a
non-string key, what happens to the javascript object?

1. How are "other" languages to make or handle multi-valued returns?

2. What happens to cons cells at the boundary between languages where
one has immutable cons cells, the other mutable?

3. Further, what happens when a language with linked lists as the
default sequence type and one with arrays as the default sequence type
interact?  Consider, for example, the various options that pymacs
provides.

4. What about operators, and the various mechanisms and naming
conventions, single/multiple dispatch, AND the guarantees provided by
each operator mechanism in any given language?

5. What do you want to do about subclassing and multiple inheritance?
In a single-inheritance language like Java, the compiler is free to
assume that in

  if (x instanceof Foo) { }
  if (x instanceof Bar) { }

that if Bar and Foo are distinct concrete classes, and one is not a
subclass of the other, the second if can be replaced with an else if.
Method resolution order is not consistent between MI languages,
either, and may be further complicated by how they are to interact
with generic functions.

6. Languages may also make type-related requirements that need to be
checked somehow; such as some languages not permitting the initialiser
to be called more than once, final methods, &c.

7. In what way to generic functions and methods interact?  How about
classes and typeclasses?

8. What happens when a language that is strict about arity interacts
with a language that does not?  Does calling a javascript function
from python with the wrong number of arguments raise a TypeError, or
silently pass undefined?  Does the inverse pass None?

9. Do the error handling and exception mechanisms intersect - say,
what Java exception does my Elisp args-out-of-range become?  Exception
hierarchies can be *very* different across object spaces.

10. How do methods overloaded on static type look from a dynamic
language? Oh, and what about virtual/non-virtual methods and extension
methods as on ecma CLI?

I'm not saying that these questions don't have answers, but I am
saying that in order for the abstraction to compose well, there needs
to be a way to express the solutions to problems like these, and
preferably, a way to drop down to a lower level and specify a
different behaviour.  This is by no means a complete list (although as
you can probably see, it's a problem I've meditated on regularly),
just a taste.

However: Guile was *designed* to be a multi-language VM, and has a far
nicer object model and MOP than something like the JVM or CLI.  If
scheme were the common denominator, it might be so much nicer than
what we've seen before that the jarring that comes from crossing
several object space boundaries is not as severe.  I just mean to be
careful about whether you are suggesting moving to a single VM,
implementing an automated FFI tool with several backends, or solving
this problem generally, because they are each projects of very
different scale.

-- 
William Leslie

Re: A Modest Proposal

[Ludovic Courtès]

Hi,

William ML Leslie <william.leslie.ttg@gmail.com> writes:

> 0. Objects in javascript are maps from string keys to any value.  If
> they are to be cast as hashes or dictionaries when passed to a
> language that allows non-string keys, and the language adds a
> non-string key, what happens to the javascript object?

For a simpler example, see Elisp’s #nil in Guile.  The question of how
Scheme code should handle it wasn’t an easy one...

Thanks,
Ludo’.

Re: A Modest Proposal

[Noah Lavine]

These are all good points. Integrating different languages is hard.

Quite frankly, I am not yet sure that sharing a VM would be a good
idea for us, but as we talk more about it it is seeming more and more
reasonable. I sent my original message because I saw that CLisp was
becoming an embeddable lisp interpreter with an architecture similar
to Guile's, and it seemed to me that the two projects were so similar
that they might be able to collaborate productively.

I think the idea of moving to a common VM might work, because Scheme
and Common Lisp are so much closer than even Scheme and Python, or two
other similar languages. I think almost all of the semantics would map
nicely from one language to the other (although I haven't thought
about it very hard yet). It would be even easier with Guile than with
another Scheme, because Guile has already worked on integrating ELisp,
which shares some features with CL (i.e. uses nil as #f and '()).
However, I haven't thought hard about integrating Guile and CL yet.

On the surface, though, I think it could have advantages not
necessarily for us, but for other GNU projects and users, which I
outlined in a previous email. What do other people think of the idea?

Noah

On Wed, Apr 13, 2011 at 10:06 PM, William ML Leslie
<william.leslie.ttg@gmail.com> wrote:
> On 14 April 2011 01:23, Pascal J. Bourguignon <pjb@informatimago.com> wrote:
>> In general, there may be a need for a very good lisp virtual machine to
>> run and integrate lisp code in general (CL, various schemes, and other
>> sorts of lisp-like languages, we could include perhaps implementations
>> of python,  ruby,  smalltalk,  javascript, etc).  From well afar, it
>> looks like the JVM is not good enough for lisp (ABCL, Clojure, seem to
>> have some difficulties to implement some basic lisp features on the
>> JVM).
>
> The problem of language integration has become common enough that we
> already have the opportunity to make some observations about what did
> work, and what didn't.  A concerted approach to working with distinct
> object spaces / models would be a mammoth task, it is entirely a
> different undertaking to integrating two particular languages.
>
> The particular issue I find is that the mechanisms for defining how
> two object spaces interact don't compose well.  If you work in a Java
> environment, for example, you may know how to interact with or create
> Java objects from JRuby, Kawa, Jython, E and Clojure, but that doesn't
> let you interact between any of these languages in a sane way.  Quick,
> what is the process for reifying my Jython or Kawa function as a
> callable in JRuby?
>
> This common-denominator approach applies equally well outside these
> safe bytecode VM environments - there are all sorts of mechanisms for
> generating FFI code for specific VMs given a C or C++ interface.  For
> the most part, they make you feel like you are writing C or C++.
>
> Another thing worth thinking about involves promoting idiomatic usage
> despite differences in languages.  If you've ever implemented a
> language inside a VM that was not designed for it, you will know what
> I mean when I ask questions like:
>
> 0. Objects in javascript are maps from string keys to any value.  If
> they are to be cast as hashes or dictionaries when passed to a
> language that allows non-string keys, and the language adds a
> non-string key, what happens to the javascript object?
>
> 1. How are "other" languages to make or handle multi-valued returns?
>
> 2. What happens to cons cells at the boundary between languages where
> one has immutable cons cells, the other mutable?
>
> 3. Further, what happens when a language with linked lists as the
> default sequence type and one with arrays as the default sequence type
> interact?  Consider, for example, the various options that pymacs
> provides.
>
> 4. What about operators, and the various mechanisms and naming
> conventions, single/multiple dispatch, AND the guarantees provided by
> each operator mechanism in any given language?
>
> 5. What do you want to do about subclassing and multiple inheritance?
> In a single-inheritance language like Java, the compiler is free to
> assume that in
>
>  if (x instanceof Foo) { }
>  if (x instanceof Bar) { }
>
> that if Bar and Foo are distinct concrete classes, and one is not a
> subclass of the other, the second if can be replaced with an else if.
> Method resolution order is not consistent between MI languages,
> either, and may be further complicated by how they are to interact
> with generic functions.
>
> 6. Languages may also make type-related requirements that need to be
> checked somehow; such as some languages not permitting the initialiser
> to be called more than once, final methods, &c.
>
> 7. In what way to generic functions and methods interact?  How about
> classes and typeclasses?
>
> 8. What happens when a language that is strict about arity interacts
> with a language that does not?  Does calling a javascript function
> from python with the wrong number of arguments raise a TypeError, or
> silently pass undefined?  Does the inverse pass None?
>
> 9. Do the error handling and exception mechanisms intersect - say,
> what Java exception does my Elisp args-out-of-range become?  Exception
> hierarchies can be *very* different across object spaces.
>
> 10. How do methods overloaded on static type look from a dynamic
> language? Oh, and what about virtual/non-virtual methods and extension
> methods as on ecma CLI?
>
> I'm not saying that these questions don't have answers, but I am
> saying that in order for the abstraction to compose well, there needs
> to be a way to express the solutions to problems like these, and
> preferably, a way to drop down to a lower level and specify a
> different behaviour.  This is by no means a complete list (although as
> you can probably see, it's a problem I've meditated on regularly),
> just a taste.
>
> However: Guile was *designed* to be a multi-language VM, and has a far
> nicer object model and MOP than something like the JVM or CLI.  If
> scheme were the common denominator, it might be so much nicer than
> what we've seen before that the jarring that comes from crossing
> several object space boundaries is not as severe.  I just mean to be
> careful about whether you are suggesting moving to a single VM,
> implementing an automated FFI tool with several backends, or solving
> this problem generally, because they are each projects of very
> different scale.
>
> --
> William Leslie
>
>