Guile emacs thread (again)

Guile emacs thread (again)

[Christopher Allan Webber]

Hello!  It's been a while since the topic has come up on this list, but
many of us are interested in it, and maybe some developers don't know,
and I hadn't seen any conversations on the list despite recent progress.

Anyway, it seems that BT Templeton's emacs on guile project this summer
has gone fairly well, and it looks like almost everything runs.  See:
  http://www.emacswiki.org/emacs/GuileEmacs

I remember reading Andy Wingo's email about this a few years ago,
"guile and emacs and elisp, oh my!":
  https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00665.html

I found it very inspiring.  It seems those things are fairly close.

So this email is partly a:
 - What now?  What's the chance of work towards guilemacs moving over to
   an official emacs git branch, and that port happening, anytime soon?
 - Is anyone running it?  How's it going for you?

(I'm mid-compile of the guile wip branch right now...!)
 - Chris

Re: Guile emacs thread (again)

[Eli Zaretskii]

> From: Christopher Allan Webber <cwebber@dustycloud.org>
> Date: Thu, 11 Sep 2014 11:29:09 -0500
> 
> Anyway, it seems that BT Templeton's emacs on guile project this summer
> has gone fairly well, and it looks like almost everything runs.  See:
>   http://www.emacswiki.org/emacs/GuileEmacs
> 
> I remember reading Andy Wingo's email about this a few years ago,
> "guile and emacs and elisp, oh my!":
>   https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00665.html
> 
> I found it very inspiring.  It seems those things are fairly close.
> 
> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?

Well, the "User-visible issues" there sound pretty scary to me.
Evidently, even building it, and even on GNU/Linux, is not without
problems ("it compiles and starts up if you struggle a little").

Please also note the "best just test on GNU/Linux for now" comment,
and a similar TODO item "find beta testers for non-GNU platforms".
This matches my experience with Guile -- it needs some serious work to
be reasonably portable to platforms other than GNU.  Right now, too
many important features will be simply disabled if the configure
script finds they are not provided by the standard libraries.  Emacs
is eons ahead of that.

And it doesn't help that one needs a hacked version of Guile for this
(i.e. have 2 different Guile versions installed on the same machine).

So IMO, Someone® should invest a non-trivial amount of work to at
least resolve the main issues mentioned on the TODO page.  Otherwise,
I doubt that someone will consider using such an Emacs for serious
work, and without that it will never get past the alpha-testing stage,
whether it is an official branch of the Emacs repo or not.

Also, I think there are some design issues that need to be discussed,
rather than decided by a single individual, because the considerations
for how to unify Guile with Emacs, i.e. which Emacs objects and
methods should be based on Guile, are entirely non-trivial.  One
example is non-ASCII text and string representation -- AFAICS
GuileEmacs wants to convert them back and forth from/to Scheme
strings, but the internal representation of text in Emacs is not 100%
UTF-8, it has a couple of extensions, and it's not clear how best to
support that.

Another potential issue is (or could be) the use of mmap for buffer
memory -- I guess the current build simply sidesteps that because mmap
isn't used on GNU/Linux, but otherwise how would this play with libgc
and threads?

I'm sure these 2 are just the tip of an iceberg.

> (I'm mid-compile of the guile wip branch right now...!)

Did you succeed?

Re: Guile emacs thread (again)

[Grim Schjetne]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Hello!  It's been a while since the topic has come up on this list, but
> many of us are interested in it, and maybe some developers don't know,
> and I hadn't seen any conversations on the list despite recent progress.
>
> Anyway, it seems that BT Templeton's emacs on guile project this summer
> has gone fairly well, and it looks like almost everything runs.  See:
>   http://www.emacswiki.org/emacs/GuileEmacs
>
> I remember reading Andy Wingo's email about this a few years ago,
> "guile and emacs and elisp, oh my!":
>   https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00665.html
>
> I found it very inspiring.  It seems those things are fairly close.
>
> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?
>  - Is anyone running it?  How's it going for you?

As someone who spends almost every waking hour in Emacs, both for fun
and for profit, I'm very interested in this project. Unfortunately I
haven't had time to look at the latest progress, I'll hopefully be able
to do this soon.

Re: Guile emacs thread (again)

[Taylan Ulrich Bayirli/Kammer]

Foreword: I'm who enthusiastically edited all the EmacsWiki pages to
report on the latest status, but I didn't bother the ML or so
precisely because the current state is still far from being ready for
normal usage.  Sorry if I caused unnecessary hype.

Now to clear some things up:

Eli Zaretskii writes:

> Well, the "User-visible issues" there sound pretty scary to me.
> Evidently, even building it, and even on GNU/Linux, is not without
> problems ("it compiles and starts up if you struggle a little").

Note that replacing the Elisp engine with libguile has been done just
now; this is the first publicly available version of it, so it
"compiles and starts up if you struggle a little" indeed. :) And then
it's slow (extremely so in some specific things) and bug-ridden from a
user perspective.  You could call it pre-alpha I suppose.

(The Guile-Emacs from last year and such had libguile embedded in
Emacs, and using some of its data types in Emacs's Elisp engine, but
was not using it to execute Elisp.)

> And it doesn't help that one needs a hacked version of Guile for
> this (i.e. have 2 different Guile versions installed on the same
> machine).

Of course the changes will flow into Guile's master or stable branch
when ready.  I guess you meant this wrt. making this a branch of
upstream Emacs right now, in which case I agree; that would be weird
if it doesn't work with the latest Guile release.

> So IMO, Someone® should invest a non-trivial amount of work to at
> least resolve the main issues mentioned on the TODO page.
> Otherwise, I doubt that someone will consider using such an Emacs
> for serious work, and without that it will never get past the
> alpha-testing stage, whether it is an official branch of the Emacs
> repo or not.

It's indeed scary that there is exactly one developer working on
Guile-Emacs since years, and they're only active during GSoC periods
(though "doing god's work" in those periods).  I wish to Some Day™
start working on it myself, but for now [reasons], and I'm afraid I
can't promise anything for the future either because [uncomfortable
reasons].  I hope some good people will join the effort, since it has
come so far.

> [...] One example is non-ASCII text and string representation --
> AFAICS GuileEmacs wants to convert them back and forth from/to
> Scheme strings, but the internal representation of text in Emacs is
> not 100% UTF-8, it has a couple of extensions, and it's not clear
> how best to support that.

If I understood BT Templeton (bipt) right on IRC, converting strings
back and forth is only a temporary solution.  I hope we find a way to
merge the two string types.

Taylan

Re: Guile emacs thread (again)

[Eli Zaretskii]

> Date: Tue, 16 Sep 2014 20:07:21 +0200
> From: Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com>
> Cc: eliz@gnu.org
> 
> > Well, the "User-visible issues" there sound pretty scary to me.
> > Evidently, even building it, and even on GNU/Linux, is not without
> > problems ("it compiles and starts up if you struggle a little").
> 
> Note that replacing the Elisp engine with libguile has been done just
> now; this is the first publicly available version of it

That's clear (I did a "git log" ;-).  My comment was regarding the
possibility to use that seriously in its present state.

> If I understood BT Templeton (bipt) right on IRC, converting strings
> back and forth is only a temporary solution.  I hope we find a way to
> merge the two string types.

That will need a lot of talking first.  There are a lot of subtle
differences, and the fact that Guile based its non-ASCII support on
libunistring doesn't help.

Re: Guile emacs thread (again)

[Lluís]

Christopher Allan Webber writes:
[...]
> I found it very inspiring.  It seems those things are fairly close.

> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?
>  - Is anyone running it?  How's it going for you?

I just read the thread, and many different and entwined points have been raised,
which I think make the discussion harder to follow. Thus I would like to
summarize the issues I've read regarding Guile (feel free to reply with any
points you think I've missed or misinterpreted):

* Manpower / responsiveness

  There's not much to say here. Whatever layers Emacs decides to use (if any),
  there should be an appropriate response delay to handle issues.

* Direction / design

  There could be design decisions conflicting between Emacs' and Guile's
  desires. Such issues can only be resolved through mutual and good-hearted
  understanding.

* Multi-language support

  As someone said, using Guile does not necessarily mean exposing all these
  languages in Emacs (i.e., just use Guile's lower-level compiler/VM layers).

* Development model / organization

  It has been argued that Guile's development model does not give enough
  stability assurances. I'm sure that's something that can be easily fixed if
  Guile's developers slightly change their workflow to maintain a more stable
  branch that Emacs (and other projects) can use.

* It's a GNU project

  If that really counts for any purpose.


Thus I would say that the only critical point here is the first one
(manpower/responsiveness). To some extent, the second point could be a problem
too (direction/design), but targeting Emacs to the lower-level layers of Guile
could limit the impact of this factor.


Thanks,
  Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Guile emacs thread (again)

[Lally Singh]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

On Wed, Sep 17, 2014 at 3:29 PM, Lluís <xscript@gmx.net> wrote:

> [..]
> Thus I would say that the only critical point here is the first one
> (manpower/responsiveness). To some extent, the second point could be a
> problem
> too (direction/design), but targeting Emacs to the lower-level layers of
> Guile
> could limit the impact of this factor.
>

We should also be mindful of the effects of what such a large project as
emacs would have suddenly coming into a project with a substantially
smaller community.  The emacs'ers could end up taking over, even if nobody
intended for that to happen.

[-- Attachment #2: Type: text/html, Size: 986 bytes --]

Re: Guile emacs thread (again)

[Robin Templeton]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Hello!  It's been a while since the topic has come up on this list, but
> many of us are interested in it, and maybe some developers don't know,
> and I hadn't seen any conversations on the list despite recent progress.
>
> Anyway, it seems that BT Templeton's emacs on guile project this summer
> has gone fairly well, and it looks like almost everything runs.  See:
>   http://www.emacswiki.org/emacs/GuileEmacs
>
> I remember reading Andy Wingo's email about this a few years ago,
> "guile and emacs and elisp, oh my!":
>   https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00665.html
>
> I found it very inspiring.  It seems those things are fairly close.
>
> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?
>  - Is anyone running it?  How's it going for you?
>
> (I'm mid-compile of the guile wip branch right now...!)
>  - Chris

First, I'd like to clarify some things about Guile-Emacs for those just
tuning in. The goal of the project is to create a better implementation
of Elisp than Emacs provides -- providing full compatibility with
existing Elisp packages -- and to use that as Emacs's Lisp
implementation. That implementation is Guile-Elisp, a compiler targeting
the Guile VM, and Guile-Emacs is the variant of Emacs based on this
compiler.

What will this mean for Emacs? Better performance and new language
features. Guile provides a fast virtual machine and a sophisticated
compiler infrastructure, and the Guile VM outperforms the Elisp bytecode
interpreter on a variety of benchmarks. Guile natively provides many
features from Scheme and Common Lisp not present in Elisp, including a
full numeric tower, structure types, CLOS-based object orientation, a
foreign function interface, delimited continuations, a module system,
hygienic macros, multiple values, and threads.

It will not mean that Elisp programs will need to be rewritten in
Scheme, or even that Emacs will necessarily support extensions written
in Scheme (much less JavaScript!). Interaction with other languages is a
possibility, but the exact relationship between Elisp and other
languages will be up to the Emacs maintainers and community. Elisp is
here to stay; Guile-Emacs will not remove Elisp but will instead enable
and accelerate its evolution.

The fifth Guile-Emacs Summer of Code project has concluded successfully.
Thanks to Andy Wingo for supervising my work, and to Google for funding
the project. This summer is when everything really came together. The
project previously had two independent components developed in parallel:
Guile-Elisp and Guile-Emacs. Guile-Emacs used libguile for garbage
collection and object representation, but retained the original
interpreter and bytecode interpreter and did not, initially, use the
compiler at all. The two components have been successfully united. Emacs
now compiles all Elisp code using Guile-Elisp, and all programs are
executed on the Guile VM. The old interpreter and bytecode interpreter
have been excised entirely. Guile-Emacs loads, compiles and executes the
programs included in Emacs by default, plus Gnus, Org-Mode, and more.

Additionally, Guile-Emacs was rebased onto a recent development version
of Emacs. Guile-Elisp was ported to guile master, which will become
Guile 2.2 and includes a new and faster virtual machine and upgraded
compiler tower. Other additions include compiler macros a la Common Lisp
used for inlining calls to Guile primitives, a reasonably complete FFI,
and Scheme Interaction Mode. The build system has been updated so that
Guile-Emacs can build itself cleanly from a fresh git checkout, with no
assistance from an existing Emacs installation and no undocumented
command line jiggery-pokery. Now the build procedure is "./configure &&
make".

Furthermore, I made a number of modifications to low-level support code
for Guile compatibility. The TL;DR on this is that the language changes
introduced in Guile-Elisp shouldn't matter much for normal programs, but
do require adjustment to low-level code, including debuggers,
macroexpanders and code-walkers. I'll write up a summary of the
differences separately, but the intent is that Guile-Emacs should be no
more disruptive than the addition of lexical binding from an Elisp
programmer's perspective.

For more information on Guile-Emacs, please see the Wiki page at
<http://www.emacswiki.org/emacs/GuileEmacs>, and keep an eye on
<http://www.guile-emacs.org/> for future updates.

Happy hacking,
Robin
-- 
Inteligenta persono lernas la lingvon Esperanton rapide kaj facile.
Esperanto estas moderna, kultura lingvo por la mondo. Simpla, fleksebla,
belsona, Esperanto estas la praktika solvo de la problemo de universala
interkompreno. Lernu la interlingvon Esperanton!

Re: Guile emacs thread (again)

[Christopher Allan Webber]

Robin Templeton writes:

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>> Hello!  It's been a while since the topic has come up on this list, but
>> many of us are interested in it, and maybe some developers don't know,
>> and I hadn't seen any conversations on the list despite recent progress.
>>
>> Anyway, it seems that BT Templeton's emacs on guile project this summer
>> has gone fairly well, and it looks like almost everything runs.  See:
>>   http://www.emacswiki.org/emacs/GuileEmacs
>>
>> I remember reading Andy Wingo's email about this a few years ago,
>> "guile and emacs and elisp, oh my!":
>>   https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00665.html
>>
>> I found it very inspiring.  It seems those things are fairly close.
>>
>> So this email is partly a:
>>  - What now?  What's the chance of work towards guilemacs moving over to
>>    an official emacs git branch, and that port happening, anytime soon?
>>  - Is anyone running it?  How's it going for you?
>>
>> (I'm mid-compile of the guile wip branch right now...!)
>>  - Chris
>
> First, I'd like to clarify some things about Guile-Emacs for those just
> tuning in. The goal of the project is to create a better implementation
> of Elisp than Emacs provides -- providing full compatibility with
> existing Elisp packages -- and to use that as Emacs's Lisp
> implementation. That implementation is Guile-Elisp, a compiler targeting
> the Guile VM, and Guile-Emacs is the variant of Emacs based on this
> compiler.
>
> What will this mean for Emacs? Better performance and new language
> features. Guile provides a fast virtual machine and a sophisticated
> compiler infrastructure, and the Guile VM outperforms the Elisp bytecode
> interpreter on a variety of benchmarks. Guile natively provides many
> features from Scheme and Common Lisp not present in Elisp, including a
> full numeric tower, structure types, CLOS-based object orientation, a
> foreign function interface, delimited continuations, a module system,
> hygienic macros, multiple values, and threads.
>
> It will not mean that Elisp programs will need to be rewritten in
> Scheme, or even that Emacs will necessarily support extensions written
> in Scheme (much less JavaScript!). Interaction with other languages is a
> possibility, but the exact relationship between Elisp and other
> languages will be up to the Emacs maintainers and community. Elisp is
> here to stay; Guile-Emacs will not remove Elisp but will instead enable
> and accelerate its evolution.
>
> The fifth Guile-Emacs Summer of Code project has concluded successfully.
> Thanks to Andy Wingo for supervising my work, and to Google for funding
> the project. This summer is when everything really came together. The
> project previously had two independent components developed in parallel:
> Guile-Elisp and Guile-Emacs. Guile-Emacs used libguile for garbage
> collection and object representation, but retained the original
> interpreter and bytecode interpreter and did not, initially, use the
> compiler at all. The two components have been successfully united. Emacs
> now compiles all Elisp code using Guile-Elisp, and all programs are
> executed on the Guile VM. The old interpreter and bytecode interpreter
> have been excised entirely. Guile-Emacs loads, compiles and executes the
> programs included in Emacs by default, plus Gnus, Org-Mode, and more.
>
> Additionally, Guile-Emacs was rebased onto a recent development version
> of Emacs. Guile-Elisp was ported to guile master, which will become
> Guile 2.2 and includes a new and faster virtual machine and upgraded
> compiler tower. Other additions include compiler macros a la Common Lisp
> used for inlining calls to Guile primitives, a reasonably complete FFI,
> and Scheme Interaction Mode. The build system has been updated so that
> Guile-Emacs can build itself cleanly from a fresh git checkout, with no
> assistance from an existing Emacs installation and no undocumented
> command line jiggery-pokery. Now the build procedure is "./configure &&
> make".
>
> Furthermore, I made a number of modifications to low-level support code
> for Guile compatibility. The TL;DR on this is that the language changes
> introduced in Guile-Elisp shouldn't matter much for normal programs, but
> do require adjustment to low-level code, including debuggers,
> macroexpanders and code-walkers. I'll write up a summary of the
> differences separately, but the intent is that Guile-Emacs should be no
> more disruptive than the addition of lexical binding from an Elisp
> programmer's perspective.
>
> For more information on Guile-Emacs, please see the Wiki page at
> <http://www.emacswiki.org/emacs/GuileEmacs>, and keep an eye on
> <http://www.guile-emacs.org/> for future updates.
>
> Happy hacking,
> Robin

Hey Robin, thanks for this really great email.  I was excited for
guilemacs before, but I'm even more excited now.  :)

Re: Guile emacs thread (again)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    It will not mean that Elisp programs will need to be rewritten in
    Scheme, or even that Emacs will necessarily support extensions written
    in Scheme

Supporting Emacs extensions written in Scheme is part of the goal I
had in mind when I proposed this.  That's the tangible benefit that we
would get from Scheme support.  If all Scheme does is serve as a platform
for Emacs Lisp, it is no real advance.

It is ok if it is necessary to write these extensions in a slightly
unusual style to get them to interoperate properly with Emacs Lisp
programs.  For instance, maybe it would require using special
functions to test for Elisp nil.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Guile emacs thread (again)

[Eli Zaretskii]

> Date: Sat, 20 Sep 2014 09:20:16 -0400
> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Supporting Emacs extensions written in Scheme is part of the goal I
> had in mind when I proposed this.  That's the tangible benefit that we
> would get from Scheme support.  If all Scheme does is serve as a platform
> for Emacs Lisp, it is no real advance.

Then perhaps the goals of the Guile-Emacs project should be shifted,
such that the first goal is to have Emacs that can use Scheme
extensions, and push the goal of having Emacs Lisp based on Scheme
farther into the future.  Doing so would make sense to me, since this
is what other GNU projects do (Make, GDB), and it postpones the need
for solving a whole lot of non-trivial problems, some of which were
discussed here.  It would also take the sting out of the fears
expressed by some that Guile development team might not be up to the
task of supporting an infrastructure which Emacs cannot do without.
If Guile is just another extension mechanism, losing it will not be a
fatal blow.

Re: Guile emacs thread (again)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > Supporting Emacs extensions written in Scheme is part of the goal I
    > had in mind when I proposed this.  That's the tangible benefit that we
    > would get from Scheme support.  If all Scheme does is serve as a platform
    > for Emacs Lisp, it is no real advance.

    Then perhaps the goals of the Guile-Emacs project should be shifted,
    such that the first goal is to have Emacs that can use Scheme
    extensions, and push the goal of having Emacs Lisp based on Scheme
    farther into the future.

The extensions written in Scheme need to interoperate with all the
extensions written in Emacs Lisp.  Whether the roadmap for getting
there ought to be changed, I am not sure.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Guile emacs thread (again)

[Stefan Monnier]

> Then perhaps the goals of the Guile-Emacs project should be shifted,
> such that the first goal is to have Emacs that can use Scheme
> extensions,

On the contrary, this should be one of the last goals.
The main goal should be to maximize the amount of existing code that
Emacs can benefit from.

IOW for Emacs-specific libraries, this requires bug-compatible Elisp
support, and for the rest it requires an FFI (and if some of the rest
can be supported more directly by sharing the underlying VM, that's even
better, tho I'd simply consider it as a "particular good kind of FFI").

Being able to access existing Scheme libraries is great.  But being able
to write Emacs extensions in Scheme is definitely not a priority and
might even be harmful to Emacs by making splitting the community and
wasting time on improving interoperability between the languages.


        Stefan

Re: Guile emacs thread (again)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rms@gnu.org,  robin@terpri.org,  emacs-devel@gnu.org
> Date: Sun, 21 Sep 2014 12:16:07 -0400
> 
> > Then perhaps the goals of the Guile-Emacs project should be shifted,
> > such that the first goal is to have Emacs that can use Scheme
> > extensions,
> 
> On the contrary, this should be one of the last goals.
> The main goal should be to maximize the amount of existing code that
> Emacs can benefit from.

You mean, what Guile already has, like FFI?  You will have that by
using Scheme extensions.  How's that a problem?

> Being able to access existing Scheme libraries is great.  But being able
> to write Emacs extensions in Scheme is definitely not a priority and
> might even be harmful to Emacs by making splitting the community and
> wasting time on improving interoperability between the languages.

OTOH, doing what I suggest is much easier, and will allow both sides
to learn about the other one without the kind of pressure which comes
with replacing the existing Lisp interpreter with a new one.

Re: Guile emacs thread (again)

[Stefan Monnier]

> OTOH, doing what I suggest is much easier, and will allow both sides
> to learn about the other one without the kind of pressure which comes
> with replacing the existing Lisp interpreter with a new one.

That was the state of Guile-Emacs last year, pretty much (i.e. don't use
the Guile-VM byte-code engine, but keep using our own interpreters
instead, while sharing the heap with Guile, i.e. using Guile's GC).

That's OK as well.

The purpose of using the Guile-VM is that it should be able to give us
better performance.


        Stefan