Guile emacs thread (again)

Guile emacs thread (again)

[Christopher Allan Webber]

Hello!  It's been a while since the topic has come up on this list, but
many of us are interested in it, and maybe some developers don't know,
and I hadn't seen any conversations on the list despite recent progress.

Anyway, it seems that BT Templeton's emacs on guile project this summer
has gone fairly well, and it looks like almost everything runs.  See:
  http://www.emacswiki.org/emacs/GuileEmacs

I remember reading Andy Wingo's email about this a few years ago,
"guile and emacs and elisp, oh my!":
  https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00665.html

I found it very inspiring.  It seems those things are fairly close.

So this email is partly a:
 - What now?  What's the chance of work towards guilemacs moving over to
   an official emacs git branch, and that port happening, anytime soon?
 - Is anyone running it?  How's it going for you?

(I'm mid-compile of the guile wip branch right now...!)
 - Chris

Re: Guile emacs thread (again)

[Eli Zaretskii]

> From: Christopher Allan Webber <cwebber@dustycloud.org>
> Date: Thu, 11 Sep 2014 11:29:09 -0500
> 
> Anyway, it seems that BT Templeton's emacs on guile project this summer
> has gone fairly well, and it looks like almost everything runs.  See:
>   http://www.emacswiki.org/emacs/GuileEmacs
> 
> I remember reading Andy Wingo's email about this a few years ago,
> "guile and emacs and elisp, oh my!":
>   https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00665.html
> 
> I found it very inspiring.  It seems those things are fairly close.
> 
> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?

Well, the "User-visible issues" there sound pretty scary to me.
Evidently, even building it, and even on GNU/Linux, is not without
problems ("it compiles and starts up if you struggle a little").

Please also note the "best just test on GNU/Linux for now" comment,
and a similar TODO item "find beta testers for non-GNU platforms".
This matches my experience with Guile -- it needs some serious work to
be reasonably portable to platforms other than GNU.  Right now, too
many important features will be simply disabled if the configure
script finds they are not provided by the standard libraries.  Emacs
is eons ahead of that.

And it doesn't help that one needs a hacked version of Guile for this
(i.e. have 2 different Guile versions installed on the same machine).

So IMO, Someone® should invest a non-trivial amount of work to at
least resolve the main issues mentioned on the TODO page.  Otherwise,
I doubt that someone will consider using such an Emacs for serious
work, and without that it will never get past the alpha-testing stage,
whether it is an official branch of the Emacs repo or not.

Also, I think there are some design issues that need to be discussed,
rather than decided by a single individual, because the considerations
for how to unify Guile with Emacs, i.e. which Emacs objects and
methods should be based on Guile, are entirely non-trivial.  One
example is non-ASCII text and string representation -- AFAICS
GuileEmacs wants to convert them back and forth from/to Scheme
strings, but the internal representation of text in Emacs is not 100%
UTF-8, it has a couple of extensions, and it's not clear how best to
support that.

Another potential issue is (or could be) the use of mmap for buffer
memory -- I guess the current build simply sidesteps that because mmap
isn't used on GNU/Linux, but otherwise how would this play with libgc
and threads?

I'm sure these 2 are just the tip of an iceberg.

> (I'm mid-compile of the guile wip branch right now...!)

Did you succeed?

Re: Guile emacs thread (again)

[Grim Schjetne]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Hello!  It's been a while since the topic has come up on this list, but
> many of us are interested in it, and maybe some developers don't know,
> and I hadn't seen any conversations on the list despite recent progress.
>
> Anyway, it seems that BT Templeton's emacs on guile project this summer
> has gone fairly well, and it looks like almost everything runs.  See:
>   http://www.emacswiki.org/emacs/GuileEmacs
>
> I remember reading Andy Wingo's email about this a few years ago,
> "guile and emacs and elisp, oh my!":
>   https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00665.html
>
> I found it very inspiring.  It seems those things are fairly close.
>
> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?
>  - Is anyone running it?  How's it going for you?

As someone who spends almost every waking hour in Emacs, both for fun
and for profit, I'm very interested in this project. Unfortunately I
haven't had time to look at the latest progress, I'll hopefully be able
to do this soon.

Emacs Lisp's future (was: Guile emacs thread (again))

[Stefan Monnier]

> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?
>  - Is anyone running it?  How's it going for you?

Good questions.  I've had the opportunity to think a bit more about
Emacs Lisp and its possible evolution and I'm still not sure what to
think about it.

I see a few different options for Emacs Lisp.

First, of course we can keep on evolving Elisp on its own.  This has
worked OK for the last 30 years, so it's not such a terrible choice.
The main problems I see with that:
- Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
  noticed more often.
- Lack of some features, most notably FFI and concurrency.
- Lack of manpower.

This last point is for me the strongest motivation to try and move to
some other system, where we could use other people's work.

One such option is Guile-Emacs.  This presumably would give us a faster
implementation (at least in theory, their bytecode is significantly
more efficient), would give us an FFI, and would give us more manpower
since we'd be benefiting from the work done on Guile.

Note that while Guile does come with support for threading, it doesn't
immediately let us use concurrency in Guile-Emacs, because of all the
issues of synchronizing access to shared data, with all the existing
Emacs code (both C and Elisp) assuming that this problem doesn't exist.
IOW, language support for concurrency is just a first step on the way to
letting Emacs Lisp use concurrency.

Another detail that needs to be spelled out is the difference between
the language and its implementation.  Guile-Emacs provides 2 languages:
Emacs Lips and Scheme (well, it also provides a few more, but that's
not important).  Many people are thinking "cool, so I'll be able to
write extensions in Scheme", but I'm not sure defining Emacs as "this
editor that comes with N extensions languages" is a good idea.

One of the main reasons for Emacs's enduring success is its large set of
third party packages so obviously we can't drop support for Elisp any
time soon.  And as much as I like Scheme, I'm very much unconvinced that
it's really so much better that it's worth converting packages from
Elisp to Scheme.

So if we go for Guile-Emacs, we'll be stuck with Guile, i.e. we'd
have (old and new) packages that use Elisp, new packages that use
Scheme, maybe yet other new packages that use, say, Javascript (or some
other language support by Guile).  That would make the work of Emacs
(and GNU ELPA) maintenance harder.

And of course, if Guile's own manpower dries up, Emacs would be forced
to keep supporting Guile, which is more work than supporting just Elisp.

So, I think that ideally, we'd want to stick to Elisp, or some
evolution thereof.  Sadly, I don't see how to evolve Elisp into Scheme:
they are closely related languages, but the differences are large enough
that it seems hard to reconcile them.

The only standard language into which Elisp can evolve, AFAICT, is
Common Lisp.  [ Now some readers get disappointed, while some others
become excited.  ]  There are some incompatibilities between the two
languages, but I can imagine working them out over the years, or even
living with them without too much trouble, such that we could use
Common-Lisp libraries in Emacs.

Of course, that's for the language side, but on the implementation side,
I don't really know what Common-Lisp implementation we could re-use
(both GNU implementations are dormant, so there's no manpower for us
tap into).  Still: there are many Common-Lisp implementations out there,
so there's probably one that could work for us.


        Stefan

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lennart Borgman]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

On Tue, Sep 16, 2014 at 5:50 PM, Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> The main problems I see with that:
> - Elisp is slow and as CPUs aren't getting faster, its slowness makes
> itself
>   noticed more often.
> - Lack of some features, most notably FFI and concurrency.
> - Lack of manpower.
>

Perhaps also the lack of possibility to enhance Emacs with code written in
other languages? I think for example that Javascript will be something most
future programmers will know. Could Guile make it easier to enhance Emacs
with Javascript (as an alternative to Elisp)?

[-- Attachment #2: Type: text/html, Size: 1152 bytes --]

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Jorgen Schaefer]

On Tue, 16 Sep 2014 18:03:17 +0200
Lennart Borgman <lennart.borgman@gmail.com> wrote:

> On Tue, Sep 16, 2014 at 5:50 PM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
> 
> > The main problems I see with that:
> > - Elisp is slow and as CPUs aren't getting faster, its slowness
> > makes itself
> >   noticed more often.
> > - Lack of some features, most notably FFI and concurrency.
> > - Lack of manpower.
> >
> 
> Perhaps also the lack of possibility to enhance Emacs with code
> written in other languages? I think for example that Javascript will
> be something most future programmers will know. Could Guile make it
> easier to enhance Emacs with Javascript (as an alternative to Elisp)?

I think the (often-cited, not just here) idea of supporting multiple
languages is a red herring, mostly. Every extension language supported
adds some burden on those who want to understand what their editor
does, not just use pre-packaged code. One of the great things about
Emacs is that, once I know ELisp, I have a good chance of understanding
and modifying any extension I see. And learning Emacs Lisp is not
exactly hard.

But we do not have to be all theoretical here. There is an editor which
supports a dozen extension languages. The paradoxical thing to notice
when you look at vim plugins is that most of them are written in VimL,
including rather complex ones like NERD-tree and fugitive. I'd argue
that VimL is a tiny bit harder to learn and use than ELisp. There are
various reasons for why most plugins are written in it, but I do think
that this is a pretty good indicator that the lack of "common" languages
for extension is not exactly high on the list of problems for an editor.

There are plenty of things in ELisp itself that I'd put much higher on
that list.

- Lack of a common structured datatype. While there's cl-defstruct, the
  support is a bit limited (C-h f does not work well with it), and a
  lot of code simply does not use it, making it seem a bit like a
  red-haired stepchild instead of a core recommended language feature.
  Alists and plists are usually used where modern languages would use
  structured datatypes, or even some hack with cons cells or lists and
  indexed access.
- Hashes are one of those data types that are used all over the place
  in other languages, but you see them rarely in Emacs Lisp, again often
  losing out to alists and plists. This might be related to the
  standard library functions being a bit baroque. (There's some
  third-party hash library somewhere.)
- Speaking of third-party libraries, s.el, dash.el and f.el provide
  things that really ought to be in core Emacs.
- The regex engine is annoying to use. Providing some interface to PCRE
  would be a great step forward, and does not even have to be
  backwards-incompatible.
- There are tons of warts in Emacs Lisp. nth vs. elt for example,
  with their exciting incompatible calling conventions.

One thing I think would benefit Emacs Lisp as a language a lot would be
a standard library cleanup. An effort to go through the libraries that
come with Emacs, separate them into "standard library" and "extensions
that come with Emacs", and then go through the "standard library",
provide sane names when necessary (like setcar is provided for rplaca)
and deprecating the old ones, or simply deprecate all but one version of
functions with overlapping use (nth or elt, pick one). Finally, add
standard libraries for common functionality that is currently lacking
(see above).

The next step would be going through the "extensions that come with
Emacs" and make sure they all use namespace prefixes for anything but
very specific commands meant for users to use with M-x. Only standard
library functions are allowed to be namespace-free.

These things would make Emacs Lisp a lot easier to use and also easier
to learn for new users.

This is all doable, but it needs manpower (#3 on Stefans list). Which
is manpower that would not be doing other cool stuff on Emacs.

Regards,
Jorgen

Re: Emacs Lisp's future

[Lars Brinkhoff]

Jorgen Schaefer <forcer@forcix.cx> writes:
> - Lack of a common structured datatype.

That was what I was saying a year ago.  My offer to work on this still
stands.

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lally Singh]

[-- Attachment #1: Type: text/plain, Size: 2045 bytes --]

On Wed, Sep 17, 2014 at 2:24 PM, Jorgen Schaefer <forcer@forcix.cx> wrote:

> On Tue, 16 Sep 2014 18:03:17 +0200
> Lennart Borgman <lennart.borgman@gmail.com> wrote:
> > Perhaps also the lack of possibility to enhance Emacs with code
> > written in other languages? I think for example that Javascript will
> > be something most future programmers will know. Could Guile make it
> > easier to enhance Emacs with Javascript (as an alternative to Elisp)?
>
> I think the (often-cited, not just here) idea of supporting multiple
> languages is a red herring, mostly. Every extension language supported
> adds some burden on those who want to understand what their editor
> does, not just use pre-packaged code. One of the great things about
> Emacs is that, once I know ELisp, I have a good chance of understanding
> and modifying any extension I see. And learning Emacs Lisp is not
> exactly hard.
>

I think a policy of "if written for emacs, do it in elisp" is a good one,
but let's acknowledge the advantage of easy linking/calling into other code
bases that may come with having a multi-language-compatible runtime system.
 I'm sure we've all seen some systems that we'd love to invoke directly
from elisp.


> [snipping some very good points]

One thing I think would benefit Emacs Lisp as a language a lot would be
> a standard library cleanup. An effort to go through the libraries that
> come with Emacs, separate them into "standard library" and "extensions
> that come with Emacs", and then go through the "standard library",
> provide sane names when necessary (like setcar is provided for rplaca)
> and deprecating the old ones, or simply deprecate all but one version of
> functions with overlapping use (nth or elt, pick one). Finally, add
> standard libraries for common functionality that is currently lacking
> (see above).
>

I completely agree that there's plenty of work needed there, but:
 - If staying with elisp, this is a separate discussion
 - If not staying with elisp, these problems can be addressed during
conversion.

[-- Attachment #2: Type: text/html, Size: 2850 bytes --]

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Alexis]

Jorgen Schaefer writes:

> I think the (often-cited, not just here) idea of supporting multiple
> languages is a red herring, mostly. Every extension language supported
> adds some burden on those who want to understand what their editor
> does, not just use pre-packaged code. One of the great things about
> Emacs is that, once I know ELisp, I have a good chance of
> understanding and modifying any extension I see.

+1. It seems to me that much discussion around this issue focuses on
being able to use one's favourite language to extend Emacs, without
considering what it might be like to:

* start using an extension that seems like it might significantly assist
  one's productivity;
* find a show-stopping bug in it; and
* discover that the extension is written in a programming language one
  loves to loathe.

Also, to an (admittedly very limited) extent, it's /already/ possible to
use a number of languages other than ELisp in Emacs, for at least some
things, via org-babel:

http://orgmode.org/manual/Evaluating-code-blocks.html#Evaluating-code-blocks

A list of languages currently supported by org-babel can be found at:

http://orgmode.org/manual/Languages.html#Languages

i would be interested to know what experiences people might have had in
using org-babel as part of an Emacs extension ....

> Speaking of third-party libraries, s.el, dash.el and f.el provide
> things that really ought to be in core Emacs.

Agreed. i've found it quite surprising that simple functions like
-flatten and s-repeat aren't available in ELisp by default.

> The regex engine is annoying to use. Providing some interface to PCRE
> would be a great step forward, and does not even have to be
> backwards-incompatible.

i like this suggestion. i'm rather comfortable with Perl5 REs, and can
find myself frustrated trying to create REs in ELisp. Having said that,
the issue is not usually the syntax of ELisp REs per se (e.g. needing to
escape things like capturing parentheses or the alternatives pipe); it's
needing to escape various things /further/ because REs can only be
specified in the form of a standard ELisp string. On several occasions
i've ended up using a combination of pcre-to-elisp and re-builder to try
to work out if the problem is too few backslashes, too many backslashes,
or both.


Alexis.

Re: Emacs Lisp's future

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 676 bytes --]

On 09/17/2014 07:07 PM, Alexis wrote:
> i like this suggestion. i'm rather comfortable with Perl5 REs, and can
> find myself frustrated trying to create REs in ELisp. Having said that,
> the issue is not usually the syntax of ELisp REs per se (e.g. needing to
> escape things like capturing parentheses or the alternatives pipe); it's
> needing to escape various things /further/ because REs can only be
> specified in the form of a standard ELisp string. On several occasions
> i've ended up using a combination of pcre-to-elisp and re-builder to try
> to work out if the problem is too few backslashes, too many backslashes,
> or both.

That's why we have rx.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Regular expression creation [was: Re: Emacs Lisp's future]

[Alexis]

Daniel Colascione writes:

> On 09/17/2014 07:07 PM, Alexis wrote:
>> i like this suggestion. i'm rather comfortable with Perl5 REs, and can
>> find myself frustrated trying to create REs in ELisp. Having said that,
>> the issue is not usually the syntax of ELisp REs per se (e.g. needing to
>> escape things like capturing parentheses or the alternatives pipe); it's
>> needing to escape various things /further/ because REs can only be
>> specified in the form of a standard ELisp string. On several occasions
>> i've ended up using a combination of pcre-to-elisp and re-builder to try
>> to work out if the problem is too few backslashes, too many backslashes,
>> or both.
>
> That's why we have rx.

i find rx .... unwieldy. For me, rx is to PCREs as a putative function
'apply-function-to-each-element-of-sequence' is to 'mapcar'. i don't
want an English-language wrapper for regular expressions; i want regular
expressions that don't suffer from backslashitis. And, given the several
discussions by a number of people over the last several months -
including recently - about alternatives to the current RE-creation
system in Emacs, it seems to me that rx is hardly the final piece of the
Emacs Lisp RE puzzle.


Alexis.

Re: Regular expression creation [was: Re: Emacs Lisp's future]

[Aurélien Aptel]

[-- Attachment #1: Type: text/plain, Size: 525 bytes --]

Le 19 sept. 2014 00:14, "Alexis" <flexibeast@gmail.com> a écrit :
>
>
> Daniel Colascione writes:
>
> > On 09/17/2014 07:07 PM, Alexis wrote:
> i find rx .... unwieldy. For me, rx is to PCREs as a putative function
> 'apply-function-to-each-element-of-sequence' is to 'mapcar'. i don't
> want an English-language wrapper for regular expressions; i want regular
> expressions that don't suffer from backslashitis. And, given the several

See http://lists.gnu.org/archive/html/help-gnu-emacs/2014-07/msg00086.html

[-- Attachment #2: Type: text/html, Size: 802 bytes --]

Re: Regular expression creation [was: Re: Emacs Lisp's future]

[Alexis]

Aurélien Aptel writes:

> Le 19 sept. 2014 00:14, "Alexis" <flexibeast@gmail.com> a écrit :
>>
>> Daniel Colascione writes:
>>
>> > On 09/17/2014 07:07 PM, Alexis wrote:
>> i find rx .... unwieldy. For me, rx is to PCREs as a putative function
>> 'apply-function-to-each-element-of-sequence' is to 'mapcar'. i don't
>> want an English-language wrapper for regular expressions; i want regular
>> expressions that don't suffer from backslashitis. And, given the several
>
> See http://lists.gnu.org/archive/html/help-gnu-emacs/2014-07/msg00086.html

Yes, that was a great discussion - thank you for initiating it, and for
your explorations in this area! i might indeed try making use of that
`my-raw-string` function when creating REs .... But, setting aside the
debates around raw strings, i still feel something like Stefan's
suggestion for a built-in `re-escape` function would probably solve most
of the issues i have with ELisp REs.


Alexis.

Re: Regular expression creation [was: Re: Emacs Lisp's future]

[Stefan Monnier]

> debates around raw strings, i still feel something like Stefan's
> suggestion for a built-in `re-escape` function would probably solve most
> of the issues i have with ELisp REs.

And note that re-escape should be easy to write.  I guess the main issue
would be to decide what escape character to use (assuming "(" is used for
regexp-grouping, what should be used to match a plain open-parenthesis?
"\(" is not an option since it's the same string as "(", so you either
have to go with "\\(" which (as we know) sucks, or choose another escape
char than backslash).


        Stefan

Re: Regular expression creation [was: Re: Emacs Lisp's future]

[Alexis]

Stefan Monnier writes:

>> debates around raw strings, i still feel something like Stefan's
>> suggestion for a built-in `re-escape` function would probably solve most
>> of the issues i have with ELisp REs.
>
> And note that re-escape should be easy to write.  I guess the main issue
> would be to decide what escape character to use (assuming "(" is used for
> regexp-grouping, what should be used to match a plain open-parenthesis?
> "\(" is not an option since it's the same string as "(", so you either
> have to go with "\\(" which (as we know) sucks, or choose another escape
> char than backslash).

Hmm, good point .... Perhaps "!" could be the escape character? It seems
to me like a character that might be less likely to need to be literally
matched (or matched less frequently) than other visible ASCII
characters, and also seems to me to be reasonably memorable for this
purpose ....


Alexis.

Re: Regular expression creation [was: Re: Emacs Lisp's future]

[Filipp Gunbin]

On 19/09/2014 14:07 +0400, Alexis wrote:

> Hmm, good point .... Perhaps "!" could be the escape character? It seems
> to me like a character that might be less likely to need to be literally
> matched (or matched less frequently) than other visible ASCII
> characters, and also seems to me to be reasonably memorable for this
> purpose ....

It's meaning of "negate the following" is so widespread that it would be
confusing to use it.

-- 
    Filipp

Re: Regular expression creation [was: Re: Emacs Lisp's future]

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1693 bytes --]

On 09/18/2014 02:54 PM, Alexis wrote:
> 
> Daniel Colascione writes:
> 
>> On 09/17/2014 07:07 PM, Alexis wrote:
>>> i like this suggestion. i'm rather comfortable with Perl5 REs, and can
>>> find myself frustrated trying to create REs in ELisp. Having said that,
>>> the issue is not usually the syntax of ELisp REs per se (e.g. needing to
>>> escape things like capturing parentheses or the alternatives pipe); it's
>>> needing to escape various things /further/ because REs can only be
>>> specified in the form of a standard ELisp string. On several occasions
>>> i've ended up using a combination of pcre-to-elisp and re-builder to try
>>> to work out if the problem is too few backslashes, too many backslashes,
>>> or both.
>>
>> That's why we have rx.
> 
> i find rx .... unwieldy. For me, rx is to PCREs as a putative function
> 'apply-function-to-each-element-of-sequence' is to 'mapcar'. i don't
> want an English-language wrapper for regular expressions; i want regular
> expressions that don't suffer from backslashitis. And, given the several
> discussions by a number of people over the last several months -
> including recently - about alternatives to the current RE-creation
> system in Emacs, it seems to me that rx is hardly the final piece of the
> Emacs Lisp RE puzzle.

rx is sometimes awkward for small patterns, but it's a boon for larger
ones: see the work I did in compile.el, which involved rewriting the
"gnu" pattern with rx. It's also pretty easy to embed string regular
expressions inside rx forms.

Anyway, I'd love to see pcre2el or equivalent in core. It's a nice
packing for converting between regular expression formats.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Andreas Röhler]

On 17.09.2014 20:24, Jorgen Schaefer wrote:
> On Tue, 16 Sep 2014 18:03:17 +0200
> Lennart Borgman <lennart.borgman@gmail.com> wrote:
[ ... ]
> One thing I think would benefit Emacs Lisp as a language a lot would be
> a standard library cleanup. An effort to go through the libraries that
> come with Emacs, separate them into "standard library" and "extensions
> that come with Emacs", and then go through the "standard library",
> provide sane names when necessary (like setcar is provided for rplaca)
> and deprecating the old ones, or simply deprecate all but one version of
> functions with overlapping use (nth or elt, pick one). Finally, add
> standard libraries for common functionality that is currently lacking
> (see above).
>

[ ... ]

+1

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > provide sane names when necessary (like setcar is provided for rplaca)

setcar is not entirely equivalent to rplaca -- they return different
values.  That's why I gave it a different name.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Ivan Andrus]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

On Sep 18, 2014, at 9:17 AM, Richard Stallman <rms@gnu.org> wrote:

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

provide sane names when necessary (like setcar is provided for rplaca)


setcar is not entirely equivalent to rplaca -- they return different
values.  That's why I gave it a different name.


That appears to no longer be true (emacs-24 branch):

rplaca is an alias for `setcar' in `subr.el'.

(rplaca CELL NEWCAR)

Set the car of CELL to be NEWCAR.  Returns NEWCAR.

-Ivan

[-- Attachment #2: Type: text/html, Size: 941 bytes --]

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    That appears to no longer be true (emacs-24 branch):

    rplaca is an alias for `setcar' in `subr.el'.

    (rplaca CELL NEWCAR)

    Set the car of CELL to be NEWCAR.  Returns NEWCAR.

If I recall right, rplaca traditionally returned CELL, not NEWCAR.
Thus, this change seems like an error.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Andreas Röhler]

On 19.09.2014 01:19, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>      That appears to no longer be true (emacs-24 branch):
>
>      rplaca is an alias for `setcar' in `subr.el'.
>
>      (rplaca CELL NEWCAR)
>
>      Set the car of CELL to be NEWCAR.  Returns NEWCAR.
>
> If I recall right, rplaca traditionally returned CELL, not NEWCAR.
> Thus, this change seems like an error.
>

Hi Richard,

IIUC this was just an example.

Once mentioned a multitude of string-strip functions, used to remove whitespace-characters from beginning or end of string - while missing a canoncial form.

Maybe past upcoming release discuss the standard library should pay.

Best,

Andreas

Re: Emacs Lisp's future

[Andreas Schwab]

Ivan Andrus <darthandrus@gmail.com> writes:

> On Sep 18, 2014, at 9:17 AM, Richard Stallman <rms@gnu.org> wrote:
>
> setcar is not entirely equivalent to rplaca -- they return different
> values.  That's why I gave it a different name.
>
>
> That appears to no longer be true (emacs-24 branch):

For some value of "no longer".

Changes in Emacs 1.6

* rplaca is now a synonym for setcar, and rplacd for setcdr.
 eql is now a synonym for eq; it turns out that the Common Lisp
 distinction between eq and eql is insignificant in Emacs.
 numberp is a new synonym for integerp.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs Lisp's future

[David Kastrup]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Ivan Andrus <darthandrus@gmail.com> writes:
>
>> On Sep 18, 2014, at 9:17 AM, Richard Stallman <rms@gnu.org> wrote:
>>
>> setcar is not entirely equivalent to rplaca -- they return different
>> values.  That's why I gave it a different name.
>>
>>
>> That appears to no longer be true (emacs-24 branch):
>
> For some value of "no longer".
>
> Changes in Emacs 1.6
>
> * rplaca is now a synonym for setcar, and rplacd for setcdr.
>  eql is now a synonym for eq; it turns out that the Common Lisp
>  distinction between eq and eql is insignificant in Emacs.
>  numberp is a new synonym for integerp.

Well, the other changes have not survived when Emacs gained floating
point numbers.  Version 19.0 or so?

-- 
David Kastrup

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Emilio Lopes]

2014-09-16 18:03 GMT+02:00 Lennart Borgman <lennart.borgman@gmail.com>:
> Perhaps also the lack of possibility to enhance Emacs with code written in
> other languages? I think for example that Javascript will be something most
> future programmers will know. Could Guile make it easier to enhance Emacs
> with Javascript (as an alternative to Elisp)?

"It's nice that the students coming to us already know Java.  We just
have to teach them how to program."

                -- Michael Sperber

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 16 Sep 2014 11:50:54 -0400
> Cc: emacs-devel@gnu.org
> 
> One such option is Guile-Emacs.  This presumably would give us a faster
> implementation (at least in theory, their bytecode is significantly
> more efficient)

If by "bytecode" you mean the *.go files, then we should also keep in
mind that these are not architecture-independent as *.elc files are.
Not a catastrophe, of course, but something to remember.

> And of course, if Guile's own manpower dries up, Emacs would be forced
> to keep supporting Guile, which is more work than supporting just Elisp.

A data point: judging by "git log", Guile is currently developed and
maintained by about 3 active developers.

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lars Brinkhoff]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> The only standard language into which Elisp can evolve, AFAICT, is
> Common Lisp.  [ Now some readers get disappointed, while some others
> become excited.  ]

Excited.

> [...] we could use Common-Lisp libraries in Emacs.

That's almost possible already, because there is a Common Lisp
implementation (say 90% complete) for Emacs, including a compiler from
CL to Elisp.

Re: Emacs Lisp's future

[Stefan Monnier]

> That's almost possible already, because there is a Common Lisp
> implementation (say 90% complete) for Emacs, including a compiler from
> CL to Elisp.

Compiling from CL to Elisp means that the two aren't integrated, so
calling code from one to the other is not seamless.


        Stefan

Re: Emacs Lisp's future

[Lars Brinkhoff]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> Compiling from CL to Elisp means that the two aren't integrated, so
> calling code from one to the other is not seamless.

I don't know why you say that.  In my implementation it's currently
not entirely seamless, but close, and I can easily imagine ways to
make it so.

But anyway, this is not the time nor the place to discuss that.  It's
not really relevant to what you wrote about Emacs Lisp's future.
Sorry for the noise.

Re: Emacs Lisp's future

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So this email is partly a:
>>  - What now?  What's the chance of work towards guilemacs moving over to
>>    an official emacs git branch, and that port happening, anytime soon?
>>  - Is anyone running it?  How's it going for you?
>
> Good questions.  I've had the opportunity to think a bit more about
> Emacs Lisp and its possible evolution and I'm still not sure what to
> think about it.
>
> I see a few different options for Emacs Lisp.
>
> First, of course we can keep on evolving Elisp on its own.  This has
> worked OK for the last 30 years, so it's not such a terrible choice.
> The main problems I see with that:
> - Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
>   noticed more often.
> - Lack of some features, most notably FFI and concurrency.
> - Lack of manpower.
>
> This last point is for me the strongest motivation to try and move to
> some other system, where we could use other people's work.
>
> One such option is Guile-Emacs.  This presumably would give us a faster
> implementation (at least in theory, their bytecode is significantly
> more efficient), would give us an FFI, and would give us more manpower
> since we'd be benefiting from the work done on Guile.

That's not all that much manpower.  If you take a look at the commits in
the master branch that are not merges from the stable branch, I think
that more than 90% are from Andy Wingo.  And the turnaround times on the
issue tracker are not all that impressive.  It doesn't help that Andy
does not generally bother with the bug tracker, so most bugs actually
having a chance of getting fixed are those in the stable branch.

> Note that while Guile does come with support for threading,

<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14756>

Bug report about basic threading functionality not working, no response
since June 2013.

> it doesn't immediately let us use concurrency in Guile-Emacs, because
> of all the issues of synchronizing access to shared data, with all the
> existing Emacs code (both C and Elisp) assuming that this problem
> doesn't exist.  IOW, language support for concurrency is just a first
> step on the way to letting Emacs Lisp use concurrency.

Bug report about basic list functionality not working for lists of more
than 10009 members:

<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17485>

The proposed fix is rejected, but no alternative fix is done.

<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13101> is an example
for the inability to compile large expressions on Guile-2.0.  The basic
resume for this bug is "large expressions like that will only work with
Guile-2.2+", but since Guile-2.2 is pretty much a one-person venture, it
is very unclear how it will ever be suitable for a "stable release".

> Another detail that needs to be spelled out is the difference between
> the language and its implementation.  Guile-Emacs provides 2
> languages: Emacs Lips and Scheme (well, it also provides a few more,
> but that's not important).

It might very well become important if it leads to Emacs extensibility
in more procedural-syntax languages like Lua (the data model and basic
application of Emacs does not really call for functional programming, so
a functional programming language has no real advantage).  I see that as
mostly pie-in-the-sky material right now since getting Scheme right and
finding a workable balance between code complexity, performance,
scalability and reliability of the basic feature set seems more
important to focus on before fanning out.  But it is an interesting
long-term perspective.

> One of the main reasons for Emacs's enduring success is its large set
> of third party packages so obviously we can't drop support for Elisp
> any time soon.  And as much as I like Scheme, I'm very much
> unconvinced that it's really so much better that it's worth converting
> packages from Elisp to Scheme.

Scheme is not "better" than Elisp but it's certainly more standardized.
Since it is mostly employed for other tasks, code reusability for Emacs
will not be much of an ongoing advantage in practice.  I mean, how did
the Common Lisp based Emacs clone Hemlock fare?  I think nobody
remembers it any more.

> So if we go for Guile-Emacs, we'll be stuck with Guile, i.e. we'd have
> (old and new) packages that use Elisp, new packages that use Scheme,
> maybe yet other new packages that use, say, Javascript (or some other
> language support by Guile).  That would make the work of Emacs (and
> GNU ELPA) maintenance harder.
>
> And of course, if Guile's own manpower dries up, Emacs would be forced
> to keep supporting Guile, which is more work than supporting just
> Elisp.

And its dark corners are in areas entirely foreign to text editing.

> So, I think that ideally, we'd want to stick to Elisp, or some
> evolution thereof.  Sadly, I don't see how to evolve Elisp into
> Scheme: they are closely related languages, but the differences are
> large enough that it seems hard to reconcile them.

The main road to a common effort/codebase would be a common VM and byte
code I think.  The Guile compiler is layered, so one could hug the Elisp
byte compiler into a Guile compiler level considerably more remote from
the low-level machine than the VM and basically use all of its
compiler/optimizer structure without prescribing the high-level
language.

I think that this is the basic Emacs-Guile approach and it would make it
reasonably easy to extend Emacs with either Elisp or Scheme without
really having the languages merge in any manner.

I think that the basic idea is quite workable.  What I doubt is that the
GUILE project will readily scale to accommodate Emacs.  We've seen a lot
of growing pains when Emacs jumped on board of Bazaar, a project
unprepared for the challenges of a project of Emacs' size with regard to
both computational and social scalability.

Moving the responsibility for the Emacs low-level language layer and
management (for example, byte-compiled files will be created and managed
on disk by GUILE, and I am not convinced that GUILE is even close to
prepared to deal with a code base and add-on packages on the scale of a
typical Emacs-plus-extensions, plus multiple versions and installations)
into a different project will be a quite larger challenge and rather
permanent commitment than Bazaar was.

Given the current size of GUILE, this would require quite a large leap
of faith for which I see no return jump.

-- 
David Kastrup

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Bug report about basic list functionality not working for lists of more
> than 10009 members:
>
> <URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17485>

One of our procedures has a scalability problem.  Summarizing this as
"basic list functionality not working for lists of more than 10009
members" is a gross misrepresentation.

I responded within a day or so, and we had a conversation over the next
few days, but quickly ran into the problem that you wanted to change one
of our other API procedures in a non-standard way that conflicts with
our existing behavior, and you simply wouldn't take "no" for an answer.

> The proposed fix is rejected, but no alternative fix is done.

I would be glad to fix the problem myself, but I was hoping that you
would adapt your patch without changing that other API procedure, so
that you would get credit for your work.

> <URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13101> is an example
> for the inability to compile large expressions on Guile-2.0.  The basic
> resume for this bug is "large expressions like that will only work with
> Guile-2.2+"

Is it really a show-stopper that Guile 2.0 has limitations in the sizes
of some expression types?  Even C has similar limitations, and yet
somehow it seems to be successful.

      Mark

Re: Emacs Lisp's future

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Bug report about basic list functionality not working for lists of more
>> than 10009 members:
>>
>> <URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17485>
>
> One of our procedures has a scalability problem.  Summarizing this as
> "basic list functionality not working for lists of more than 10009
> members" is a gross misrepresentation.
>
> I responded within a day or so, and we had a conversation over the next
> few days, but quickly ran into the problem that you wanted to change one
> of our other API procedures in a non-standard way that conflicts with
> our existing behavior, and you simply wouldn't take "no" for an answer.

I should mention that most of the conversation we had regarding this
other API procedure (length+) occurred within another bug report:

  http://bugs.gnu.org/17296

Although it may appear from bug #17485 that I failed to respond to your
last messages, in fact the conversation had shifted to bug #17296 where
you failed to respond to my reasons for rejecting your proposed change.

      Mark

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Bug report about basic list functionality not working for lists of more
>> than 10009 members:
>>
>> <URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17485>
>
> One of our procedures has a scalability problem.  Summarizing this as
> "basic list functionality not working for lists of more than 10009
> members" is a gross misrepresentation.

It's "drop-right" in srfi-1, the basic list processing library. 10009 is
a typo, it's actually somewhere between 1000 and 10000 that GUILE bombs
out.  drop-right is basic enough that it is used in the implementation
of reduce-right, propagating the scalability problem.  I don't see it as
a gross misrepresentation, but haggling about that would be missing the
point anyway.

> I responded within a day or so, and we had a conversation over the next
> few days, but quickly ran into the problem that you wanted to change one
> of our other API procedures in a non-standard way

The "non-standard way" being the exact behavior of the Scheme reference
implementation for srfi-1.

> that conflicts with our existing behavior, and you simply wouldn't
> take "no" for an answer.

I gave up on this problem after pointing out that I saw no sensible
solution within the restrictions you wanted to see implemented and
I pointed that out.

At any rate, all this is missing the point.  The point was that GUILE is
not a vibrant project at a scale that can comfortably absorb the impact
of becoming Emacs' main Lisp engine.

Becoming Emacs' main Lisp engine will require timely answers to
limitations in the VM, compilers, library routines, and Scheme standard
interpretation that turn out to affect Emacs operations and Lisp
implementations.  In particular, it will not be an option to flag errors
for every operation that has no well-defined semantics in Scheme when
the corresponding operation in Common Lisp or Elisp happens to be
well-defined and needed.

And parting ways again in case the cooperation turns out bad is entirely
different from parting ways with Bazaar, since we are talking about
replacing Emacs' internals here rather than an external tool.

So the situation with GUILE-Emacs more or less can be characterized as:
you can check it in any time you want but you can never leave it.

>> <URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13101> is an
>> example for the inability to compile large expressions on Guile-2.0.
>> The basic resume for this bug is "large expressions like that will
>> only work with Guile-2.2+"
>
> Is it really a show-stopper that Guile 2.0 has limitations in the
> sizes of some expression types?

Semantic compiles grammars into code.  "some expression types" in that
bug report is a large "cond" statement, exactly the kind of thing a
typical action-annotated grammar will be compiled into by a parser
generator.  GUILE compiles a cond statement into nested if statements.
The VM does not offer any other renditions of conditionals, so _every_
way of expressing this function in terms of an open-coded block of code
rather than, say, tables containing closures, will fail compilation.

Yes, this can turn into sort of a scalability problem in my book.  And
Emacs is big.

> Even C has similar limitations, and yet somehow it seems to be
> successful.

C restricts the number of cases in a switch statement (which would be
the rough equivalent here)?  That's news to me.

-- 
David Kastrup

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> I responded within a day or so, and we had a conversation over the next
>> few days, but quickly ran into the problem that you wanted to change one
>> of our other API procedures in a non-standard way
>
> The "non-standard way" being the exact behavior of the Scheme reference
> implementation for srfi-1.

*sigh*, must we repeat our discussion here on emacs-devel, when it's
already in the bug tracker at http://bugs.gnu.org/17296 ?

To summarize: the spec says it's an error to pass 'length+' an improper
list.  Guile has always returned #f in that case, and existing code may
depend on that.  You want to change it to return the number of pairs in
the improper list.  Yes, that's what the reference implementation does.
I think it's too risky to change Guile's behavior in this case from one
non-error to another non-error, because existing code may silently start
misbehaving, and it would also cause things like (map + - * /) to return
() without reporting any error.

If you want to continue this discussion about 'length+', can you please
respond to my last message at http://bugs.gnu.org/17296 instead of here?

>> Is it really a show-stopper that Guile 2.0 has limitations in the
>> sizes of some expression types?
[...]
> Yes, this can turn into sort of a scalability problem in my book.  And
> Emacs is big.

The problem is fixed on the master branch, which is the only branch that
supports guile-emacs anyway.  So, why are limitations in Guile 2.0
relevant for guile-emacs?

Incidentally, the 'drop-right' scalability problem you complain above
above is also fixed on the master branch, and plenty of others.  The
reason is that our stacks are now automatically expanded as needed,
limited only by the available memory, so we can now write recursive
procedures in a natural way without ugly hacks to make them scalable.

See http://wingolog.org/archives/2014/03/17/stack-overflow for more.

      Mark

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> Is it really a show-stopper that Guile 2.0 has limitations in the
>>> sizes of some expression types?
> [...]
>> Yes, this can turn into sort of a scalability problem in my book.  And
>> Emacs is big.
>
> The problem is fixed on the master branch, which is the only branch
> that supports guile-emacs anyway.

Ok, so guile-emacs will not work with the "stable" branch.  If I take a
look at the "master" branch which is the only one supporting
guile-emacs, I see things like

    commit 2c02a21023c946a3d31c43417d440d6babbf2622
    Author: Andy Wingo <wingo@pobox.com>
    Date:   Sun Jun 29 14:29:20 2014 +0200

        Remove namesets.

        This was a failed experiment.  It had good space complexity but terrible
        time complexity.

        * module/Makefile.am: Update.
        * module/language/cps/nameset.scm: Remove.

[...]

    commit ec412d75627aeffbd816ac351eabcd1b533540c6
    Author: Andy Wingo <wingo@pobox.com>
    Date:   Thu Jun 19 08:49:05 2014 +0200

        Rewrite type inference pass to use namesets

        * module/Makefile.am: Build types.scm early, but don't block the rest of
          the build on it.

        * module/language/cps/types.scm: Rewrite to use namesets.

        * module/language/cps/dce.scm:
        * module/language/cps/type-fold.scm: Adapt to interface changes.

    commit 97ed2e77ab22e1695c5c4df6f5f6cfd98b90636f
    Author: Andy Wingo <wingo@pobox.com>
    Date:   Sun Jun 8 18:50:07 2014 +0200

        New module: (language cps nameset)

        * module/language/cps/nameset.scm: New file.
        * module/Makefile.am: Add new file.


which make clear that the master branch in GUILE is _highly_
experimental.  If you take a look at the GUILE developer list archives
at
<URL:http://lists.gnu.org/archive/html/guile-devel/2014-09/threads.html>
and previous months, you'll find that Andy Wingo does not participate in
any discussions there.

So basically GUILE master is his private playground which he works on in
bursts of activity without publicly visible communication.

I don't see that basing Emacs on that kind of setup is remotely
realistic without serious reorganization of how GUILE organizes and
views its development processes.  Emacs is a remarkably _stable_ and
reliable piece of software considering its size and age.

> So, why are limitations in Guile 2.0 relevant for guile-emacs?
>
> Incidentally, the 'drop-right' scalability problem you complain above
> above is also fixed on the master branch, and plenty of others.  The
> reason is that our stacks are now automatically expanded as needed,
> limited only by the available memory, so we can now write recursive
> procedures in a natural way without ugly hacks to make them scalable.

Personally, I don't consider automatically expanding stack size a
"solution" to a routine unnecessarily shoving all elements of an
arbitrarily large data structure interspersed with return addresses onto
a VM stack before processing it.  But then tastes may differ.  I started
programming at a time when I paid the equivalent of €400 for 64kB of
RAM, so throwing RAM at a problem was expensive enough to be habit
forming.

> See http://wingolog.org/archives/2014/03/17/stack-overflow for more.

At any rate, this is again a distraction.  The topic is how suitable
GUILE is for turning into the Lisp foundation of Emacs.

And I think that apart from reestablishing that I am a dishonest
manipulative asshole who does not know what he is talking about, we got
some insights into the current state of GUILE development that may help
in refining goals and obstacles in the process of making GUILE the Lisp
implementation underlying Emacs.

-- 
David Kastrup

Re: Emacs Lisp's future

[Nic Ferrier]

David Kastrup <dak@gnu.org> writes:

> And I think that apart from reestablishing that I am a dishonest
> manipulative asshole who does not know what he is talking about, we got
> some insights into the current state of GUILE development that may help
> in refining goals and obstacles in the process of making GUILE the Lisp
> implementation underlying Emacs.

LOL.

Me too.


Nic

Re: Emacs Lisp's future

[mhw]

David Kastrup <dak@gnu.org> writes:

> That's not all that much manpower.  If you take a look at the commits in
> the master branch that are not merges from the stable branch, I think
> that more than 90% are from Andy Wingo.

That's an interesting way to pretend that Ludovic and I don't exist, by
excluding merges.  Why should our contributions be excluded just because
they start out on the stable-2.0 branch and later flow to master by way
of merges?

      Mark

Re: Emacs Lisp's future

[David Kastrup]

mhw@netris.org writes:

> David Kastrup <dak@gnu.org> writes:
>
>> That's not all that much manpower.  If you take a look at the commits in
>> the master branch that are not merges from the stable branch, I think
>> that more than 90% are from Andy Wingo.
>
> That's an interesting way to pretend that Ludovic and I don't exist, by
> excluding merges.

Work on the stable branch is supposedly maintenance rather than
forward-looking development.

It's actually a good sign for a project's stability if more people work
on maintenance than on new things.  But I was commenting on the amount
of manpower getting work done on new things.

> Why should our contributions be excluded just because they start out
> on the stable-2.0 branch and later flow to master by way of merges?

Would you claim that the stable-2.0 branch is where new developments are
generally done?  That would seem like a somewhat unusual development
model.

-- 
David Kastrup

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> mhw@netris.org writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> That's not all that much manpower.  If you take a look at the commits in
>>> the master branch that are not merges from the stable branch, I think
>>> that more than 90% are from Andy Wingo.
>>
>> That's an interesting way to pretend that Ludovic and I don't exist, by
>> excluding merges.
>
> Work on the stable branch is supposedly maintenance rather than
> forward-looking development.
>
> It's actually a good sign for a project's stability if more people work
> on maintenance than on new things.  But I was commenting on the amount
> of manpower getting work done on new things.

Plenty of "new things" have been added to the stable-2.0 branch.
Just read the NEWS file for 2.0.

  http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob_plain;f=NEWS;hb=stable-2.0

>> Why should our contributions be excluded just because they start out
>> on the stable-2.0 branch and later flow to master by way of merges?
>
> Would you claim that the stable-2.0 branch is where new developments are
> generally done?  That would seem like a somewhat unusual development
> model.

Yes, it's unusual, but whenever possible we add new modules and other
features to the stable-2.0 branch, so that users of 2.0 will benefit
from them.  The exceptions are:

* Work related to the new compiler/VM, which have changed substantially
  on the master branch.

* Disruptive changes that carry a significant risk of adding new bugs.

* Changes that would break API or ABI compatibility.

     Mark

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> mhw@netris.org writes:
>>
>>> David Kastrup <dak@gnu.org> writes:
>>>
>>>> That's not all that much manpower.  If you take a look at the commits in
>>>> the master branch that are not merges from the stable branch, I think
>>>> that more than 90% are from Andy Wingo.
>>>
>>> That's an interesting way to pretend that Ludovic and I don't exist, by
>>> excluding merges.
>>
>> Work on the stable branch is supposedly maintenance rather than
>> forward-looking development.
>>
>> It's actually a good sign for a project's stability if more people work
>> on maintenance than on new things.  But I was commenting on the amount
>> of manpower getting work done on new things.
>
> Plenty of "new things" have been added to the stable-2.0 branch.
> Just read the NEWS file for 2.0.
>
>   http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob_plain;f=NEWS;hb=stable-2.0

That also lists plenty of things that have been _removed_ during the
stable-2.0 branch, breaking existing uses of it:

    Making the vector interface operate only on a single representation will
    allow future versions of Guile to compile loops involving vectors to
    more efficient native code.

    ** 'htons', 'htonl', 'ntohs', 'ntohl'

    These procedures, like their C counterpart, were used to convert numbers
    to/from network byte order, typically in conjunction with the
    now-deprecated uniform vector API.

    This functionality is now covered by the bytevector and binary I/O APIs.
    See "Interpreting Bytevector Contents as Integers" in the manual.

    ** 'gc-live-object-stats'

    It hasn't worked in the whole 2.0 series.  There is no replacement,
    unfortunately.

    ** 'scm_c_program_source'

    This internal VM function was not meant to be public.  Use
    'scm_procedure_source' instead.


For me personally, LilyPond's ongoing efforts to port to GUILE-2.0 have
been bitten by

    ** `define-public' is no a longer curried definition by default

    The (ice-9 curried-definitions) should be used for such uses.  See the
    manual for details.

It's noteworthy that the replacement (ice-9 curried-definitions) is not
working correctly in the current version of GUILE distributed in Ubuntu
and there are in fact _still_ broken functions in current stable-2.0.

Making a "stable" branch a moving target where functionality gets
removed leads to situations where code linking libguile dynamically
stops working.

That kind of policy would likely also end up to be problematic for
Emacs.

>>> Why should our contributions be excluded just because they start out
>>> on the stable-2.0 branch and later flow to master by way of merges?
>>
>> Would you claim that the stable-2.0 branch is where new developments
>> are generally done?  That would seem like a somewhat unusual
>> development model.
>
> Yes, it's unusual, but whenever possible we add new modules and other
> features to the stable-2.0 branch, so that users of 2.0 will benefit
> from them.  The exceptions are:
>
> * Work related to the new compiler/VM, which have changed substantially
>   on the master branch.
>
> * Disruptive changes that carry a significant risk of adding new bugs.
>
> * Changes that would break API or ABI compatibility.

Like removing functionality or moving it to separate modules?

At the current point of time, GUILE is a project that is mostly catering
for itself, so breaking its own rules whenever it is convenient affects
a limited circle outside of GUILE.  If GUILE wants to posit itself as
basic infrastructure for the GNU project, this setup is going to lead to
more problems than it does now.

-- 
David Kastrup

Re: Emacs Lisp's future

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The only standard language into which Elisp can evolve, AFAICT, is
> Common Lisp.  [ Now some readers get disappointed, while some others
> become excited.  ]

Yay, excited.

I think there's a somewhat greater cultural overlap between Emacs Lisp
and Common Lisp people ("get things done") than between Emacs Lisp
people and Scheme people ("interesting academically").

Anyway, I think the dangled sorta-promise that Emacs would eventually
shift to Guile might have stifled Emacs Lisp development.  Whenever
somebody has brought up the issue of evolving Emacs Lisp (to
multi-threadedness or whatever's fun), they're usually discouraged by
others piping in with "oh, Emacs is moving to Guile, anyway, so don't
bother".

Perhaps if we just say firmly "Emacs Lisp is Emacs Lisp, and should
evolve in a Common Lisp direction, but without the whole-sale porting to
a third-party system: Please start hacking now" we'd attract people
interested in doing low-level language stuff.

Doesn't that sound like a fun opportunity?

*crosses fingers*

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Emacs Lisp's future

[Nic Ferrier]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I think there's a somewhat greater cultural overlap between Emacs Lisp
> and Common Lisp people ("get things done") than between Emacs Lisp
> people and Scheme people ("interesting academically").
>
> Anyway, I think the dangled sorta-promise that Emacs would eventually
> shift to Guile might have stifled Emacs Lisp development.  Whenever
> somebody has brought up the issue of evolving Emacs Lisp (to
> multi-threadedness or whatever's fun), they're usually discouraged by
> others piping in with "oh, Emacs is moving to Guile, anyway, so don't
> bother".

I don't think that's true.

From my perspective, what's stopping more people getting involved is the
community, which is sometimes quite negative and the tooling, which is
baroque.

The tooling is being fixed. I am *very* excited about that.

I hope the tooling will bring more people which will re-energize the
current community and add positivity.


Nic

Re: Emacs Lisp's future

[David Kastrup]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> I think there's a somewhat greater cultural overlap between Emacs Lisp
>> and Common Lisp people ("get things done") than between Emacs Lisp
>> people and Scheme people ("interesting academically").
>>
>> Anyway, I think the dangled sorta-promise that Emacs would eventually
>> shift to Guile might have stifled Emacs Lisp development.  Whenever
>> somebody has brought up the issue of evolving Emacs Lisp (to
>> multi-threadedness or whatever's fun), they're usually discouraged by
>> others piping in with "oh, Emacs is moving to Guile, anyway, so don't
>> bother".
>
> I don't think that's true.
>
> From my perspective, what's stopping more people getting involved is the
> community, which is sometimes quite negative and the tooling, which is
> baroque.

Well, let's take a comparison:

git shortlog -s --since "1 month ago" origin/master

in a current GUILE repository gives

     2  Andy Wingo

while in a current Emacs repository it gives

     4  Alan Mackenzie
     1  Alp Aker
     4  Christoph Scholtes
     1  Christopher Schmidt
     5  Daniel Colascione
     1  Detlev Zundel
    27  Dmitry Antipov
    50  Eli Zaretskii
     1  Fabián Ezequiel Gallina
    26  Glenn Morris
     1  Ivan Shmakov
     5  Jan D.
     1  Jay Belanger
     1  João Távora
     1  Kan-Ru Chen
     1  Karol Ostrovsky
     3  Katsumi Yamaoka
     8  Ken Brown
     1  Ken Olum
     1  Lars Ljung
     2  Lars Magne Ingebrigtsen
     4  Leo Liu
    15  Michael Albinus
     1  Michael Heerdegen
    34  Paul Eggert
     1  Rasmus Pank Roulund
     1  Reuben Thomas
     9  Sam Steingold
    16  Stefan Monnier
     2  Thierry Volpiatto
     1  YAMAMOTO Mitsuharu
     8  martin rudalics

That does not particularly make Emacs look like a project keeping people
from getting involved.  Of course, the project/repository structure of
both projects is different, but the basic idea that forward-looking
development happens in the master branch is loosely common to both.

If we take GNU LilyPond, incidentally based on GUILE 1.8, for reference,
we get something like

git shortlog -s --since "1 month ago" origin
    60  David Kastrup
     4  James Lowe
     6  Janek Warchoł
     3  Jean-Charles Malahieude
     4  Julien Rioux
     5  Keith OHara
    19  Phil Holmes
     3  Trevor Daniels
     1  Walter Garcia-Fontes

Now this is a project that has to suffer from a lead developer who is
considered to be sometimes (or more) quite negative, and the tooling,
namely GUILE 1.8 which is considered outdated and unmaintained for
something like 5 years or so, can also be called baroque.

The result is a quite more peaked distribution of contributions per
developer than with Emacs which has sort of a plateau at the top.

So I think that the news of Emacs' demise due to the named reasons is
quite exaggerated.  There is always room for improvement, of course, but
no particular need to panic.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Nic Ferrier <nferrier@ferrier.me.uk>
> Date: Wed, 17 Sep 2014 11:53:16 +0100
> 
> I hope the tooling will bring more people which will re-energize the
> current community and add positivity.

Sorry, but I will believe it when I see it.

Re: Emacs Lisp's future

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nic Ferrier <nferrier@ferrier.me.uk>
>> Date: Wed, 17 Sep 2014 11:53:16 +0100
>> 
>> I hope the tooling will bring more people which will re-energize the
>> current community and add positivity.
>
> Sorry, but I will believe it when I see it.


When I started coding clojure a couple of years ago, I went through a
process of rediscovering the state of the art in Emacs even though I
have been using it for 20 years; not just the Clojure specific stuff,
but things like magit.

In terms of tooling, I would argue strongly that package.el and more
recently MELPA stable have made a significant difference to the
community. It is now incredibly easy for me to update one of my
packages, do a release and have it in peoples hands.

I write code in the first instance for my own benefit. Decreasing the
additional effort that I need to make to release it to others is  an
important contribution.

I am with Nic; I think it's already happening.

Phil

Re: Emacs Lisp's future

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> First, of course we can keep on evolving Elisp on its own.  This has
> worked OK for the last 30 years, so it's not such a terrible choice.
> The main problems I see with that:
> - Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
>   noticed more often.

I've been going through some 10 year old elisp of mine recently. The
thing that surprises me is how many times I mention performance in it. I
rarely worry about this these days. Elisp performance as is seems rarely
an issue.

Where I would say that there is an issue is that too much of Emacs is
written in C. Having a faster elisp would allow moving more into lisp
and thus having more of Emacs extensible dynamically.

> - Lack of some features, most notably FFI and concurrency.
> - Lack of manpower.

I'd add a fourth. People who want to extend Emacs for their own purposes
have to learn it. Having JS extensibility would be an enourmous win.


> So if we go for Guile-Emacs, we'll be stuck with Guile, i.e. we'd
> have (old and new) packages that use Elisp, new packages that use
> Scheme, maybe yet other new packages that use, say, Javascript (or some
> other language support by Guile).  That would make the work of Emacs
> (and GNU ELPA) maintenance harder.

Thinking of Emacs as an entire ecosystem, most of Emacs is already
maintained independently from either Emacs core or GNU ELPA. For common
languages (like Javascript), the maintaince might be ligtened because
more people would be available.

Of course, it is worth mentioning that some of the big advantages from a
language like Javascript come from the implementation. A guile based
implementation wouldn't benefit from the intensive JS VM development
that has happend over the years.

Phil

Re: Emacs Lisp's future

[Nic Ferrier]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> I'd add a fourth. People who want to extend Emacs for their own purposes
> have to learn it. Having JS extensibility would be an enourmous win.

From what I've seen that isn't true. Most people learn just enough to
fix their problem.

Emacs Lisp is many people's least understood practical language (by
which I mean, they understand enough to do something with it). A bit
like Javascript used to be (how do I verify a form before it submits
again?)


Nic

Re: Emacs Lisp's future

[Stefan Monnier]

> I've been going through some 10 year old elisp of mine recently.
> The thing that surprises me is how many times I mention performance in
> it.  I rarely worry about this these days.  Elisp performance as is
> seems rarely an issue.

I think this reflects 2 things:
- machines still did get faster over the last 10 years (much less so
  than over the preceding 10 years, but also probably much more so than
  over the next 10 years).
- you've internalized Elisp performance sufficiently that you
  self-censor yourself such that you don't bump into it.

> Where I would say that there is an issue is that too much of Emacs is
> written in C.  Having a faster elisp would allow moving more into lisp
> and thus having more of Emacs extensible dynamically.

That's part of the idea, yes.

> I'd add a fourth. People who want to extend Emacs for their own purposes
> have to learn it.  Having JS extensibility would be an enourmous win.

I'm far from convinced.  It may seem that way if you don't think too
hard about what it would look like, but once you realize that your JS
hacker will have to learn how to manipulate Elisp lists, Elisp vectors,
Elisp symbols, how to access Elisp functions and Elisp variables ...

That JS code won't look very much like your typical JS code.  It will be
so baroque that they will "have to learn it".  And they may even hate it
more (because it surely will look ugly).

Interoperability between languages is *hard*.  Doing it so the result is
lightweight and elegant (and moderately efficient) is rarely possible.


        Stefan

Re: Emacs Lisp's future

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've been going through some 10 year old elisp of mine recently.
>> The thing that surprises me is how many times I mention performance in
>> it.  I rarely worry about this these days.  Elisp performance as is
>> seems rarely an issue.
>
> I think this reflects 2 things:
> - machines still did get faster over the last 10 years (much less so
>   than over the preceding 10 years, but also probably much more so than
>   over the next 10 years).

Well, Emacs is a text editor. The CPU demands of the task haven't
increased that much over the last 10 years either.

> - you've internalized Elisp performance sufficiently that you
>   self-censor yourself such that you don't bump into it.

Have you looked at any of my code recently? Still, yes, you may have a
point.



>
>> Where I would say that there is an issue is that too much of Emacs is
>> written in C.  Having a faster elisp would allow moving more into lisp
>> and thus having more of Emacs extensible dynamically.
>
> That's part of the idea, yes.
>
>> I'd add a fourth. People who want to extend Emacs for their own purposes
>> have to learn it.  Having JS extensibility would be an enourmous win.
>
> I'm far from convinced.  It may seem that way if you don't think too
> hard about what it would look like, but once you realize that your JS
> hacker will have to learn how to manipulate Elisp lists, Elisp vectors,
> Elisp symbols, how to access Elisp functions and Elisp variables ...
>
> That JS code won't look very much like your typical JS code.  It will be
> so baroque that they will "have to learn it".  And they may even hate it
> more (because it surely will look ugly).
>
> Interoperability between languages is *hard*.  Doing it so the result is
> lightweight and elegant (and moderately efficient) is rarely possible.

Yes, you are absolutely right. I am using Clojure for its Java interop
at the moment, and it has been done well. Replicating that would be no
mean feat.

Phil

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
> Date: Wed, 17 Sep 2014 14:54:22 +0100
> Cc: emacs-devel@gnu.org
> 
> > - machines still did get faster over the last 10 years (much less so
> >   than over the preceding 10 years, but also probably much more so than
> >   over the next 10 years).
> 
> Well, Emacs is a text editor. The CPU demands of the task haven't
> increased that much over the last 10 years either.

You are wrong here.  Emacs 23 added many CPU-intensive features, like
visual-line-mode.  Emacs 24 added bidirectional display, which makes
redisplay need roughly twice as much juice as Emacs 23 needed.  And
these are only two examples that pop up in my head after a 3-sec
thought; I'm sure there are more.

So it's not like we use up every additional CPU cycle the chips are
giving us, but we are certainly using significantly more than we did
10 years ago.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> You are wrong here.  Emacs 23 added many CPU-intensive features, like
> visual-line-mode.  Emacs 24 added bidirectional display, which makes
> redisplay need roughly twice as much juice as Emacs 23 needed.  And
> these are only two examples that pop up in my head after a 3-sec
> thought; I'm sure there are more.
>
> So it's not like we use up every additional CPU cycle the chips are
> giving us, but we are certainly using significantly more than we did
> 10 years ago.

One of the most costly changes was post 20.2 (?) when buffer and string
positions became expressed as character offsets rather than byte
offsets.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Wed, 17 Sep 2014 16:30:44 +0200
> 
> One of the most costly changes was post 20.2 (?) when buffer and string
> positions became expressed as character offsets rather than byte
> offsets.

Yes, well, but that was more than 10 years ago (1997, if you are right
and it was 20.2).

Re: Emacs Lisp's future

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
>> Date: Wed, 17 Sep 2014 14:54:22 +0100
>> Cc: emacs-devel@gnu.org
>> 
>> > - machines still did get faster over the last 10 years (much less so
>> >   than over the preceding 10 years, but also probably much more so than
>> >   over the next 10 years).
>> 
>> Well, Emacs is a text editor. The CPU demands of the task haven't
>> increased that much over the last 10 years either.
>
> You are wrong here.  Emacs 23 added many CPU-intensive features, like
> visual-line-mode.  Emacs 24 added bidirectional display, which makes
> redisplay need roughly twice as much juice as Emacs 23 needed.  And
> these are only two examples that pop up in my head after a 3-sec
> thought; I'm sure there are more.
>
> So it's not like we use up every additional CPU cycle the chips are
> giving us, but we are certainly using significantly more than we did
> 10 years ago.


What can I say? I don't sit around waiting for emacs to do stuff
nowadays when I am using it. And I've just implemented a package that
copies, then searches and replaces an entire buffer on every keypress.
And I don't notice it running for moderate size files.

The manual talks about the performance danger of overlay, but I've just
put an overlay an every word in a 300 line buffer, and I can't notice
that either.

Even though Emacs now takes considerably more than Eight Megabytes, it
is not Constantly Swapping. CPU has grown quicker than Emacs demands.

Phil

performance isn't a concern in ... Emacs Lisp's future

[Nic Ferrier]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> What can I say? I don't sit around waiting for emacs to do stuff
> nowadays when I am using it. And I've just implemented a package that
> copies, then searches and replaces an entire buffer on every keypress.
> And I don't notice it running for moderate size files.
>
> The manual talks about the performance danger of overlay, but I've just
> put an overlay an every word in a 300 line buffer, and I can't notice
> that either.
>
> Even though Emacs now takes considerably more than Eight Megabytes, it
> is not Constantly Swapping. CPU has grown quicker than Emacs demands.

Another reason is we learned how to write better code. In the old days
we'd all be hacking 10k line C++ files.

But programmers with that level of maturity are still around. Many of
them using Eclipse.


Nic

Re: performance isn't a concern in ... Emacs Lisp's future

[David Kastrup]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> phillip.lord@newcastle.ac.uk (Phillip Lord) writes:
>
>> What can I say? I don't sit around waiting for emacs to do stuff
>> nowadays when I am using it. And I've just implemented a package that
>> copies, then searches and replaces an entire buffer on every keypress.
>> And I don't notice it running for moderate size files.
>>
>> The manual talks about the performance danger of overlay, but I've just
>> put an overlay an every word in a 300 line buffer, and I can't notice
>> that either.
>>
>> Even though Emacs now takes considerably more than Eight Megabytes, it
>> is not Constantly Swapping. CPU has grown quicker than Emacs demands.
>
> Another reason is we learned how to write better code. In the old days
> we'd all be hacking 10k line C++ files.

In the old days there was no C++.  There still isn't in the Emacs
codebase, and src/xdisp.c has 30k lines.

> But programmers with that level of maturity are still around.  Many of
> them using Eclipse.

Shrug.  I've written cross compiler and assembler for FORTH systems
fitting into a few kilobytes and not even organized into files but
rather 1kB blocks of source code for compiling industrial applications.
I had to write bootstrap loader and BIOS for my first self-soldered
system before being able to boot the first time.

I can probably still rattle off the execution cycles for most Z80
assembly language commands and (from a later job) most of those for
Pentium I (though I'm fuzzy these days what the second integer pipeline
was not getting used for).

I am sure you'll find better reasons to call me immature than my too
long computing history purportedly rendering me unable to write good
code.

By the way: one of the most lucid pieces of code I had the pleasure to
come across was a Reversi program written in Z80 assembly language for
which only had the binaries and needed to disassemble it in order to
adapt it to a different OS and storage location.  It was a textbook
variant of Alpha-Beta pruning with scoring tables mapped excellently to
the processor architecture, using the index registers very consistently,
and with the code structured into clearly recognizable tasks.

No, we haven't really "learned how to write better code", just like
electronic instruments have not taught us how to write better music.

Having a larger variety of wrappings for content does not really change
what substance actually lies at the core.

-- 
David Kastrup

Re: Emacs Lisp's future

[Thorsten Jolitz]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> The manual talks about the performance danger of overlay, but I've just
> put an overlay an every word in a 300 line buffer, and I can't notice
> that either.

I had a different experience when I tried to implement org-hlc.el (Org
Hidden Line Cookies [fn:1]) with overlays and buffer change hooks. I always
test with a 7000+ lines elisp file, and user experience was pretty close
to 'unusable' in a file of that size, so I came to the conclusion that
Emacs perfomance is actually quite limited in that area. 

* Footnotes

[fn:1] org-hlc shows the number of hidden lines in a folded subtree in
little cookie that looks like this [#20] in the folded headline.

-- 
cheers,
Thorsten

Re: Emacs Lisp's future

[Stefan Monnier]

> Well, Emacs is a text editor.  The CPU demands of the task haven't
> increased that much over the last 10 years either.

Remember when Emacs was the big&slow editor?  Now compare Emacs to
Eclipse and you'll see that the resources that can be used for "text
editing" are ... limitless


        Stefan

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Interoperability between languages is *hard*.  Doing it so the result
> is lightweight and elegant (and moderately efficient) is rarely
> possible.

Well, the Elisp and Scheme integration is pretty great.

Maybe you know all of the following, but for those who didn't:

- There's no data type conversion because the same object in memory is
  the right type in both languages, including procedures/functions.

- A convenient `import-guile-module' syntax in Elisp could be
  implemented relatively trivially, which would walk a module and put
  procedure bindings into the function namespace and other bindings into
  the variable namespace of Elisp (both with a user-provided prefix).
  So you can have the API of any Guile module in Elisp as if it were
  just an Elisp library.

- There's simply no efficiency issue.  After you import something from
  the other language, you just have it; it's indistinguishable from
  something you defined in the current language, analogous to how C
  implemented functions (subrs) are generally indistinguishable from
  Elisp implemented functions; they're now all "libguile procedure
  objects."

- I can trivially import the Elisp function and variable namespaces in
  Scheme code with a pair of different prefixes.  I then have the whole
  Elisp API in my Scheme code, sans Elisp macros.

And here some issues, maybe-issues, and non-issues we have:

- Currently strings are an exception to the first point.  I hope that
  won't be too difficult to fix.

- The handling of nil vs. Scheme #f and '() is admittedly a wart, but
  will probably remain hidden, not causing issues in code you write.

- Currently Guile's Elisp implementation is buggy and slow in some
  things, but this is unrelated to the interoperation of the languages.

- Conceptually, Scheme macros can work in Elisp.  Might need some work
  to actually make this work, or it might be trivial; I don't know.

- The biggest long-term issue might be that Elisp macros will not work
  in Scheme, and this is relevant only to people who want to use Elisp
  APIs in Scheme, not affecting Elisp users.

If Guile-Emacs succeeds, you really will be able to

(import-guile-module (gl) gl-)  ;`gl-' is the prefix

and write OpenGL code in Elisp.

You can probably already do that in the current state of Guile-Emacs,
except that `import-guile-module' needs to be implemented; for now you
can individually do e.g.:

(defalias 'gl-set-draw-buffer (guile-ref (gl) set-gl-draw-buffer))


I think this is worth pursuing.  It opens Emacs to a whole new world.
Yes, a lot of work needs to be done, but I see no fundamental problems.

If some Emacs developers gave Guile-Emacs some love, it might land in a
couple years I'd guess, and by then Guile will possibly have some form
of just-in-time and/or ahead-of-time native compilation for Elisp as
well as Scheme...


Taylan

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> Interoperability between languages is *hard*.  Doing it so the result
>> is lightweight and elegant (and moderately efficient) is rarely
>> possible.
>
> Well, the Elisp and Scheme integration is pretty great.
>
> Maybe you know all of the following, but for those who didn't:
>
> - There's no data type conversion because the same object in memory is
>   the right type in both languages, including procedures/functions.

Uh, no?  In Lisp, functions are bound to the function cell of a symbol
which is different from the value cell of a symbol.  In GUILE, nothing
is bound to a symbol at all.  Modules establish a mapping between
symbols and values, and functions do not have separate mappings from
values.

> - A convenient `import-guile-module' syntax in Elisp could be
>   implemented relatively trivially, which would walk a module and put
>   procedure bindings into the function namespace and other bindings into
>   the variable namespace of Elisp (both with a user-provided prefix).
>   So you can have the API of any Guile module in Elisp as if it were
>   just an Elisp library.

See?

> - There's simply no efficiency issue.  After you import something from
>   the other language, you just have it; it's indistinguishable from
>   something you defined in the current language, analogous to how C
>   implemented functions (subrs) are generally indistinguishable from
>   Elisp implemented functions; they're now all "libguile procedure
>   objects."

That's a misrepresentation since an import does not change the state of
a "libguile procedure object" but rather involves the establishing of
bindings.  The data structures are also not compatible: Lisp lists end
with a value that serves as a _false_ boolean, Scheme lists end with a
value that serves as a _true_ boolean.

> And here some issues, maybe-issues, and non-issues we have:

[...]

> - The handling of nil vs. Scheme #f and '() is admittedly a wart, but
>   will probably remain hidden, not causing issues in code you write.

I have problems seeing how it can remain hidden.

> - Currently Guile's Elisp implementation is buggy and slow in some
>   things, but this is unrelated to the interoperation of the
>   languages.

If it is unrelated to the interoperation of the languages, this would
imply that GUILE itself is buggy and slow.  You might want to reconsider
that statement.

> - Conceptually, Scheme macros can work in Elisp.  Might need some work
>   to actually make this work, or it might be trivial; I don't know.
>
> - The biggest long-term issue might be that Elisp macros will not work
>   in Scheme, and this is relevant only to people who want to use Elisp
>   APIs in Scheme, not affecting Elisp users.

Huh, this actually is a surprise to me since defmacro works pretty much
the same in GUILE/Scheme as it would in Elisp.

> Yes, a lot of work needs to be done, but I see no fundamental
> problems.

The main problem I see is that Emacs is a large project regarding its
code and developer base compared to GUILE and I don't see that GUILE can
easily scale up.

The priorities of Elispers and Schemers regarding language features and
focuses are different, and I don't see overrolling GUILE with the
requirements, code, interests, and developers of Emacs going smoothly.

> If some Emacs developers gave Guile-Emacs some love, it might land in
> a couple years I'd guess, and by then Guile will possibly have some
> form of just-in-time and/or ahead-of-time native compilation for Elisp
> as well as Scheme...

Elisp is not really a special case here since stuff like native
compilation would happen at a decidedly lower layer than the
Elisp/Scheme distinction.  GUILE has a rather impressive number of
compiler layers and it is probably only the top layer that would make a
difference between Elisp and Scheme while something like the two bottom
layers would distinguish between native compilation and bytecode.

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

>> - There's no data type conversion because the same object in memory is
>>   the right type in both languages, including procedures/functions.
>
> Uh, no?  In Lisp, functions are bound to the function cell of a symbol
> which is different from the value cell of a symbol.  In GUILE, nothing
> is bound to a symbol at all.  Modules establish a mapping between
> symbols and values, and functions do not have separate mappings from
> values.

I'm not talking about the language-semantic locations of objects, rather
that for example the same "physical" object in your RAM is at the same
time an Elisp function, and a Scheme procedure.  `functionp' and
`procedure?' both check for the same data type.  (Likewise for other
types.)  No mutation, packing, serialization/deserialization, or any
other things needs to happen to a byte sequence in memory for it to be
usable from either language.

There's probably a simpler way to word what I have in mind and it's
currently escaping me; hope this was clear enough.  :)

>> - There's simply no efficiency issue.  After you import something from
>>   the other language, you just have it; it's indistinguishable from
>>   something you defined in the current language, analogous to how C
>>   implemented functions (subrs) are generally indistinguishable from
>>   Elisp implemented functions; they're now all "libguile procedure
>>   objects."
>
> That's a misrepresentation since an import does not change the state of
> a "libguile procedure object" but rather involves the establishing of
> bindings. [...]

Yes?  I'm just saying that once you put a libguile procedure into the
function slot of an Elisp symbol, it's indistinguishable from an Elisp
function or a subr being in that slot.  There is no overhead while
calling it.  (Elisp functions are compiled into libguile procedures
anyway.)

Actually, this is just a special-case of the first point, regarding the
applicable data type (libguile procedures) being handled in a uniform
way without extra overhead in the case it "comes from the other
language."

>> - The handling of nil vs. Scheme #f and '() is admittedly a wart, but
>>   will probably remain hidden, not causing issues in code you write.
>
> I have problems seeing how it can remain hidden.

Please mention any concrete problematic cases you can think up.

>> - Currently Guile's Elisp implementation is buggy and slow in some
>>   things, but this is unrelated to the interoperation of the
>>   languages.
>
> If it is unrelated to the interoperation of the languages, this would
> imply that GUILE itself is buggy and slow.  You might want to reconsider
> that statement.

It implies Guile's Elisp implementation is buggy and slow.  That is one
unfinished component of Guile.

(The main body of the Elisp implementation that powers Guile-Emacs
resides within Guile itself; I think some key Emacs data types like
buffers and windows are missing if you use it purely from Guile instead
of running Guile-Emacs, but you can use it as a general-purpose
language.)

>> - The biggest long-term issue might be that Elisp macros will not work
>>   in Scheme, and this is relevant only to people who want to use Elisp
>>   APIs in Scheme, not affecting Elisp users.
>
> Huh, this actually is a surprise to me since defmacro works pretty much
> the same in GUILE/Scheme as it would in Elisp.

The problem is that the Elisp macro will just output, say, the symbol
`mapc' into the middle of your Scheme code, where it will not be bound
to the Elisp `mapc' function during subsequent compilation.

Of course, you can otherwise use defmacro in Scheme, though it's highly
discouraged since you have hygienic macros, which guarantee correct
bindings.

>> Yes, a lot of work needs to be done, but I see no fundamental
>> problems.
>
> The main problem I see is that Emacs is a large project regarding its
> code and developer base compared to GUILE and I don't see that GUILE
> can easily scale up.

Matters such as portability and bug-fixing speed which are related to
"manpower" seem open to improvement indeed.  I think that would actually
be a way in which the Guile-Emacs effort would pay back to Guile.

If I'm not mistaken, adopting ECL or some other foreign piece of
software would mean that the upstream developers of it aren't guaranteed
to cooperate, which could be problematic despite the overall maturity of
that piece of software.

Guile on the other hand is GNU software; it's the official GNU extension
language that's being massively revitalized to be worth of its name.
GCC, GDB, and more can be extended in Guile.  It's tied with Guix as
well; the package manager for the upcoming official GNU distribution.
After so many years, things are really getting together in technical
terms in the GNU project, and Guile is what's pulling it all together.
That's why I'm so interested in Guile myself, and think that Emacs just
ought to be a part of this as well.  There is some massive potential
here for the whole of the GNU system getting a lot more attention, and
Emacs itself will benefit from it greatly.

One needs to give it time of course, but I think it's the right
direction, and is what will be most beneficial to Emacs individually as
well in the long term, in case one doesn't care about GNU.

And of course, a large portion of the work is already done here.
Another option would mean starting from scratch.

> The priorities of Elispers and Schemers regarding language features and
> focuses are different, and I don't see overrolling GUILE with the
> requirements, code, interests, and developers of Emacs going smoothly.

The priorities of Elispers seem extremely diverse.  People hacked up web
servers with Elisp and seem motivated to do crazier things.  All the
Guile libraries should be welcome.

>> If some Emacs developers gave Guile-Emacs some love, it might land in
>> a couple years I'd guess, and by then Guile will possibly have some
>> form of just-in-time and/or ahead-of-time native compilation for Elisp
>> as well as Scheme...
>
> Elisp is not really a special case here since stuff like native
> compilation would happen at a decidedly lower layer than the
> Elisp/Scheme distinction.  GUILE has a rather impressive number of
> compiler layers and it is probably only the top layer that would make
> a difference between Elisp and Scheme while something like the two
> bottom layers would distinguish between native compilation and
> bytecode.

Precisely.  I just meant to stress out that whatever nifty low-level
features Guile will improve on, Elisp, Scheme, and other languages on
Guile will benefit from them automatically.

Taylan

Re: Emacs Lisp's future

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

On 09/17/2014 01:11 PM, Taylan Ulrich Bayirli/Kammer wrote:
> Actually, this is just a special-case of the first point, regarding the
> applicable data type (libguile procedures) being handled in a uniform
> way without extra overhead in the case it "comes from the other
> language."
> 
>>> - The handling of nil vs. Scheme #f and '() is admittedly a wart, but
>>>   will probably remain hidden, not causing issues in code you write.
>>
>> I have problems seeing how it can remain hidden.
> 
> Please mention any concrete problematic cases you can think up.

Imagine the round trip a value might take: #f in Scheme to nil in elisp
to something back in Scheme. If you alias #f, (), and nil together from
an elisp perspective, then when you need to differentiate these values,
you can't, because you've just thrown away the information you'd need to
do that. You can't make the object's true identity a hidden property of
the value either, because that breaks perfectly valid code like this:

  (defun identity-x (x) (if (eq x nil) nil x))

I'd be happy to use the lower-level portions of Guile for compiling
elisp more efficiently, but I'm adamantly against trying to support both
Scheme and elisp simultaneously. All in all, I think the effort would be
better spent elsewhere.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Stefan Monnier]

> Imagine the round trip a value might take:

There's no translation involved, so no "round-trip" problem.
Instead the problem is that there are new kinds of data.  These can come
from Scheme and look weird from the Elisp side.

Basically, Scheme has #f, (), and nil as 3 distinct objects.
So Guile-Emacs picked one of those as being Elisp's nil, so as long as
you stay all within Elisp things work just fine (presumably), but as
soon as some Scheme gets into the picture you might get new values which
are similar to nil but aren't `eq' to it.

> I'd be happy to use the lower-level portions of Guile for compiling
> elisp more efficiently, but I'm adamantly against trying to support both
> Scheme and elisp simultaneously.

I tend to agree.  But I'm not sure it's easy to really stay 100% within
Elisp with a Guile-VM.  E.g. I suspect some errors signals coming from
Guile's runtime will end up using Scheme-style data and will end up
spilling into the Elisp side if we're not extra careful.


        Stefan

Re: Emacs Lisp's future

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 09/17/2014 01:11 PM, Taylan Ulrich Bayirli/Kammer wrote:
>> Actually, this is just a special-case of the first point, regarding the
>> applicable data type (libguile procedures) being handled in a uniform
>> way without extra overhead in the case it "comes from the other
>> language."
>> 
>>>> - The handling of nil vs. Scheme #f and '() is admittedly a wart, but
>>>>   will probably remain hidden, not causing issues in code you write.
>>>
>>> I have problems seeing how it can remain hidden.
>> 
>> Please mention any concrete problematic cases you can think up.
>
> Imagine the round trip a value might take: #f in Scheme to nil in elisp
> to something back in Scheme.

It doesn't.

See
<URL:https://www.gnu.org/software/guile/manual/html_node/Nil.html#Equality-2>.
I quote:

    However, in Elisp, '(), #f, and nil are all equal (though not eq).

    (defvar f (make-scheme-false))
    (defvar eol (make-scheme-null))
    (eq f eol) ⇒ nil
    (eq nil eol) ⇒ nil
    (eq nil f) ⇒ nil
    (equal f eol) ⇒ t
    (equal nil eol) ⇒ t
    (equal nil f) ⇒ t

> If you alias #f, (), and nil together from an elisp perspective, then
> when you need to differentiate these values, you can't, because you've
> just thrown away the information you'd need to do that. You can't make
> the object's true identity a hidden property of the value either,
> because that breaks perfectly valid code like this:
>
>   (defun identity-x (x) (if (eq x nil) nil x))

(eq x nil) is just something you are not expected to do if a list or
condition may come from a Scheme function.  Of course, that includes
using x for eq-based hashing.

The GUILE manual (in the above-mentioned section) states with audacity

    Fortunately, using not and null? is in good style, so all
    well-written standard Scheme programs are correct, in Guile Scheme.

Of course, that glosses over the fact that the end of a list may well be
implicitly checked for by using sentinels and other quite legitimate
programming practices.

For example, two proper lists generally have a common tail, even if that
tail is only '(), and finding that tail is done by trimming the longer
list to the size of the shorter one and then cdr-ing the remaining lists
in synch until arriving at a matching value.

Using "equal" for that comparison is not an option, and using "eq" is no
longer guaranteed to terminate without error.  The solution is to
declare the algorithm not well-written.

Cynicism aside, it will be advisable not to venture into Scheme too much
in order to avoid having the Scheme end-of-list and the Scheme #f turn
up at all.

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> Of course, that glosses over the fact that the end of a list may well
> be implicitly checked for by using sentinels and other quite
> legitimate programming practices.
>
> For example, two proper lists generally have a common tail, even if
> that tail is only '(), and finding that tail is done by trimming the
> longer list to the size of the shorter one and then cdr-ing the
> remaining lists in synch until arriving at a matching value.
>
> Using "equal" for that comparison is not an option, and using "eq" is
> no longer guaranteed to terminate without error.  The solution is to
> declare the algorithm not well-written.

So the problem is that two same-length lists are normally guaranteed to
have an `eq' element at some point in Lisps, and Guile breaks this
guarantee.  That seems like a legitimate concern; thanks for bringing it
up.

Fortunately, making use of this should be relatively rare, and the fix
is relatively simple too.

(In the algorithm you mentioned, the naive solution is to do an
additional (and (null list1-rest) (null list2-rest)) at each cdr instead
of only an (eq list1-rest list2-rest), and the smart solution to
canonicalize the null objects at the ends during the length
calculation/truncation which will inevitably visit the ends of both
lists, if the algorithm needs to be as efficient as possible.)

Considering also that the issue will only arise during inter-operation
of the languages, which should be limited to rare situations like
internal Scheme code raising an error that's handled in Elisp, the
combined chance of this issue ever hitting a pure Elisp user seems
minuscule.

All in all, it's not a concern for Emacs users who don't mess with
Scheme, and quite a small one for those who do.

Taylan

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Of course, that glosses over the fact that the end of a list may well
>> be implicitly checked for by using sentinels and other quite
>> legitimate programming practices.
>>
>> For example, two proper lists generally have a common tail, even if
>> that tail is only '(), and finding that tail is done by trimming the
>> longer list to the size of the shorter one and then cdr-ing the
>> remaining lists in synch until arriving at a matching value.
>>
>> Using "equal" for that comparison is not an option, and using "eq" is
>> no longer guaranteed to terminate without error.  The solution is to
>> declare the algorithm not well-written.
>
> So the problem is that two same-length lists are normally guaranteed to
> have an `eq' element at some point in Lisps, and Guile breaks this
> guarantee.  That seems like a legitimate concern; thanks for bringing it
> up.
>
> Fortunately, making use of this should be relatively rare, and the fix
> is relatively simple too.
>
> (In the algorithm you mentioned, the naive solution is to do an
> additional (and (null list1-rest) (null list2-rest)) at each cdr instead
> of only an (eq list1-rest list2-rest), and the smart solution to
> canonicalize the null objects at the ends during the length
> calculation/truncation which will inevitably visit the ends of both
> lists, if the algorithm needs to be as efficient as possible.)

Well, as I said: the solution is to declare the algorithm not
well-written.

> Considering also that the issue will only arise during inter-operation
> of the languages, which should be limited to rare situations like
> internal Scheme code raising an error that's handled in Elisp, the
> combined chance of this issue ever hitting a pure Elisp user seems
> minuscule.

That is under the premise that Emacs will not be extended in _any_
language except Elisp.

So that leaves us with one of two conclusions: the advertisement of
ultimately being able to extend Emacs in multiple languages possibly
including Scheme, JavaScript and Lua is not going anywhere, or existing
Elisp code will generally need to be revised in order to work in an
Emacs running extensions written in other languages.

For example, GUILE prouds itself of including an LALR(1) parser
generator running native Scheme code.  Using it for semantic parsing in
Emacs will not be feasible if its "Scheme data model" is incompatible
with the "Elisp data model".

> All in all, it's not a concern for Emacs users who don't mess with
> Scheme, and quite a small one for those who do.

But the whole point of the exercise was to enable messing with Scheme.

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> Well, as I said: the solution is to declare the algorithm not
> well-written.

Yes, this relatively rare algorithm with a trivially modified
well-written (for Guile) variant is considered not-well-written.

> That is under the premise that Emacs will not be extended in _any_
> language except Elisp.
>
> So that leaves us with one of two conclusions: the advertisement of
> ultimately being able to extend Emacs in multiple languages possibly
> including Scheme, JavaScript and Lua is not going anywhere, or existing
> Elisp code will generally need to be revised in order to work in an
> Emacs running extensions written in other languages.

Note that I myself haven't shown much enthusiasm towards the idea of
using any non-Elisp languages, other than Scheme, and I don't intend to
force Scheme on anyone either.  Nobody does.

Probably, sooner or later the amount of Scheme extensions for Emacs will
grow.  Core Emacs need not support these if they really lead to
problems, but I suspect that will be an exceedingly rare situation.
Emacs users who use others' Scheme extensions might have to deal with
issues in them, but that in turn is a problem with said extensions; an
Elisp extension can also have any number of limitations or warts imposed
on the user, by say using defadvice or failing to follow some
conventions.

> For example, GUILE prouds itself of including an LALR(1) parser
> generator running native Scheme code.  Using it for semantic parsing
> in Emacs will not be feasible if its "Scheme data model" is
> incompatible with the "Elisp data model".

I would expect it to use neither an Elisp nor a Scheme data model, but a
language-independent one served to you in the form of procedures and
macros working on said model.  Taking a quick peek into the source, that
seems to be the case indeed; it uses `define-record-type'.

And if it did use one of the usual Elisp or Scheme types (not a disjoint
type created with `define-record-type'), then they would still be
uniform between the languages as I previously mentioned: an Elisp vector
is a Scheme vector, an Elisp cons is a Scheme pair, etc.  The one and
only wart is removing the "lists all end with the same object" guarantee
as mentioned previously, which will rarely be an issue and has a trivial
fix.

> But the whole point of the exercise was to enable messing with Scheme.

No, it was not.  For Emacsers, the benefits will be, in decreasing order
of "immediate benefit": 1) make Elisp perform better, 2) make Guile
libraries available to Elisp regardless of the language they're
implemented in, 3) make it possible to extend Emacs in Scheme.

Lists are used gratiously in Emacs/Elisp, but not as central types in
specific-purpose Guile libraries, so the nil vs. '() problem mainly hits
point 3, and point 2 to a lesser extent.  (And then the whole "problem"
is barely limited to "lists not guaranteed to terminate in the same (as
in eq) object anymore," on which code relying should be exceedingly
rare.)


To repeat: I see no fundamental issues in basing Emacs/Elisp on Guile.
There are practical issues only, like how to fix string representation.
It would be great if we started working on them. :-)

Taylan

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> Guile on the other hand is GNU software; it's the official GNU
> extension language that's being massively revitalized to be worth of
> its name.  GCC, GDB, and more can be extended in Guile.  It's tied
> with Guix as well; the package manager for the upcoming official GNU
> distribution.  After so many years, things are really getting together
> in technical terms in the GNU project, and Guile is what's pulling it
> all together.

The music typesetter GNU LilyPond is one of the oldest GNU projects
using GUILE.  So far, its attempts to port from GUILE 1.8 to GUILE 2.0
have crashed and burnt (current state is inexplicable scoping problems
and crashes in random locations).  As a consequence, it will get removed
from Debian next February when Debian discontinues support of GUILE 1.8.
LilyPond's lead developer (yours truly) has been banned from posting on
the GUILE developer list years ago.  There have been few private offers
from GUILE developers to help with the porting, but after being provided
the necessary information, the contacts have gone dead.

LilyPond is the only GUILE-based GNU project that I know of that
integrates Scheme as tightly into the basic system and language as it
would be expected from Emacs, defining dozens of native data types in
C++.  And it does not even have the Elisp/Scheme discrepancy to deal
with.

It is an actively maintained project with dozens of developers and
thousands of users and will disappear from Debian-based distributions
next year.

If Emacs developers would happen to fall from grace and/or the GUILE
developers get tired of dealing with the expectable host of problems and
just stop answering calls because of the initial enthusiasm wearing down
under a lack of man power, I don't see what would promise a
substantially different ending there.

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> The music typesetter GNU LilyPond is one of the oldest GNU projects
> using GUILE.  So far, its attempts to port from GUILE 1.8 to GUILE 2.0
> have crashed and burnt (current state is inexplicable scoping problems
> and crashes in random locations).  As a consequence, it will get removed
> from Debian next February when Debian discontinues support of GUILE 1.8.
> LilyPond's lead developer (yours truly) has been banned from posting on
> the GUILE developer list years ago.  There have been few private offers
> from GUILE developers to help with the porting, but after being provided
> the necessary information, the contacts have gone dead.
>
> LilyPond is the only GUILE-based GNU project that I know of that
> integrates Scheme as tightly into the basic system and language as it
> would be expected from Emacs, defining dozens of native data types in
> C++.  And it does not even have the Elisp/Scheme discrepancy to deal
> with.
>
> It is an actively maintained project with dozens of developers and
> thousands of users and will disappear from Debian-based distributions
> next year.
>
> If Emacs developers would happen to fall from grace and/or the GUILE
> developers get tired of dealing with the expectable host of problems and
> just stop answering calls because of the initial enthusiasm wearing down
> under a lack of man power, I don't see what would promise a
> substantially different ending there.

It's unfortunate that LilyPond was hit by the 2.0 overhaul so badly.

Fortunately, such an overhaul is very unlikely to ever happen again.

And even in the fringe case that it does, it would not affect Elisp as
much (or at all) in the way the 2.0 overhaul affected Scheme.

Taylan

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> The music typesetter GNU LilyPond is one of the oldest GNU projects
>> using GUILE.  So far, its attempts to port from GUILE 1.8 to GUILE 2.0
>> have crashed and burnt (current state is inexplicable scoping problems
>> and crashes in random locations).  As a consequence, it will get removed
>> from Debian next February when Debian discontinues support of GUILE 1.8.
>> LilyPond's lead developer (yours truly) has been banned from posting on
>> the GUILE developer list years ago.  There have been few private offers
>> from GUILE developers to help with the porting, but after being provided
>> the necessary information, the contacts have gone dead.
>>
>> LilyPond is the only GUILE-based GNU project that I know of that
>> integrates Scheme as tightly into the basic system and language as it
>> would be expected from Emacs, defining dozens of native data types in
>> C++.  And it does not even have the Elisp/Scheme discrepancy to deal
>> with.
>>
>> It is an actively maintained project with dozens of developers and
>> thousands of users and will disappear from Debian-based distributions
>> next year.
>>
>> If Emacs developers would happen to fall from grace and/or the GUILE
>> developers get tired of dealing with the expectable host of problems and
>> just stop answering calls because of the initial enthusiasm wearing down
>> under a lack of man power, I don't see what would promise a
>> substantially different ending there.
>
> It's unfortunate that LilyPond was hit by the 2.0 overhaul so badly.
>
> Fortunately, such an overhaul is very unlikely to ever happen again.

I think you are misunderstanding the problem.  The problem is not that
technical problems occured after large changes.  The problem is that
GUILE developers cannot be bothered with the fallout in affected
projects.

LilyPond is not a fringe project.  It is by far the largest GNU project
using GUILE.

> And even in the fringe case that it does, it would not affect Elisp as
> much (or at all) in the way the 2.0 overhaul affected Scheme.

GUILE development will cater better for Elisp than for Scheme?  That
does sound peculiar.  Is this pitch being made anywhere else apart from
the Emacs developer list?

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> I think you are misunderstanding the problem.  The problem is not that
> technical problems occured after large changes.  The problem is that
> GUILE developers cannot be bothered with the fallout in affected
> projects.

I don't mean to insult but I have the impression that, what the Guile
developers could not be bothered with might have been the LilyPond
maintainer rather than LilyPond itself.

> GUILE development will cater better for Elisp than for Scheme?  That
> does sound peculiar.  Is this pitch being made anywhere else apart from
> the Emacs developer list?

Guile has authority over Guile-Scheme, but not over Elisp; it has to and
will support Elisp as defined by Emacs as much as possible.  That's
pretty obvious I'd say.

If the Emacs/Guile merge became fully complete in several years and all
remnants of non-Guile Emacs disappeared, then it could be vaguely
imaginable that Guile took control over Elisp semantics, but that "ain't
gonna happen" under such Emacs community pressure. :-)  You can bet on
Guile taking Elisp only to where Elispers want it to go.

Taylan

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> I think you are misunderstanding the problem.  The problem is not that
>> technical problems occured after large changes.  The problem is that
>> GUILE developers cannot be bothered with the fallout in affected
>> projects.
>
> I don't mean to insult but I have the impression that, what the Guile
> developers could not be bothered with might have been the LilyPond
> maintainer rather than LilyPond itself.

Shrug.  Naturally, after Andy Wingo prohibited me from bothering the
Guile developer list any more, the number of contacts went down.  I sent
several major developers who had _after_ the interdictum offered to help
the respective information about the state of the GUILE 2.0 port and how
to check out and work with LilyPond.  None of them ever got back to me.

At any rate, it has been mentioned previously in this discussion that
the Emacs developer list is not always considered a happy-go-lucky
environment.  So if it is the habit of GUILE developers to take out
revenge on a project and its users for enmity with single developers,
that's also relevant for making crucial GNU software depend on GUILE.

I think this interpretation of events is not making for a much better
outlook.  Particularly because Emacs development is more often than
other GNU projects governed by unpopular political decisions.  A habit
of retaliation and "see-where-this-will-get-you" in order to pressure
for a change in project lead is not likely going to work out well.

I don't think that you are better off selling this situation as a
personal vendetta, and it is not like the GUILE 2 problem was not
already there when I started to get involved with LilyPond.

>> GUILE development will cater better for Elisp than for Scheme?  That
>> does sound peculiar.  Is this pitch being made anywhere else apart
>> from the Emacs developer list?
>
> Guile has authority over Guile-Scheme, but not over Elisp; it has to
> and will support Elisp as defined by Emacs as much as possible.
> That's pretty obvious I'd say.

"This is not possible" will be defined under the constraints of GUILE
remaining Scheme according to GUILE's vision of interpretating the
Scheme standard and its further evolution.

> If the Emacs/Guile merge became fully complete in several years and
> all remnants of non-Guile Emacs disappeared, then it could be vaguely
> imaginable that Guile took control over Elisp semantics, but that
> "ain't gonna happen" under such Emacs community pressure. :-) You can
> bet on Guile taking Elisp only to where Elispers want it to go.

The question now is not where GUILE will take Elisp but rather whether
its interpretation of Elisp can get close enough to make a switch
feasible in the first place.

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> At any rate, it has been mentioned previously in this discussion that
> the Emacs developer list is not always considered a happy-go-lucky
> environment.  So if it is the habit of GUILE developers to take out
> revenge on a project and its users for enmity with single developers,
> that's also relevant for making crucial GNU software depend on GUILE.

I don't see the Guile developers taking revenge on anyone --especially
not LilyPond as a whole-- but rather flaying off your efforts to take
revenge on them by passing on your anger.

No need to be happy-go-lucky, just less insulting.

You could probably continue taking part on guile-devel if you were more
cooperative.

> I think this interpretation of events is not making for a much better
> outlook.  Particularly because Emacs development is more often than
> other GNU projects governed by unpopular political decisions.  A habit
> of retaliation and "see-where-this-will-get-you" in order to pressure
> for a change in project lead is not likely going to work out well.

It's ironic that you would talk about retaliation and
"see-where-this-will-get-you" behavior, or am I misunderstanding? :-)

It feels more like that's your attitude, despite there being no pressure
from the Guile side.

> I don't think that you are better off selling this situation as a
> personal vendetta, and it is not like the GUILE 2 problem was not
> already there when I started to get involved with LilyPond.

Let's be honest here, it was obvious from your first mail on this thread
that there's something personal going on.  I knew absolutely nothing
about your history on guile-devel (did not even recognize your name),
yet guessed immediately that there was something fishy.

>> Guile has authority over Guile-Scheme, but not over Elisp; it has to
>> and will support Elisp as defined by Emacs as much as possible.
>> That's pretty obvious I'd say.
>
> "This is not possible" will be defined under the constraints of GUILE
> remaining Scheme according to GUILE's vision of interpretating the
> Scheme standard and its further evolution.

Reminder that Guile-Emacs, in its current alpha state, already runs ERC,
Gnus, rcirc, Dired, term, comint, TRAMP, c-mode, etc.  There will
definitely not be much that is impossible.

> The question now is not where GUILE will take Elisp but rather whether
> its interpretation of Elisp can get close enough to make a switch
> feasible in the first place.

See above.

Also, compilation, not interpretation. :}

Taylan

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> At any rate, it has been mentioned previously in this discussion that
>> the Emacs developer list is not always considered a happy-go-lucky
>> environment.  So if it is the habit of GUILE developers to take out
>> revenge on a project and its users for enmity with single developers,
>> that's also relevant for making crucial GNU software depend on GUILE.
>
> I don't see the Guile developers taking revenge on anyone --especially
> not LilyPond as a whole-- but rather flaying off your efforts to take
> revenge on them by passing on your anger.

It was you who stated that you considered it likely that LilyPond was
not given any support of GUILE developers because of me being a persona
non grata.  At some point of time you have to decide whether you mean
what you say or not.

> No need to be happy-go-lucky, just less insulting.
>
> You could probably continue taking part on guile-devel if you were more
> cooperative.

Since I am unconditionally not permitted to post there, that is
hypothetical.

>> I think this interpretation of events is not making for a much better
>> outlook.  Particularly because Emacs development is more often than
>> other GNU projects governed by unpopular political decisions.  A
>> habit of retaliation and "see-where-this-will-get-you" in order to
>> pressure for a change in project lead is not likely going to work out
>> well.
>
> It's ironic that you would talk about retaliation and
> "see-where-this-will-get-you" behavior, or am I misunderstanding? :-)
>
> It feels more like that's your attitude, despite there being no
> pressure from the Guile side.

As there will be "no pressure" from the GUILE side once they decide
Emacs becomes too inconvient to support.

>> I don't think that you are better off selling this situation as a
>> personal vendetta, and it is not like the GUILE 2 problem was not
>> already there when I started to get involved with LilyPond.
>
> Let's be honest here, it was obvious from your first mail on this
> thread that there's something personal going on.  I knew absolutely
> nothing about your history on guile-devel (did not even recognize your
> name), yet guessed immediately that there was something fishy.

Not particularly hard to guess since I pointed _out_ that there was
something fishy.

>> "This is not possible" will be defined under the constraints of GUILE
>> remaining Scheme according to GUILE's vision of interpretating the
>> Scheme standard and its further evolution.
>
> Reminder that Guile-Emacs, in its current alpha state, already runs
> ERC, Gnus, rcirc, Dired, term, comint, TRAMP, c-mode, etc.  There will
> definitely not be much that is impossible.

Emacs is the platform for an ecosystem supporting thousands of packages.
That requires a rather high level of compatibility.

While you have ventured to say that you consider the move to Guile-Emacs
as less disruptive as that to lexical binding, you have glossed over the
fact that lexical binding has to be _explicitly_ enabled on a
file-by-file basis, meaning that old code will run unchanged and with
dynamic bindings.

One cannot help the impression that there is a certain elasticity to the
claims of what will be possible under which conditions that make them
less than useful for long-range planning.

Brad's status report in contrast was rather to the point, and the web
page at <URL:http://www.emacswiki.org/emacs/GuileEmacsTodo> paints a
more realistic picture of the current situation as well.

At the current point of time, it definitely appears that the marketing
department should not fear being overtaken by the engineering
department, even though the latter is making solid progress.

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> It was you who stated that you considered it likely that LilyPond was
> not given any support of GUILE developers because of me being a persona
> non grata.

That's not "revenge."

> As there will be "no pressure" from the GUILE side once they decide
> Emacs becomes too inconvient to support.

"Not gonna happen." :-)

> Not particularly hard to guess since I pointed _out_ that there was
> something fishy.

Your attitude?  I must have missed that.

> Emacs is the platform for an ecosystem supporting thousands of packages.
> That requires a rather high level of compatibility.

Bug reports welcome.  (With as little insults as possible.)

> While you have ventured to say that you consider the move to Guile-Emacs
> as less disruptive as that to lexical binding, you have glossed over the
> fact that lexical binding has to be _explicitly_ enabled on a
> file-by-file basis, meaning that old code will run unchanged and with
> dynamic bindings.

Lexical binding changes language semantics, Guile-Emacs doesn't.

> One cannot help the impression that there is a certain elasticity to the
> claims of what will be possible under which conditions that make them
> less than useful for long-range planning.

Consider the following: the name of this thread is "Emacs Lisp's
future", and people talked about all sorts of changes to Elisp itself
that they would like.  And we two here are pondering whether Guile-Emacs
*might* end up causing some *minor* changes to Elisp.
(Backwards-incompatibilities rather; new features might be plenty.)

> Brad's status report in contrast was rather to the point, and the web
> page at <URL:http://www.emacswiki.org/emacs/GuileEmacsTodo> paints a
> more realistic picture of the current situation as well.

Do you see any points there that mention incompatibilities between Emacs
Elisp and Guile Elisp semantics?

> At the current point of time, it definitely appears that the marketing
> department should not fear being overtaken by the engineering
> department, even though the latter is making solid progress.

I think the marketing department you have in mind consists of me, who is
not exactly a Guile developer.  *Hangs head in shame.*  Sorry that my
enthusiasm over Guile-Emacs and a more unified GNU system have annoyed
you; no reason to accuse Guile of marketing.  Give me all the blame.

I might not reply to further mails though.

Taylan

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Brad's status report in contrast was rather to the point, and the web
>> page at <URL:http://www.emacswiki.org/emacs/GuileEmacsTodo> paints a
>> more realistic picture of the current situation as well.
>
> Do you see any points there that mention incompatibilities between
> Emacs Elisp and Guile Elisp semantics?

My fault: that's actually the Todo list (which is also, uh, diverging
just so slightly from your characterization of the current situation).
The discussion of incompatible semantics and some of the consequences is
at <URL:http://www.emacswiki.org/emacs/GuileEmacs>.

>> At the current point of time, it definitely appears that the
>> marketing department should not fear being overtaken by the
>> engineering department, even though the latter is making solid
>> progress.
>
> I think the marketing department you have in mind consists of me, who
> is not exactly a Guile developer.  *Hangs head in shame.* Sorry that
> my enthusiasm over Guile-Emacs and a more unified GNU system have
> annoyed you; no reason to accuse Guile of marketing.  Give me all the
> blame.

The problem is not one of enthusiasm, it is one of extemporizing
non-existent information and reinterpreting existing information based
on one's preconceptions.  While how people want things to be is not
completely unrelated to how they will end up, that is much more the case
when they can significantly contribute themselves to the things moving
where they want them to be.

But the main challenges here are of technical nature.  I see a real
problem regarding the #f '() nil situation.  If you take a look, say, at
<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17474>, you'll find
that thinking outside of the Scheme standard box even regarding useful
choices for unspecified behavior is quite unpopular with GUILE.

That's unfortunate for the chances of moving VM behavior and primitives
in manners consciously more friendly towards Lisp semantics.  The
priority of GUILE very much lies with Scheme, with a narrow view towards
filling functionality left open by the Scheme standard with anything
more useful than throwing errors.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com>
> Date: Wed, 17 Sep 2014 17:07:20 +0200
> Cc: Phillip Lord <phillip.lord@newcastle.ac.uk>, emacs-devel@gnu.org
> 
> - Currently strings are an exception to the first point.  I hope that
>   won't be too difficult to fix.

Since Emacs is mainly about manipulating text, this is a very
important point, and should IMO be addressed very soon, if Guile-Emacs
is not going to be a pipe dream.  Emacs has a unified infrastructure
for dealing with text in buffers and strings, whereas Guile has only
strings, and the implementation of i18n features for text is too
dependent on the locales support in the underlying libc.

Re: Emacs Lisp's future

[Stefan Monnier]

> > Interoperability between languages is *hard*.  Doing it so the result
> > is lightweight and elegant (and moderately efficient) is rarely
> > possible.
> Well, the Elisp and Scheme integration is pretty great.

These are arguably very closely related languages, so it's a kind of
"best case" scenario.  And even in that scenario, there are significant
dark corners.

Of course, you can learn to program in "Emacs Scheme" fairly quickly
if you're familiar with Scheme, but it's not quite like programming
in Scheme.  And the docstrings can't be trusted because they point you
to Elisp macros you can't use.  And ...

Now, think about what would happen when trying to interoperate with JS,
Python, younameit, ...


        Stefan

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Thinking of Emacs as an entire ecosystem, most of Emacs is already
    maintained independently from either Emacs core or GNU ELPA.

That is a shocking statement -- is it really true?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     Thinking of Emacs as an entire ecosystem, most of Emacs is already
>     maintained independently from either Emacs core or GNU ELPA.
>
> That is a shocking statement -- is it really true?

Well, I thought to check, since I wasn't sure and the answer is actually
no, it's about 50% in terms of file size. This is comparing *.el sizes.

My actual figures are emacs (52M, 1411 lisp files), elpa (16M and 231)
and MELPA (36M, 9530). I would guess that MELPA has most of the external
packages; but I could be wrong. The low number of pacages in elpa
compared to MELPA is probably a good thing, since there are a lot of
very small ones in MELPA.

Still, there is a lot of the emacs ecosystem outside of Emacs core or
ELPA, even if it's not more.

The methodology is below!

Phil




cd emacs-core (i.e. all the emacs lisp in /usr/share/emacs/24.3)
find . -name "*elc" -exec rm {} \;
find . -name "*gz" -exec gzip -d {} \;
find . -name "*el" | wc
   1411    1411   35522
du -shc *
52M	lisp
52M	total

cd elpa
find . -name "*el" | wc
    231     231    8264

 du -shc packages/
16M	packages/
16M	total


cd melpa
cd packages
rm *tar
rm *entry
cd ..
find . -name "*el" | wc
   9530    9536  398787
du -shc packages
36M	packages
36M	total

Re: Emacs Lisp's future

[Rasmus]

[-- Attachment #1: Type: text/plain, Size: 2632 bytes --]

Hi,

Richard Stallman <rms@gnu.org> writes:

>     Thinking of Emacs as an entire ecosystem, most of Emacs is already
>     maintained independently from either Emacs core or GNU ELPA.
>
> That is a shocking statement -- is it really true?

I don't know.  Probably not.  The most important modes *I use* are
part of GNU (core + GNU ELPA:.  Org, prog-modes, Gnus, Company and
AUCTeX.  Surprisingly, EMMS is not in GNU ELPA.

Using the download statistics from MELPA (attached) there's some
packages with more than 20000 downloads.  Now I'm not clear how this
metrics come about, e.g. are upgrades counted as download and what it
the time window etc.?

Downloads:
(apply '+ (mapcar 'cadr (remove-if (lambda (x) (member (car x) elpa)) melpa))) => 4060634

Popular packages not part of GNU ELPA (> 20000 downloads).  I've added
license when I know it.

    - magit (gpl3), vc on steroids for git only.
    - dash and s (both gpl3) are list and string functions.
    - autocomplete is like company in GNU ELPA.
    - flycheck is supposedly like flymode?
    - helm, pkg-info, popup (gpl3), epl.  No idea.
    - cider, lisp environment, maybe the successor of slime? 
    - smex (gpl3), ido-ubiquitous (gpl3): ido extensions
    - paredit (gpl3), smartparens: parentheses functions, useful for lisp.
    - Gh, github api functions, I guess.
    - expand-region, rainbow-delimiters, multiple-cursors, convenience
      Functions, I guess.
    - diminish, remove stuff from the modeline.

magit                | 83518 
dash                 | 78019 
auto-complete        | 65754
popup                | 59504
git-commit-mode      | 59418
s                    | 58258
flycheck             | 52859
helm                 | 50878
git-rebase-mode      | 50865
epl                  | 48113
pkg-info             | 46034
projectile           | 39443
cider                | 38332
smex                 | 36783
clojure-mode         | 36467
f                    | 36240
zenburn-theme        | 35887
markdown-mode        | 35721
expand-region        | 35027
haskell-mode         | 34747
smartparens          | 32636
ido-ubiquitous       | 31920
rainbow-delimiters   | 30480
ace-jump-mode        | 29956
paredit              | 28493
multiple-cursors     | 27497
web-mode             | 26960
solarized-theme      | 26151
gitconfig-mode       | 24642
elisp-slime-nav      | 24482
diminish             | 24190
inf-ruby             | 23904
gitignore-mode       | 23447
exec-path-from-shell | 21889
gh                   | 21598
php-mode             | 21231
yaml-mode            | 20169

Hope it helps,
Rasmus

--
I hear there's rumors on the, uh, Internets. . .

[-- Attachment #2: melpa.el --]
[-- Type: application/emacs-lisp, Size: 38836 bytes --]

Re: Emacs Lisp's future

[Phillip Lord]

Rasmus <rasmus@gmx.us> writes:

> Hi,
>
> Richard Stallman <rms@gnu.org> writes:
>
>>     Thinking of Emacs as an entire ecosystem, most of Emacs is already
>>     maintained independently from either Emacs core or GNU ELPA.
>>
>> That is a shocking statement -- is it really true?
>
> Using the download statistics from MELPA (attached) there's some
> packages with more than 20000 downloads.  Now I'm not clear how this
> metrics come about, e.g. are upgrades counted as download and what it
> the time window etc.?

Yep, upgrades will be downloads. A number of packages on there are used
for integration testing. So, I pull down dash on every checking for
several of my projects. In time, I am sure dash will become the most
downloaded package because of this.

Once Emacs switches to git, I would guess the percentage of people on
this list who do not install magit will shrink to a tiny number. My
guesses are often wrong though!

Phil

Re: Emacs Lisp's future

[Rasmus]

Hi Phillip,

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> Rasmus <rasmus@gmx.us> writes:
>
>> Hi,
>>
>> Richard Stallman <rms@gnu.org> writes:
>>
>>>     Thinking of Emacs as an entire ecosystem, most of Emacs is already
>>>     maintained independently from either Emacs core or GNU ELPA.
>>>
>>> That is a shocking statement -- is it really true?
>>
>> Using the download statistics from MELPA (attached) there's some
>> packages with more than 20000 downloads.  Now I'm not clear how this
>> metrics come about, e.g. are upgrades counted as download and what it
>> the time window etc.?
>
> Yep, upgrades will be downloads. A number of packages on there are used
> for integration testing. So, I pull down dash on every checking for
> several of my projects. In time, I am sure dash will become the most
> downloaded package because of this.

I don't "get" dash and s, but lots of people developing more
impressive things than I will ever be able to do seem happy with it so
who am I to judge?  When I write a patch, I intend it to become part
of core, so dash/s are not options.

> Once Emacs switches to git, I would guess the percentage of people on
> this list who do not install magit will shrink to a tiny number. My
> guesses are often wrong though!

Magit is nice, and probably for a git gui *the best* you can get, but
I'm still frustrated by it (how do I format-patch?!) and must often
default to the terminal.  The manual and things like staging are
wonderful though.

—Rasmus

PS: I should have included this statistics before

;; Potential ELPA  downloads
(- (apply '+ (mapcar 'cadr melpa))
   (apply '+ (mapcar 'cadr (remove-if (lambda (x) (member (car x) elpa)) melpa)))) => 196884

But this number makes little sense, as the most important components
of Emacs are already part of the Emacs tarball (dired, gnus, org,
prog-modes).  Consider Python where Emacs has a *very* good mode
build-in.  Elpy "only" has 7790 downloads (but thanks for your work
Jörgen, it's awesome!).

—Rasmus

-- 
Governments should be afraid of their people

Re: Emacs Lisp's future

[Phillip Lord]

Rasmus <rasmus@gmx.us> writes:

>> Yep, upgrades will be downloads. A number of packages on there are used
>> for integration testing. So, I pull down dash on every checking for
>> several of my projects. In time, I am sure dash will become the most
>> downloaded package because of this.
>
> I don't "get" dash and s, but lots of people developing more
> impressive things than I will ever be able to do seem happy with it so
> who am I to judge?  When I write a patch, I intend it to become part
> of core, so dash/s are not options.

I think dash just makes list operations a lot easier. So, I use this,
for instance, to nil a tree of markers.

(defun m-buffer-nil-marker (markers)
 (-map
   (lambda (marker)
     (set-marker marker nil))
   (-flatten markers)))

I could also have done

(--map (set-marker it nil) 
      (-flatten markers))
     

or even

(--tree-map
     (set-marker it nil)
   markers)


I always found list operation in emacs a bit low-level. Dash also uses a
lot names familar to people from other languages which help. I agree
about getting into core, but with package.el coming on a-pace this might
be less of a worry in future.


>
>> Once Emacs switches to git, I would guess the percentage of people on
>> this list who do not install magit will shrink to a tiny number. My
>> guesses are often wrong though!
>
> Magit is nice, and probably for a git gui *the best* you can get, but
> I'm still frustrated by it (how do I format-patch?!) and must often
> default to the terminal.  The manual and things like staging are
> wonderful though.

I drop to the command line also. The manual is good though -- I think it
says "with magit you'll probably only have to drop through to the
command line once a day". True enough.

Phil

Re: Emacs Lisp's future

[Rasmus]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> Rasmus <rasmus@gmx.us> writes:
>
>>> Yep, upgrades will be downloads. A number of packages on there are used
>>> for integration testing. So, I pull down dash on every checking for
>>> several of my projects. In time, I am sure dash will become the most
>>> downloaded package because of this.
>>
>> I don't "get" dash and s, but lots of people developing more
>> impressive things than I will ever be able to do seem happy with it so
>> who am I to judge?  When I write a patch, I intend it to become part
>> of core, so dash/s are not options.
>
> I think dash just makes list operations a lot easier. So, I use this,
> for instance, to nil a tree of markers.
>
> (defun m-buffer-nil-marker (markers)
>  (-map
>    (lambda (marker)
>      (set-marker marker nil))
>    (-flatten markers)))

Flatten in nontrivial, but assuming you got something similar there's
no difference from mapcar or mapc, no?

> I could also have done
>
> (--map (set-marker it nil) 
>       (-flatten markers))
>
> or even
>
> (--tree-map
>      (set-marker it nil)
>    markers)

That is cool, yes, but not really (much) easier, and perhaps a bit
less readable.

Anyway, we're getting off topic here.

> I always found list operation in emacs a bit low-level. Dash also uses a
> lot names familar to people from other languages which help. I agree
> about getting into core, but with package.el coming on a-pace this might
> be less of a worry in future.

But this drives good stuff into external packages.  When search-costs
are non-nil this could be bad. . .  It's bad enough that, say, AUCTeX
is not part of the Emacs tarball.

Cheers,
Rasmus

-- 
⠠⠵

Re: Emacs Lisp's future

[Stefan Monnier]

> (defun m-buffer-nil-marker (markers)
>  (-map
>    (lambda (marker)
>      (set-marker marker nil))
>    (-flatten markers)))

I must say I dislike flatten.  Here for example, it just creates a list
to iterate over some elements.

> (--tree-map
>      (set-marker it nil)
>    markers)

That's much better, for instance.


        Stefan

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If dash is a useful tool, we should incorporate it into Emacs.
Who is the author of dash?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Rasmus]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> If dash is a useful tool, we should incorporate it into Emacs.
> Who is the author of dash?

Magnar Sveen (Cc) is the author of dash.el and s.el together with
contributors.

—Rasmus

-- 
The Kids call him Billy the Saint

Re: Emacs Lisp's future

[Nic Ferrier]

Rasmus <rasmus@gmx.us> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>> If dash is a useful tool, we should incorporate it into Emacs.
>> Who is the author of dash?
>
> Magnar Sveen (Cc) is the author of dash.el and s.el together with
> contributors.

This is a very big problem. A lot of non-GNU-Emacs emacs packages (stuff
on Melpa, stuff on Marmalade) have depends to s and dash. So they can't
be integrated into Emacs until s and dash are.

My understanding is that Magnars lost control of contributors to s and
dash some time ago. Getting people to sign papers will be hard.

It's very sad.

I am torn between a much more open and distributed Emacs (which I
suspect rms won't like) and having Emacs stay guaranteed free and pretty
much consistent.

But that's what we did when we introduced packages. How else did people
think it would end?


Nic

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Nic Ferrier writes:

 > I am torn between a much more open and distributed Emacs (which I
 > suspect rms won't like)

What's not open or distributed about Emacs?  Maintaining legal
paperwork is a cost and an inconvenience, but the GPL itself legally
guarantees openness and in practice Emacs development is highly
distributed.  ELPA is only going to provide more cases where people
want to "sign papers", or to gather "papers" from their coauthors.  I
can't see this as a problem -- Emacs will acquire more copyrights than
it would have otherwise.

I suppose it's theoretically possible that the body of unassigned and
perhaps unassignable Emacs Lisp will grow faster than the body of
assigned Emacs Lisp, but I doubt it.  Even if it does, most people do
obey the rules, and the body of free software will increase.

 > and having Emacs stay guaranteed free 

There are no guarantees.  It is certainly possible that an SCO-like
attack could be made on Emacs, especially via submarine patent.  The
FSF legal policy merely makes it less likely to succeed, and provides
the FSF the option of shifting costs onto contributors if it fails to
defend the copyrights they claimed to assign.

 > But that's what we did when we introduced packages. How else did
 > people think it would end?

Huh?  Packages simply are a way of making long-standing practices
(development and distribution by satellite projects) more convenient
for end users, permitting multiple development cycles for "core"
modules if desired by the core developers, and to some extent blurring
the line between core-developer and end-user convenience.

As long as GNU ELPA exists, there is no reason to believe that Emacs
is at any more legal risk than ever.  The only legal "danger" to Emacs
created by third-party packages is that, because of the greater
convenience, people who otherwise would not have used them at all may
be exposed to "buccaneer" packages from folks who play fast and loose
with the GPL.  As long as everybody is aware that GNU ELPA protects
them from that, it's their choice.  You may not like knowing some will
make the "wrong" choice, and I know RMS doesn't like it, but what do
you propose as the alternative that is safer?  The package system
itself doesn't favor or enable GPL violations and other unfree
practices, except possibly as a side effect of more free software
being available to violate (unlike "permissive" licensing, which does
enable unfree distribution and intentionally does so).

The practical problem created by packages is (to some eyes) a blessing
in disguise.  By enabling convenient separate development and
distribution of many more Lisp packages than would otherwise exist,
separate from Emacs core, it "exports" many APIs that would otherwise
be considered internal.  Developers wishing to change these "exported"
APIs will encounter a lot more resistance, especially from package
maintainers whose development cycles differ from the core's.  But many
projects have found such discipline to be useful in encouraging
development of a community of library modules and applications.

Consider-the-source-ly y'rs,


Steve

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Nic Ferrier writes:
>
>  > I am torn between a much more open and distributed Emacs (which I
>  > suspect rms won't like)
>
> What's not open or distributed about Emacs?  Maintaining legal
> paperwork is a cost and an inconvenience, but the GPL itself legally
> guarantees openness and in practice Emacs development is highly
> distributed.  ELPA is only going to provide more cases where people
> want to "sign papers", or to gather "papers" from their coauthors.  I
> can't see this as a problem -- Emacs will acquire more copyrights than
> it would have otherwise.
>
> I suppose it's theoretically possible that the body of unassigned and
> perhaps unassignable Emacs Lisp will grow faster than the body of
> assigned Emacs Lisp, but I doubt it.  Even if it does, most people do
> obey the rules, and the body of free software will increase.
>
>  > and having Emacs stay guaranteed free 
>
> There are no guarantees.  It is certainly possible that an SCO-like
> attack could be made on Emacs, especially via submarine patent.  The
> FSF legal policy merely makes it less likely to succeed, and provides
> the FSF the option of shifting costs onto contributors if it fails to
> defend the copyrights they claimed to assign.

The cost-shifting is not likely to ever trigger since it is rather
explicitly only applicable for deliberate or at least conscious
inclusion of material from uncleared origin.

I think it's more a protection against blatant submarines.  What the
assignments do does not significantly protect against "an attack on
Emacs".  Rather they make sure that the FSF can aggressively pursue
violations of the GPL on Emacs without having to fear muddy waters
resulting in large legal costs and settlements.

Now that's my personal take of the situation.  I cannot vouch for
Richard's though: after all, the main trigger for creating the GPL _was_
a cease-and-desist call concerning the redistribution of derivatives of
Gosling's Emacs variant, causing a bunch of lost work.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > The cost-shifting is not likely to ever trigger since it is rather
 > explicitly only applicable for deliberate or at least conscious
 > inclusion of material from uncleared origin.

Not the one I signed.  The one I signed for XEmacs in 1998 makes me
unconditionally responsible for any costs incurred by the FSF in an
unsuccessful defense of my contribution from a copyright challenge.  I
have no legal protection in that event.  Just the good sense of the
FSF legal staff.

Re: Emacs Lisp's future

[Nic Ferrier]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Nic Ferrier writes:
>
>  > I am torn between a much more open and distributed Emacs (which I
>  > suspect rms won't like)
>
> What's not open or distributed about Emacs?  Maintaining legal
> paperwork is a cost and an inconvenience, but the GPL itself legally
> guarantees openness and in practice Emacs development is highly
> distributed.  ELPA is only going to provide more cases where people
> want to "sign papers", or to gather "papers" from their coauthors.  I
> can't see this as a problem -- Emacs will acquire more copyrights than
> it would have otherwise.

I think Emacs is free, not "open". Slightly different goals, I applaud
both. To me, freedom is, on the whole, more important. Other people
disagree though.


> I suppose it's theoretically possible that the body of unassigned and
> perhaps unassignable Emacs Lisp will grow faster than the body of
> assigned Emacs Lisp, but I doubt it.  Even if it does, most people do
> obey the rules, and the body of free software will increase.

I don't think anything in MELPA is not free. It's certainly not all
GPLed though.

And I think we already have the situation of anti-GPL people
contributing to MELPA.


I've agreed with RMS to require the GPL or compatible for Marmalade.


> The practical problem created by packages is (to some eyes) a blessing
> in disguise.  By enabling convenient separate development and
> distribution of many more Lisp packages than would otherwise exist,
> separate from Emacs core, it "exports" many APIs that would otherwise
> be considered internal.  

And those APIs that most Emacs users may come to rely on (I use the term
"Emacs" when really I mean "most Emacs users") may not be free. Hence
freedom is potentially threatened.

The FSF has the rules it has for a reason.

The openness people don't want those rules because it restricts
"openness".

But when you relax the rules you open yourself to attack.


I'm not trying to be histrionic. I'm just saying it's there.

There are things we could do to battle the potential problem, making it
easier to assign copyright might be one thing.


Nic

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Giving freedom to users is an explicit goal of GNU Emacs.
"Openness" is not a goal.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Rasmus]

Hi Stephen,

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Nic Ferrier writes:
>
>  > I am torn between a much more open and distributed Emacs (which I
>  > suspect rms won't like)
>
> What's not open or distributed about Emacs?  Maintaining legal
> paperwork is a cost and an inconvenience, but the GPL itself legally
> guarantees openness and in practice Emacs development is highly
> distributed.  ELPA is only going to provide more cases where people
> want to "sign papers", or to gather "papers" from their coauthors.  I
> can't see this as a problem -- Emacs will acquire more copyrights than
> it would have otherwise.
> [...]

Perhaps a more realistic issue is if something that should be in core
cannot be.

Say "next Org" (or whatever...) emerges and everybody agrees that it
should be in core.  Developers of "next Org" sign the paperwork.  But
"next Org" depends on "popular foreign library".  Now you've got an
issue.  The Emacs tarball is no longer enough and I need to maintain a
portable installation including an init file with "popular foreign
repo" to get to work on, say, foreign PCs.

It is an issue whether infant "next Org" knows it's "next Org".  If it
knows, it cannot use "popular foreign library" although it would
enhance development.  If it doesn't know it's "next Org" you may end
up in the scenario above.

At the moment 146 packages depends on s or dash and can thus not be
included in core unless rewritten.  Assuming they all GPL'ed it may
not be a loss of freedom in the short run, but it can still have
negative dynamic effects.

>  > But that's what we did when we introduced packages. How else did
>  > people think it would end?
>
> Huh?  Packages simply are a way of making long-standing practices
> (development and distribution by satellite projects) more convenient
> for end users, permitting multiple development cycles for "core"
> modules if desired by the core developers, and to some extent blurring
> the line between core-developer and end-user convenience.

Agreed.

> [...]


> The practical problem created by packages is (to some eyes) a blessing
> in disguise.  By enabling convenient separate development and
> distribution of many more Lisp packages than would otherwise exist,
> separate from Emacs core, it "exports" many APIs that would otherwise
> be considered internal.  [...].

One does not preclude the other.  Last I checked Company was hosted on
github.  Org and Gnus live in separate repos.  The

API argument goes over my head.  Isn't the only API in questions
#autoload of functions?  Or do you have Richard comment on changing
functions in mind?  If disputing the latter, it has been pretty
convenient for Org.  Nicolas (the author of ox, the org exporter) has
changed all exporter in org-core and org-contrib at once, after making
changes to ox.  That option is very convenient when implementations
have yet to reach the (theoretical) limit of efficiency — however
that's measured.

Cheers,
Rasmus

-- 
May the Force be with you

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    My understanding is that Magnars lost control of contributors to s and
    dash some time ago. Getting people to sign papers will be hard.

If some contributors don't sign papers, and their contributions are
big enough to be legally significant, we may need in some cases to
rewrite their contributions.

We should get good packages into Emacs early, to prevent this sort of
problem from developing.  And we should warn the developers that
if they don't do this, it will cause problems for our support for
their packages.

I recommend we set up a group of volunteers to look for new packages
at an early (one developer or just a few) that seem promising, and
talk with them about proceeding to get those packages into Emacs soon.

We should reject the "ecosystem" viewpoint, making it clear that Emacs
consists of what's in the Emacs and ELPA repositories; other people's
Lisp programs are not Emacs.

We should discourage package developers from depending on packages
that are not in Emacs, warning them that this would cause trouble
for us and we will be peeved at them.

To some extent, we have exacerbated the problem by recommending the
packages that are outside Emacs too eagerly.  See "Where can I get
Emacs Lisp packages that don't come with Emacs?" in the Emacs FAQ.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

RE: Emacs Lisp's future

[Drew Adams]

> Emacs consists of what's in the Emacs and ELPA repositories;
> other people's Lisp programs are not Emacs.

Nonsense.  You might as well claim they are not Lisp either!

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Once Emacs switches to git, I would guess the percentage of people on
    this list who do not install magit will shrink to a tiny number.

What is magit?  Why is it preferable to using VC?
And why is it not in Emacs?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Rasmus]

The following message is a courtesy copy of an article
that has been posted to gmane.emacs.devel as well.

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     Once Emacs switches to git, I would guess the percentage of people on
>     this list who do not install magit will shrink to a tiny number.
>
> What is magit?

In their own words

    Magit is an interface to the version control system Git,
    implemented as an Emacs extension.

    Unlike Emacs' native Version Control package which strives to
    provide a unified interface to various version control systems,
    Magit only supports Git and can therefore better take advantage of
    its native features.

It has so very nice features but it very git-centric.  For instance in
the diff-view you can stash (precommit) only a subset of changed
lines, effectively splitting your uncommited work into several patches
easily.

> Why is it preferable to using VC?

A while ago VC was not good for git.  I think I remember having issues
with committing several files in one commit and with branches way back
with VC.  There has been some recent patches for VC-git so it's
probably better now.  I haven't tested VC recently (I mostly use the
command line).

Magit has a lot of traction by now.

> And why is it not in Emacs?

The current maintainer is Jonas Bernoulli (Cc) and the original author
is Marius Vollmer.

—Rasmus

-- 
Slowly unravels in a ball of yarn and the devil collects it

Re: Emacs Lisp's future

[Eric S. Raymond]

Rasmus <rasmus@gmx.us>:
> > Why is it preferable to using VC?
> 
> A while ago VC was not good for git.  I think I remember having issues
> with committing several files in one commit and with branches way back
> with VC.

I fixed the multi-file commit issue in 2006.  If there is still some issue 
with brances, please explain so I can fix it.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Emacs Lisp's future

[Emilio Lopes]

2014-09-19 2:03 GMT+02:00 Rasmus <rasmus@gmx.us>:
> Richard Stallman <rms@gnu.org> writes:
>> What is magit?
> […]
> It has so very nice features but it very git-centric.  For instance in
> the diff-view you can stash (precommit) only a subset of changed
> lines, effectively splitting your uncommited work into several patches
> easily.

An a side remark: I've tried magit sometime ago found it sluggish.

These days I'm pretty satisfied with the integration provided by `vc'
and git itself.
It's nice to read git's docs in `woman' when you write `git commit --help'.

For editing diffs, just use `git add -e' and you can use the excellent
(and underused) `diff-mode' to manipulate the patch your heart's
content.  Alternatively one can edit the output from `vc-diff' directly
and use `commit-patch'
(http://porkrind.org/missives/commit-patch-managing-your-mess) to
commit it out of the Emacs buffer.  This approach also works for svn,
bzr etc. btw.

Here is a trivial integration of `git grep':

    (defun git-grep (command-args)
      "Use the `grep' machinery to run `git grep'."
      (interactive
       (let ((sap (thing-at-point 'symbol))
             (grep-command "git grep -n "))
         (list (read-shell-command "Run git grep (like this): "
                                   (if sap (concat grep-command sap)
grep-command)))))
      (compilation-start (concat "< /dev/null " command-args) 'grep-mode))

 Emílio

Re: Emacs Lisp's future

[Jonas Bernoulli]

>> And why is it not in Emacs?
> The current maintainer is Jonas Bernoulli (Cc)

Thanks for inviting me to the conversation.

Short answer: I agree that it would be desirable for Magit to be part of
Emacs or (less so) GNU Elpa. But I would like to delay this conversation
until after Magit 2.1.0 and Emacs 24.4 have been released, and the Emacs
repository has been migrated to Git.

Longer answer: I am to busy to tackle this now.  Also I don't think that
Magit should be added to Emacs asap, as a whole, and in its current form.

I have read the remarks about packages not having to be done before
being added to Elpa.  But currently not even a releasable version of
Magit exists.  The stable branch contains many known defects and the
development branch only very recently got to a point where a release
became imaginable.  But there are some serious bugs, and the
documentation for the numerous new and improved features is
non-existent).

Instead of adding Magit as is, I would rather add certain components of
Magit to Emacs itself (after sufficiently abstracting them) and move
certain functionality into libraries that are already part of Emacs.

Once Magit has been released I am also going write some new libraries to
be used by Magit but also VC and other packages, such as `git-handler'
(file handler for Git blobs and trees), `kit' (wrappers around Git
commands commonly used in Emacs packages), and `libgit' (the same thing
but using the yet-to-be-created ffi).  But right now I am busy getting
the development branch ready so that more than a dozen brave souls can
benefit from the work I have done this year.

  Best regards,
  Jonas

Re: Emacs Lisp's future

[Christopher Allan Webber]

Jonas Bernoulli writes:

> Instead of adding Magit as is, I would rather add certain components of
> Magit to Emacs itself (after sufficiently abstracting them) and move
> certain functionality into libraries that are already part of Emacs.

That would be great.  There are some features of magit that would be
really nice to have available for all packages by being bundled with
emacs.  The popup menu system is pretty much the best, for example.

 - Chris

Re: Emacs Lisp's future

[Phillip Lord]

Christopher Allan Webber <cwebber@dustycloud.org> writes:
>> Instead of adding Magit as is, I would rather add certain components of
>> Magit to Emacs itself (after sufficiently abstracting them) and move
>> certain functionality into libraries that are already part of Emacs.
>
> That would be great.  There are some features of magit that would be
> really nice to have available for all packages by being bundled with
> emacs.  The popup menu system is pretty much the best, for example.

It is really nice. There is a similar solution in org-mode (the export
dispatcher) and even in auctex (which is simpler but generally works out
the right thing to do for you).

All of them are text, all of the fast and all of them easy to use (but
slightly different). Having a generic library that could be reused by
different packages would be great.

Phil

Re: Emacs Lisp's future

[Stefan Monnier]

>>> Instead of adding Magit as is, I would rather add certain components of
>>> Magit to Emacs itself (after sufficiently abstracting them) and move
>>> certain functionality into libraries that are already part of Emacs.
>> That would be great.  There are some features of magit that would be
>> really nice to have available for all packages by being bundled with
>> emacs.  The popup menu system is pretty much the best, for example.
> It is really nice. There is a similar solution in org-mode (the export
> dispatcher) and even in auctex (which is simpler but generally works out
> the right thing to do for you).

FWIW, I don't use any of those things, so I don't really know what kind
of "popup-menu" you're talking about.


        Stefan

Re: Emacs Lisp's future

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Instead of adding Magit as is, I would rather add certain components of
>>>> Magit to Emacs itself (after sufficiently abstracting them) and move
>>>> certain functionality into libraries that are already part of Emacs.
>>> That would be great.  There are some features of magit that would be
>>> really nice to have available for all packages by being bundled with
>>> emacs.  The popup menu system is pretty much the best, for example.
>> It is really nice. There is a similar solution in org-mode (the export
>> dispatcher) and even in auctex (which is simpler but generally works out
>> the right thing to do for you).
>
> FWIW, I don't use any of those things, so I don't really know what kind
> of "popup-menu" you're talking about.


It's not a menu at all, but a buffer. Saying in magit, I hit "b", then I
get a buffer "popping up" which looks like this....


Switches
 -t: Set upstream configuration (--track)         -m: Merged to HEAD (--merged)
 -M: Merged to master (--merged=master)           -n: Not merged to HEAD (--no-merged)
 -N: Not merged to master (--no-merged=master)
Args
 =c: Contains (--contains=)
 =m: Merged (--merged=)
 =n: Not merged (--no-merged=)

Actions
 v: Branch manager    b: Checkout          c: Create            r: Rename
 k: Delete


If I type "-t" (to set the -t option) then "--track" changes colour to
pink, so I know that I have selected it. When I finally hit "c", git
will be run to create a new branch with "--track" selected.


Org-mode works similarly; I hit C-cC-e (org-export-dispatch) and I get 


[C-b] Body only:    Off           [C-v] Visible only:     Off
[C-s] Export scope: Buffer        [C-f] Force publishing: Off
[C-a] Async export: Off


[c] Export to iCalendar
    [f] Current file              [a] All agenda files
    [c] Combine all agenda files  

[h] Export to HTML
    [H] As HTML buffer            [h] As HTML file
    [o] As HTML file and open     

[l] Export to LaTeX
    [L] As LaTeX buffer           [l] As LaTeX file
    [p] As PDF file               [o] As PDF file and open

[t] Export to Plain Text
    [A] As ASCII buffer           [a] As ASCII file
    [L] As Latin1 buffer          [l] As Latin1 file
    [U] As UTF-8 buffer           [u] As UTF-8 file

[P] Publish
    [f] Current file              [p] Current project
    [x] Choose project            [a] All projects


[&] Export stack                  [#] Insert template
[q] Exit


At the top we have options (although these don't equate to command line
options as with magit). All the keys like [c], [h] but not f or H are
highlighted. If I hit h, then the suboptions for Export to HTML light
up. 

So C-xCehh gives me "Export as HTML file", C-x-C-eho means and open in a
browser. I don't have to remember the options, because the menu guides
me through the process.

Auctex uses a normal complete on C-cC-c except that the default
"Compile" or "View" is almost always correct.

Phil

Re: Emacs Lisp's future

[Thorsten Jolitz]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>>> Instead of adding Magit as is, I would rather add certain components of
>>> Magit to Emacs itself (after sufficiently abstracting them) and move
>>> certain functionality into libraries that are already part of Emacs.
>>
>> That would be great.  There are some features of magit that would be
>> really nice to have available for all packages by being bundled with
>> emacs.  The popup menu system is pretty much the best, for example.
>
> It is really nice. There is a similar solution in org-mode (the export
> dispatcher) and even in auctex (which is simpler but generally works out
> the right thing to do for you).
>
> All of them are text, all of the fast and all of them easy to use (but
> slightly different). Having a generic library that could be reused by
> different packages would be great.

I like these 'dispatcher popup-menus' too, they are a very nice UI
feature in cases there are a lot of choices for the user. 

I once tried to reuse them in a project, but there is generally no
separation of UI and logic, its all mixed up, so starting from scratch
seemed easier than reusing what is already there.

How would a generic UI generator look like in this case? Wouldn't it be
very complicated to prompt the user for all the dispatcher-menu-UI
properties? Where to put what on the dispatcher UI?

May be an Org table would be a good input format:

#+BEGIN_ORG
| (key . fun) |             |             |
|-------------+-------------+-------------|
|             |             |             |
| (key . fun) | (key . fun) | (key . fun) |
| (key . fun) | (key . fun) |             |
|             |             |             |
| (key . fun) | (key . fun) |             |
|-------------+-------------+-------------|
|             | (key . fun) |             |
#+END_ORG


so that the user specifies the dispatcher-grid as an Org table, with hlines
and key/function pairs, and a generic dispatcher-ui-generator function
or macro then gets the nested-list representation of that Org table as
argument:

#+BEGIN_SRC emacs-lisp :results output raw
(org-table-map-tables
 (lambda () (print (org-table-to-lisp))))
#+END_SRC

#+results:
(("(key . fun)" "" "") hline ("" "" "") ("(key . fun)" "(key . fun)"
"(key . fun)") ("(key . fun)" "(key . fun)" "") ("" "" "") ("(key
. fun)" "(key . fun)" "") hline ("" "(key . fun)" ""))


or something like this, using the org-parser (untested):

#+BEGIN_SRC emacs-lisp :results pp
(org-element-map (org-element-parse-buffer) '(table table-row) 'identity)
#+END_SRC


-- 
cheers,
Thorsten

Re: Emacs Lisp's future

[Phillip Lord]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> phillip.lord@newcastle.ac.uk (Phillip Lord) writes:
>> It is really nice. There is a similar solution in org-mode (the export
>> dispatcher) and even in auctex (which is simpler but generally works out
>> the right thing to do for you).
>>
>> All of them are text, all of the fast and all of them easy to use (but
>> slightly different). Having a generic library that could be reused by
>> different packages would be great.
>
> I like these 'dispatcher popup-menus' too, they are a very nice UI
> feature in cases there are a lot of choices for the user. 
>
> I once tried to reuse them in a project, but there is generally no
> separation of UI and logic, its all mixed up, so starting from scratch
> seemed easier than reusing what is already there.

Yes, I would agree, although I would say that starting from scratch but
with the agreement of the org and magit developers to replace their
GUIs.


> How would a generic UI generator look like in this case? Wouldn't it be
> very complicated to prompt the user for all the dispatcher-menu-UI
> properties? Where to put what on the dispatcher UI?

A tree of property lists would be my way to go. Magit is, I think,
something like this at the moment.


> May be an Org table would be a good input format:
>
>
> #+BEGIN_ORG
> | (key . fun) |             |             |
> |-------------+-------------+-------------|
> |             |             |             |
> | (key . fun) | (key . fun) | (key . fun) |
> | (key . fun) | (key . fun) |             |
> |             |             |             |
> | (key . fun) | (key . fun) |             |
> |-------------+-------------+-------------|
> |             | (key . fun) |             |
> #+END_ORG
>
>
>
> so that the user specifies the dispatcher-grid as an Org table, with hlines
> and key/function pairs, and a generic dispatcher-ui-generator function
> or macro then gets the nested-list representation of that Org table as
> argument:
>
>
> #+BEGIN_SRC emacs-lisp :results output raw
> (org-table-map-tables
>  (lambda () (print (org-table-to-lisp))))
> #+END_SRC

That would be a good addition, I think.

Phil

Re: Emacs Lisp's future

[Nic Ferrier]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> phillip.lord@newcastle.ac.uk (Phillip Lord) writes:
> May be an Org table would be a good input format:
>
> #+BEGIN_ORG
> | (key . fun) |             |             |
> |-------------+-------------+-------------|
> |             |             |             |
> | (key . fun) | (key . fun) | (key . fun) |
> | (key . fun) | (key . fun) |             |
> |             |             |             |
> | (key . fun) | (key . fun) |             |
> |-------------+-------------+-------------|
> |             | (key . fun) |             |
> #+END_ORG


You should totally do this. I'd love to play with it.


Nic

Re: Emacs Lisp's future

[Thorsten Jolitz]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> phillip.lord@newcastle.ac.uk (Phillip Lord) writes:
>> May be an Org table would be a good input format:
>>
>> #+BEGIN_ORG
>> | (key . fun) |             |             |
>> |-------------+-------------+-------------|
>> |             |             |             |
>> | (key . fun) | (key . fun) | (key . fun) |
>> | (key . fun) | (key . fun) |             |
>> |             |             |             |
>> | (key . fun) | (key . fun) |             |
>> |-------------+-------------+-------------|
>> |             | (key . fun) |             |
>> #+END_ORG
>
>
> You should totally do this. I'd love to play with it.

'discover-context-menus' look like this:

#+BEGIN_SRC emacs-lisp
(dired
 (description "DIRectory EDitor")
 (actions
  ("Navigation"
   ("DEL" "unmark backward" dired-unmark-backward)
   ("RET" "find file" dired-find-file)
   ("SPC" "next line" dired-next-line)
   ("<" "prev dirline" dired-prev-dirline)
   (">" "next dirline" dired-next-dirline)
   ("^" "up directory" dired-up-directory)
   ("j" "goto file" dired-goto-file)
   ("i" "maybe insert subdir" dired-maybe-insert-subdir)
   ("n" "next line" dired-next-line)
   ("p" "previous line" dired-previous-line)
   ("v" "view file" dired-view-file)
   ("w" "copy filename as kill" dired-copy-filename-as-kill)) ...))
#+END_SRC

and it could easily be derived form an org table:


| Navigation |  |
| (DEL . dired-find-file) |    (RET . dired-find-file)|




-- 
cheers,
Thorsten

Re: Emacs Lisp's future

[Thorsten Jolitz]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> Nic Ferrier <nferrier@ferrier.me.uk> writes:
>
>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>
>>> phillip.lord@newcastle.ac.uk (Phillip Lord) writes:
>>> May be an Org table would be a good input format:
>>>
>>> #+BEGIN_ORG
>>> | (key . fun) |             |             |
>>> |-------------+-------------+-------------|
>>> |             |             |             |
>>> | (key . fun) | (key . fun) | (key . fun) |
>>> | (key . fun) | (key . fun) |             |
>>> |             |             |             |
>>> | (key . fun) | (key . fun) |             |
>>> |-------------+-------------+-------------|
>>> |             | (key . fun) |             |
>>> #+END_ORG
>>
>>
>> You should totally do this. I'd love to play with it.
>
> 'discover-context-menus' look like this:
>
> #+BEGIN_SRC emacs-lisp
> (dired
>  (description "DIRectory EDitor")
>  (actions
>   ("Navigation"
>    ("DEL" "unmark backward" dired-unmark-backward)
>    ("RET" "find file" dired-find-file)
>    ("SPC" "next line" dired-next-line)
>    ("<" "prev dirline" dired-prev-dirline)
>    (">" "next dirline" dired-next-dirline)
>    ("^" "up directory" dired-up-directory)
>    ("j" "goto file" dired-goto-file)
>    ("i" "maybe insert subdir" dired-maybe-insert-subdir)
>    ("n" "next line" dired-next-line)
>    ("p" "previous line" dired-previous-line)
>    ("v" "view file" dired-view-file)
>    ("w" "copy filename as kill" dired-copy-filename-as-kill)) ...))
> #+END_SRC
>
> and it could easily be derived form an org table:
>
>
> | Navigation |  |
> | (DEL . dired-find-file) |    (RET . dired-find-file)|

[Ups, sorry ...  did C-c C-c in the table, but wasn't in Org-mode but in
message-mode, so sent the message accidentally]

Calling 'org-table-to-lisp on this table

| Navigation              |                         |
| DEL dired-find-file     | RET dired-find-file     |

 yields:

(("Navigation" "") ("DEL dired-find-file" "RET dired-find-file"))

Using that as arg for this (imperfect) mapping function

#+BEGIN_SRC emacs-lisp
(defun tj/org-tbl-to-context-menu (org-tbl-as-lisp)
   (mapcar
   (lambda (--row)
     (mapcar
      (lambda (--cell)
	(let ((cell-split (split-string --cell " " t)))
	  (if (eq (length cell-split) 2)
	      (list (car cell-split)
		    (mapconcat
		     'identity
		     (split-string
		      (cadr cell-split) "\\(?:dired\\)?-" t)
		     " ")
		    (intern (cadr cell-split)))
	    cell-split)))
      --row))
   org-tbl-as-lisp))
#+END_SRC

yields something (at least somehow) similar to 'discover-context-menus':

#+BEGIN_SRC emacs-lisp
((("Navigation") nil) (("DEL" "find file" dired-find-file) ("RET" "find
file" dired-find-file)))
#+END_SRC

but to make it useful, makey.el would need to consider empty tbl-cells
as well as empty tbl-rows as well as hlines when drawing the UI grid.

-- 
cheers,
Thorsten

Re: Emacs Lisp's future

[Nicolas Richard]

Thorsten Jolitz <tjolitz@gmail.com> writes:
> How would a generic UI generator look like in this case? Wouldn't it be
> very complicated to prompt the user for all the dispatcher-menu-UI
> properties? Where to put what on the dispatcher UI?

Look into the packages "makey" and "discover". The former was factored
out of magit and discover was built on top of that.

See: http://www.masteringemacs.org/articles/2013/12/21/discoverel-discover-emacs-context-menus/

I don't know how "makey" and the corresponding magit code relate
nowadays. I hope they cooperate, but I have no idea. Mickey Petersen
(cc'ed) might tell us more about the current state.

-- 
Nicolas Richard

Re: Emacs Lisp's future

[Thorsten Jolitz]

Nicolas Richard <theonewiththeevillook@yahoo.fr> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>> How would a generic UI generator look like in this case? Wouldn't it be
>> very complicated to prompt the user for all the dispatcher-menu-UI
>> properties? Where to put what on the dispatcher UI?
>
> Look into the packages "makey" and "discover". The former was factored
> out of magit and discover was built on top of that.
>
> See:
> http://www.masteringemacs.org/articles/2013/12/21/discoverel-discover-emacs-context-menus/
>
> I don't know how "makey" and the corresponding magit code relate
> nowadays. I hope they cooperate, but I have no idea. Mickey Petersen
> (cc'ed) might tell us more about the current state.

Never heard of these packages, so I just tried to use discover/makey
with dired for the first time - wow!  

Having this kind of UI for all of Emacs would be marvellous, because
being unable to remember all those keybindings and being too lazy to
look them up is probably the most limiting factor when using Emacs. With
this kind of UI everywhere, missing long-term memory and natural
lazyness could be beaten ...

-- 
cheers,
Thorsten

RE: Emacs Lisp's future

[Drew Adams]

> > http://www.masteringemacs.org/articles/2013/12/21/discoverel-
> > discover-emacs-context-menus/
> 
> Never heard of these packages, so I just tried to use discover/makey
> with dired for the first time - wow!
> 
> Having this kind of UI for all of Emacs would be marvellous, because
> being unable to remember all those keybindings and being too lazy to
> look them up is probably the most limiting factor when using Emacs.
> With this kind of UI everywhere, missing long-term memory and natural
> lazyness could be beaten ...

Yes, this kind of thing is very helpful.

FWIW - Looks like (a subset of) Icicles key completion, to me.  It
is a subset because it is done partially: only for some particular
modes or key prefixes (e.g. Dired mode, `C-x r' prefix).  Icicles
key completion works for all keys, and even for menu-bar bindings.

http://www.emacswiki.org/EmacsNewbieWithIcicles#KeyHelpAndCompletion

http://www.emacswiki.org/Icicles_-_Key_Completion

Re: Emacs Lisp's future

[Stefan Monnier]

> I have read the remarks about packages not having to be done before
> being added to Elpa.  But currently not even a releasable version of
> Magit exists.

The right way to think about the `elpa' branch in Emacs is "github for
Elisp" rather than "retirement home for mature packages".  So indeed
code doesn't need to be "ready for release".  Magit has been in use for
many years now, so it's definitely ready for some kind of release,
whether you call it a "pretest release" or a "only for daredevils"
doesn't really matter, because many people seem quite happy using it.

> Instead of adding Magit as is, I would rather add certain components of
> Magit to Emacs itself (after sufficiently abstracting them) and move
> certain functionality into libraries that are already part of Emacs.

That can be done as well.  But there's no reason that one should
preclude the other.


        Stefan

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

For popular packages to be distributed outside of Emacs and ELPA
has some bad consequences:

* We can't change them when we change some part of Emacs that they use.
That makes it hard for us to change interfaces.

* We don't have copyright assignments in them, so we can't move the code
into Emacs.

* They may say things that work against the GNU Project in other ways,
such as by recommending nonfree programs, or by recommending rivals of
GNU packages.

I think therefore that we do more to move the popular packages into
ELPA or Emacs.

Meanwhile, code in Emacs or ELPA should not use external packages.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Rasmus]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> For popular packages to be distributed outside of Emacs and ELPA
> has some bad consequences:
>
> * We can't change them when we change some part of Emacs that they use.
> That makes it hard for us to change interfaces.
>
> * We don't have copyright assignments in them, so we can't move the code
> into Emacs.
>
> * They may say things that work against the GNU Project in other ways,
> such as by recommending nonfree programs, or by recommending rivals of
> GNU packages.
>
> I think therefore that we do more to move the popular packages into
> ELPA or Emacs.

Great!

But this may be easier said than done.

* Take Magit.  According to github it has 124 contributors.  Even if the
  main contributors would agree to move it to ELPA, it may not be easy
  to establish contact to everyone.  Cider has 82, projectile 79 etc.
  The real number of assignments is probably less than the sum, though.

* For auto-complete there's already Company in ELPA (of course there can
  be two packages doing the same thing).

* ESS is already tied to the R project.

Something like integrating the two big ido modes (smex and
ido-ubiquitous) into ido and integrating desired functions from dash
and s may be nice short-run targets (but people with more intimidate
knowledge of Emacs core programming will no doubt have greater
insights on this).

–Rasmus

-- 
Lasciate ogni speranza o voi che entrate: siete nella mani di'machellaio

Re: Emacs Lisp's future

[Stefan Monnier]

> I think therefore that we do more to move the popular packages into
> ELPA or Emacs.

I've been the main driving force behind the addition of packages to GNU
ELPA so far.  I hope other people can give me a hand.  Most of the work
for that requires very little technical knowledge.


        Stefan

Adding packages to ELPA (was: Emacs Lisp's future)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 18 Sep 2014 22:03:12 -0400
> Cc: Rasmus <rasmus@gmx.us>, emacs-devel@gnu.org
> 
> I've been the main driving force behind the addition of packages to GNU
> ELPA so far.  I hope other people can give me a hand.

It might help if you could explain the rationale for that drive.  It
is entirely mysterious in my eyes, which is more disturbing as it is
clear to me that this is somehow very important to you.

In general, I think the main goals of the project should be stated
clearly and be known and understood, if not accepted.  If nothing
else, it helps to make the right decisions when writing my own code
and reviewing that of others.

TIA

Re: Adding packages to ELPA

[Rasmus]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> 
>> I've been the main driving force behind the addition of packages to GNU
>> ELPA so far.  I hope other people can give me a hand.
>
> It might help if you could explain the rationale for that drive.

I agree.  I can make up my own reasons for why it would be good, but
that's a poor substitute for 

> It is entirely mysterious in my eyes, which is more disturbing as it
> is clear to me that this is somehow very important to you.

> In general, I think the main goals of the project should be stated
> clearly and be known and understood, if not accepted.  If nothing
> else, it helps to make the right decisions when writing my own code
> and reviewing that of others.

>> Most of the work for that requires very little technical knowledge.

Can you be more specific on the details of the process?  Is there more
to it than opening an issue on the project's issue tracker?  Is there
a public list of people who have signed copyright or do you typically
just rgrep NEWS/AUTHOR file(s)?

As Eli points out, it would be good to have an "official" reasoning
spelled out, also to point to if opening issues.

Thanks,
Rasmus

-- 
Slowly unravels in a ball of yarn and the devil collects it

Re: Adding packages to ELPA

[Stefan Monnier]

>>> Most of the work for that requires very little technical knowledge.
> Can you be more specific on the details of the process?

- Identify a candidate for inclusion.  There are rather few packages
  which shouldn't be in GNU ELPA, so there's ample choice of candidates.
- Contact the author, get his copyright.
- Find contributors (typically by looking at the VCS history), contact
  them, get their copyright.
- Decide whether the package should be added as additional directory
  in the `elpa' branch, or as its own `externals/<pkg>' branch (the
  second is normally used for larger packages).
- Add the files or the branch to the `elpa' repository.
- Fix the files's copyright headers.
- Update the `externals-list' file if needed.
- Commit.
- Usually we'll then also want to give write access to the author, and
  encourage her to use the `elpa' repository as the main repository
  where development happens.

That's about it.

There are various things that need to be improved in the GNU ELPA
infrastructure as well, which mostly require hacking on scripts and
things like that:

- Fix the forward-diffs.py so it finds the package names even when Git
  decides to truncate the file's names in the diffstat part of the message.
- Fix the forward-diffs.py so it automatically finds the maintainers
  email, or automatically updates the list of maintainers.
- Fix the forward-diffs.py so it also works for the `externals/<pkg>' branches.
- Fix the update-archive scripts so it builds the `foo.info' file from
  the Texinfo sources and creates a matching `dir' file.
- Fix the update-archive scripts so it signs the packages.

> Is there a public list of people who have signed copyright or do you
> typically just rgrep NEWS/AUTHOR file(s)?

There's a list (on fencepost.gnu.org), but it's not public (it's only
accessible to maintainers of GNU packages, more or less).  If someone
needs access to the list, of course we can provide access.


        Stefan

Re: Adding packages to ELPA

[Nic Ferrier]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> There are various things that need to be improved in the GNU ELPA
> infrastructure as well, which mostly require hacking on scripts and
> things like that:
>
> - Fix the forward-diffs.py so it finds the package names even when Git
>   decides to truncate the file's names in the diffstat part of the message.
> - Fix the forward-diffs.py so it automatically finds the maintainers
>   email, or automatically updates the list of maintainers.
> - Fix the forward-diffs.py so it also works for the `externals/<pkg>' branches.
> - Fix the update-archive scripts so it builds the `foo.info' file from
>   the Texinfo sources and creates a matching `dir' file.
> - Fix the update-archive scripts so it signs the packages.
>
>> Is there a public list of people who have signed copyright or do you
>> typically just rgrep NEWS/AUTHOR file(s)?
>
> There's a list (on fencepost.gnu.org), but it's not public (it's only
> accessible to maintainers of GNU packages, more or less).  If someone
> needs access to the list, of course we can provide access.

Would you be interested in putting GNU ELPA on marmalade's code base?


Nic

Re: Adding packages to ELPA

[Stefan Monnier]

> Would you be interested in putting GNU ELPA on marmalade's code base?

I don't know what that means.


        Stefan

Re: Adding packages to ELPA

[Sam Steingold]

> * Eli Zaretskii <ryvm@tah.bet> [2014-09-19 09:26:04 +0300]:
>
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Thu, 18 Sep 2014 22:03:12 -0400
>> Cc: Rasmus <rasmus@gmx.us>, emacs-devel@gnu.org
>> 
>> I've been the main driving force behind the addition of packages to GNU
>> ELPA so far.  I hope other people can give me a hand.
>
> It might help if you could explain the rationale for that drive.  It
> is entirely mysterious in my eyes, which is more disturbing as it is
> clear to me that this is somehow very important to you.

The issue is control:

rms writes:
>>> For popular packages to be distributed outside of Emacs and ELPA
>>> has some bad consequences:

>>> * We can't change them when we change some part of Emacs that they use.
>>> That makes it hard for us to change interfaces.

>>> * We don't have copyright assignments in them, so we can't move the code
>>> into Emacs.

>>> * They may say things that work against the GNU Project in other ways,
>>> such as by recommending nonfree programs, or by recommending rivals of
>>> GNU packages.

>>> I think therefore that we do more to move the popular packages into
>>> ELPA or Emacs.


-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1265
http://www.childpsy.net/ http://openvotingconsortium.org
http://iris.org.il http://dhimmi.org http://palestinefacts.org
The force of gravity doubles when acting on a body on a couch.

Re: Adding packages to ELPA

[Eli Zaretskii]

> From: Sam Steingold <sds@gnu.org>
> Date: Fri, 19 Sep 2014 08:31:12 -0400
> 
> > * Eli Zaretskii <ryvm@tah.bet> [2014-09-19 09:26:04 +0300]:
> >
> >> From: Stefan Monnier <monnier@iro.umontreal.ca>
> >> Date: Thu, 18 Sep 2014 22:03:12 -0400
> >> Cc: Rasmus <rasmus@gmx.us>, emacs-devel@gnu.org
> >> 
> >> I've been the main driving force behind the addition of packages to GNU
> >> ELPA so far.  I hope other people can give me a hand.
> >
> > It might help if you could explain the rationale for that drive.  It
> > is entirely mysterious in my eyes, which is more disturbing as it is
> > clear to me that this is somehow very important to you.
> 
> The issue is control:

Thanks, but I'd like to hear Stefan's take on this in addition to
yours.

Re: Adding packages to ELPA

[Sam Steingold]

> * Eli Zaretskii <ryvm@tah.bet> [2014-09-19 16:08:22 +0300]:
>
>> From: Sam Steingold <sds@gnu.org>
>> 
>> > * Eli Zaretskii <ryvm@tah.bet> [2014-09-19 09:26:04 +0300]:
>> >
>> > It might help if you could explain the rationale for that drive.  It
>> > is entirely mysterious in my eyes, which is more disturbing as it is
>> > clear to me that this is somehow very important to you.
>> 
>> The issue is control:
>
> Thanks, but I'd like to hear Stefan's take on this in addition to
> yours.

This was not _my_ take, but a direct quote from rms.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1265
http://www.childpsy.net/ http://jihadwatch.org http://iris.org.il
http://dhimmi.org http://www.memritv.org http://www.dhimmitude.org
One can find Holy Grail or Higgs boson, but not the second sock.

Re: Adding packages to ELPA

[Stefan Monnier]

>>>> For popular packages to be distributed outside of Emacs and ELPA
>>>> has some bad consequences:

>>>> * We can't change them when we change some part of Emacs that they use.
>>>> That makes it hard for us to change interfaces.

>>>> * We don't have copyright assignments in them, so we can't move the code
>>>> into Emacs.

>>>> * They may say things that work against the GNU Project in other ways,
>>>> such as by recommending nonfree programs, or by recommending rivals of
>>>> GNU packages.

>>>> I think therefore that we do more to move the popular packages into
>>>> ELPA or Emacs.

Right, these are pretty much my motivations, indeed.
[ Except for the "say things against GNU" part since moving them to GNU
  ELPA is generally not an option anyway; or at least, not before the
  problem itself is solved .  ]

Another is to improve users's experience by having more packages
available right away without having to customize their
`package-archives' first.

Also, I think it's important to include packages in GNU ELPA *early*.
For some reason, many people think of GNU ELPA as a kind of collection
of packages that have been blessed by the powers after due
considerations and checks, which inevitably is linked to the idea that
these are mature packages.

So I regularly see answers like "yes, I'd be honored to put this in GNU
ELPA when it's finished, but it's still being developed" (and this is
typically for packages that have seen a couple releases announced on
gnu.emacs.sources).

Of course, the package is never "finished" and if it ever gets to be
mature enough for the author to maybe think he should submit it to GNU
ELPA, now we have to go track down all the contributors for paperwork.

So, it's important to be very proactive and to try and get packages into
GNU ELPA very early on, when the copyright status is much more simple,
so that the copyright assignments can be collected progressively as the
contributions get accepted.


        Stefan

Re: Adding packages to ELPA

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 19 Sep 2014 09:26:46 -0400
> 
> So, it's important to be very proactive and to try and get packages into
> GNU ELPA very early on, when the copyright status is much more simple,
> so that the copyright assignments can be collected progressively as the
> contributions get accepted.

Why not add them to Emacs instead?  Having them in Emacs will almost
immediately contribute to their quality, because much more experienced
eyes will look over them.  On top of that, we gain more potential
contributors to Emacs itself, because having write access and a
package in the core naturally leads to more involvement in core
development.

Re: Adding packages to ELPA

[Stefan Monnier]

>> So, it's important to be very proactive and to try and get packages into
>> GNU ELPA very early on, when the copyright status is much more simple,
>> so that the copyright assignments can be collected progressively as the
>> contributions get accepted.
> Why not add them to Emacs instead?

Because I find it doesn't scale as well.
- GNU ELPA packages can be released on their own schedule.
- we're not seriously affected by bugs in GNU ELPA packages since they
  can be fixed and released independently.
- the maintainer of `foo' might not like to have to download a 300MB
  repository to hack on her 300-lines package.

> Having them in Emacs will almost immediately contribute to their
> quality, because much more experienced eyes will look over them.

I'd hope that the same holds for elpa packages.  That's why you can type
"make" in `elpa' to compile and prepare all the packages (you just need
to add that directory to your package-directory-list).

I strongly encourage all Emacs contributors/maintainers to have
a checked out tree of `elpa' that they update alongside the `emacs'
repository and to treat it as a "secondary" lisp directory.

> On top of that, we gain more potential contributors to Emacs itself,
> because having write access and a package in the core naturally leads
> to more involvement in core development.

The access rights to `elpa' and to `emacs' are the same already.


        Stefan

Re: Adding packages to ELPA

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 19 Sep 2014 10:44:35 -0400
> 
> - GNU ELPA packages can be released on their own schedule.

Same is true if they are in the Emacs repository.

> - we're not seriously affected by bugs in GNU ELPA packages since they
>   can be fixed and released independently.

This is actually a disadvantage: it contributes to the lower quality
of their code.

> - the maintainer of `foo' might not like to have to download a 300MB
>   repository to hack on her 300-lines package.

ELPA is currently 52MB on my disk, which is not negligible (if 300MB
aren't).  Does this mean you will change your mind when ELPA hits
100MB?

> > Having them in Emacs will almost immediately contribute to their
> > quality, because much more experienced eyes will look over them.
> 
> I'd hope that the same holds for elpa packages.  That's why you can type
> "make" in `elpa' to compile and prepare all the packages (you just need
> to add that directory to your package-directory-list).

I currently don't build ELPA (and not even bother to keep it up to
date), and when I see a bug report about it, I simply move on.

So in summary, I see no important advantages in keeping packages
separate.  I do see disadvantages.

And after all that, I still don't understand why it is so important to
encourage people to bring more packages, if we won't make that part of
Emacs.

Re: Adding packages to ELPA

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> - the maintainer of `foo' might not like to have to download a 300MB
>>   repository to hack on her 300-lines package.
>
> ELPA is currently 52MB on my disk, which is not negligible (if 300MB
> aren't).  Does this mean you will change your mind when ELPA hits
> 100MB?


I am a person who is hit by this problem. Stefan pointed out to me that
I can work around this by putting my packages on a branch and just
checking that out. He's right, it's an important consideration. If it
becomes more of a problem as ELPA grows, then that is soluable as well,
in any of many ways.

Phil

Re: Adding packages to ELPA

[Eli Zaretskii]

> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  <emacs-devel@gnu.org>
> Date: Fri, 19 Sep 2014 16:41:14 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> - the maintainer of `foo' might not like to have to download a 300MB
> >>   repository to hack on her 300-lines package.
> >
> > ELPA is currently 52MB on my disk, which is not negligible (if 300MB
> > aren't).  Does this mean you will change your mind when ELPA hits
> > 100MB?
> 
> 
> I am a person who is hit by this problem.

What problem? to have 300MB more on your disk?  I'm sorry, but I have
hard time believing this could be a real problem for anybody these
days.

Re: Adding packages to ELPA

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If the ELPA repository gets inconveniently big, we could restructure
it so that packages have separate repositories.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Adding packages to ELPA

[Stefan Monnier]

>> - GNU ELPA packages can be released on their own schedule.
> Same is true if they are in the Emacs repository.

Not really.  There are always interferences one way or the other between
Emacs's release cycle and the packages's own release cycles.

>> - we're not seriously affected by bugs in GNU ELPA packages since they
>> can be fixed and released independently.
> This is actually a disadvantage: it contributes to the lower quality
> of their code.

I feel like we're already stretched pretty thin, so adding more packages
into Emacs's core would probably not improve those packages by much, and
if it does, it'd probably be to the detriment of others.

>> - the maintainer of `foo' might not like to have to download a 300MB
>> repository to hack on her 300-lines package.
> ELPA is currently 52MB on my disk, which is not negligible (if 300MB
> aren't).  Does this mean you will change your mind when ELPA hits
> 100MB?

Indeed, and that is also a problem.  For that reason, more of the newer
packages are added as branches rather than as directories in the main
branch.  This mean you don't need the whole repository to hack on them.


        Stefan

Re: Adding packages to ELPA

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 19 Sep 2014 12:34:54 -0400
> 
> >> - GNU ELPA packages can be released on their own schedule.
> > Same is true if they are in the Emacs repository.
> 
> Not really.  There are always interferences one way or the other between
> Emacs's release cycle and the packages's own release cycles.

No, I mean that a package that is part of Emacs can be used to create
a separate tarball disregarding the rest of Emacs.  After all, once
you checkout the repo, they are just file in a certain directory.

> >> - we're not seriously affected by bugs in GNU ELPA packages since they
> >> can be fixed and released independently.
> > This is actually a disadvantage: it contributes to the lower quality
> > of their code.
> 
> I feel like we're already stretched pretty thin, so adding more packages
> into Emacs's core would probably not improve those packages by much, and
> if it does, it'd probably be to the detriment of others.

Maybe you are right, but we will not know that unless we try.  I bet
your feeling of being stretched thin is not based on facts, just on
gut feelings.  I'm not really sure about that.

> >> - the maintainer of `foo' might not like to have to download a 300MB
> >> repository to hack on her 300-lines package.
> > ELPA is currently 52MB on my disk, which is not negligible (if 300MB
> > aren't).  Does this mean you will change your mind when ELPA hits
> > 100MB?
> 
> Indeed, and that is also a problem.  For that reason, more of the newer
> packages are added as branches rather than as directories in the main
> branch.  This mean you don't need the whole repository to hack on them.

I have 1.5TB of disk storage on my main development machine.  I refuse
to believe that these small figures make a difference to someone these
days.

Re: Adding packages to ELPA

[Stefan Monnier]

>> >> - GNU ELPA packages can be released on their own schedule.
>> > Same is true if they are in the Emacs repository.
>> Not really.  There are always interferences one way or the other between
>> Emacs's release cycle and the packages's own release cycles.
> No, I mean that a package that is part of Emacs can be used to create
> a separate tarball disregarding the rest of Emacs.  After all, once
> you checkout the repo, they are just file in a certain directory.

But if you want to make release 3.4 with some new changes, you first
need to commit those changes, and if the trunk is frozen you can't do
this commit so easily.

Can it be done, yes.  But having the package be completely separate makes
things a lot easier.

Another aspect of GNU ELPA is that people get to discover the new
packages as they get added (if they check the ELPA archive regularly, or
if they read gnu.emacs.sources), as opposed to having them smashed
together, drowned in the large list of changes in etc/NEWS.

> Maybe you are right, but we will not know that unless we try.  I bet
> your feeling of being stretched thin is not based on facts, just on
> gut feelings.  I'm not really sure about that.

I know *I*'m stretched really thin.

>> Indeed, and that is also a problem.  For that reason, more of the newer
>> packages are added as branches rather than as directories in the main
>> branch.  This mean you don't need the whole repository to hack on them.
> I have 1.5TB of disk storage on my main development machine.  I refuse
> to believe that these small figures make a difference to someone these
> days.

Nowadays it's very common to never use "just the source" but "the source
with its VCS metadata".  Everywhere.  So if your config uses package
Foo, you'll want to have that package's VCS metadata in your config.
That gets copied to every one of your various accounts.  Some of those
accounts may have significant quota constraints, or be on rather
smallish rented virtual machines, or on a small home router with no HDD.
Can those cases accommodate 300MB of extra stuff?  Possibly, but you may
have to choose between that and something else.

Also, if you have N such packages.  Are you going to use up 300MB * N of
disk space?  That can become more problematical, so you'll have to go
through extra hoops to try and share that redundant data, which you
wouldn't need to do if they were separate to start with.

And of course in some situations, the bandwidth requirement, or the
processing requirements (it takes much longer to go through that 300MB
of metadata) is more problematical than the disk space itself.  But the
argument is the same.


        Stefan

Re: Adding packages to ELPA

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 19 Sep 2014 12:34:54 -0400
>> 
>> >> - GNU ELPA packages can be released on their own schedule.
>> > Same is true if they are in the Emacs repository.
>> 
>> Not really.  There are always interferences one way or the other between
>> Emacs's release cycle and the packages's own release cycles.
>
> No, I mean that a package that is part of Emacs can be used to create
> a separate tarball disregarding the rest of Emacs.  After all, once
> you checkout the repo, they are just file in a certain directory.

How would the typical Emacs/ELPA user use that "separate tarball"?

Currently, to release a new version of Ada mode, I just update the elpa
repository, incrementing the version in ada-mode.el header.

Then users will see the new version when they next do 'list-packages',
and can install from there.

What would that process look like if I followed your approach?

Part of the point of ELPA is to make this process as easy as possible;
it seems a good design to me.

>> >> - we're not seriously affected by bugs in GNU ELPA packages since they
>> >> can be fixed and released independently.
>> > This is actually a disadvantage: it contributes to the lower quality
>> > of their code.
>> 
>> I feel like we're already stretched pretty thin, so adding more packages
>> into Emacs's core would probably not improve those packages by much, and
>> if it does, it'd probably be to the detriment of others.

+1

I find it _much_ easier to fix bugs in Ada mode, now that it is
somewhat independent of Emacs core.

I realize that means I'm not working on Emacs core (except when Ada mode
is directly affected by a bug/lack of feature), but I would not be anyway.

>> >> - the maintainer of `foo' might not like to have to download a 300MB
>> >> repository to hack on her 300-lines package.
>> > ELPA is currently 52MB on my disk, which is not negligible (if 300MB
>> > aren't).  Does this mean you will change your mind when ELPA hits
>> > 100MB?
>> 
>> Indeed, and that is also a problem.  For that reason, more of the newer
>> packages are added as branches rather than as directories in the main
>> branch.  This mean you don't need the whole repository to hack on them.
>
> I have 1.5TB of disk storage on my main development machine.  I refuse
> to believe that these small figures make a difference to someone these
> days.

For me, the problem with having Ada mode directly in ELPA is reviewing
the git commit log to see if anyone else edited the Ada mode code. I
ended up skipping that, and relying on ediff with my own repository (I
don't find it convenient to use the ELPA repository directly for
development; for one thing, I prefer monotone over git). 

I'm not clear on how to "put a package on an external branch" (yes, I
have read Stefan's posts on that; they are too cryptic), but perhaps I
should try that.

-- 
-- Stephe

Re: Adding packages to ELPA

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 20 Sep 2014 11:09:15 -0500
> 
> > No, I mean that a package that is part of Emacs can be used to create
> > a separate tarball disregarding the rest of Emacs.  After all, once
> > you checkout the repo, they are just file in a certain directory.
> 
> How would the typical Emacs/ELPA user use that "separate tarball"?
> 
> Currently, to release a new version of Ada mode, I just update the elpa
> repository, incrementing the version in ada-mode.el header.
> 
> Then users will see the new version when they next do 'list-packages',
> and can install from there.
> 
> What would that process look like if I followed your approach?

The same.  If you are saying that list-packages currently doesn't
support the Emacs repo, we should modify it so it does.  It's just
another repo, after all, there's nothing magic about it.

Re: Adding packages to ELPA

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Sat, 20 Sep 2014 11:09:15 -0500
>> 
>> > No, I mean that a package that is part of Emacs can be used to create
>> > a separate tarball disregarding the rest of Emacs.  After all, once
>> > you checkout the repo, they are just file in a certain directory.
>> 
>> How would the typical Emacs/ELPA user use that "separate tarball"?
>> 
>> Currently, to release a new version of Ada mode, I just update the elpa
>> repository, incrementing the version in ada-mode.el header.
>> 
>> Then users will see the new version when they next do 'list-packages',
>> and can install from there.
>> 
>> What would that process look like if I followed your approach?
>
> The same.  If you are saying that list-packages currently doesn't
> support the Emacs repo, we should modify it so it does.  It's just
> another repo, after all, there's nothing magic about it.

I see.

list-packages gets the core Emacs packages from the installed directory
on my disk; that's from the Emacs release I installed. It gets ELPA
packages (and others) from the web, as specified in `package-archives'.

I suppose it would be possible to add the Emacs git (or bzr) repository
to package-archives; it's not there by default.

However, that would be problematic when core Emacs packages are changed
to reflect other core changes that are not in the current Emacs release;
users of that package would have to be careful to not upgrade. That
_might_ be handled by the dependency mechanism; the new version of the
core package should depend on the new version of Emacs, which
list-packages should know is not available.

There is a similar problem with ELPA packages, of course; but it is
easier to provide backwards portability solutions there (these are
usually discouraged in the Emacs trunk, which I think is a good policy).

So it comes down to how closely coupled the package is to the Emacs
core; if the coupling is loose, an ELPA package makes more sense.

-- 
-- Stephe

Re: Adding packages to ELPA

[Stefan Monnier]

> For me, the problem with having Ada mode directly in ELPA is reviewing
> the git commit log to see if anyone else edited the Ada mode code.

That shouldn't be necessary: whenever a commit is made (by someone else
than you) which touches packages/ada-mode or packges/wisi, you should
receive an email with the corresponding diff.

> I'm not clear on how to "put a package on an external branch" (yes, I
> have read Stefan's posts on that; they are too cryptic), but perhaps I
> should try that.

- Push the tree you want onto a branch `externals/<pkg>' in the
  `elpa' repository.
- Remove the packages/<pkg> directory in the `master' branch.
- Edit `externals-list' in the `master' branch.

That should be mostly it.


        Stefan

Re: Adding packages to ELPA

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 19 Sep 2014 10:44:35 -0400
>> 
>> - GNU ELPA packages can be released on their own schedule.
>
> Same is true if they are in the Emacs repository.

I have been releasing bug fix versions of Ada mode 5.x once per month
for a while now; they are immediately available to ELPA package users of
any Emacs version. 

Emacs releases are once every 2 years at best.

This is obvious, so I must be misunderstanding what you are trying to
say. Perhaps we have different definitions of "release"?

-- 
-- Stephe

Re: Adding packages to ELPA

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 20 Sep 2014 11:00:45 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Stefan Monnier <monnier@iro.umontreal.ca>
> >> Cc: emacs-devel@gnu.org
> >> Date: Fri, 19 Sep 2014 10:44:35 -0400
> >> 
> >> - GNU ELPA packages can be released on their own schedule.
> >
> > Same is true if they are in the Emacs repository.
> 
> I have been releasing bug fix versions of Ada mode 5.x once per month
> for a while now; they are immediately available to ELPA package users of
> any Emacs version. 
> 
> Emacs releases are once every 2 years at best.
> 
> This is obvious, so I must be misunderstanding what you are trying to
> say. Perhaps we have different definitions of "release"?

What I meant is no one said you need to wait for an Emacs release to
release a package.  The Emacs repository is a public one, like the
ELPA repository.  So I don't see a difference, really.

Re: Adding packages to ELPA

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> This is obvious, so I must be misunderstanding what you are trying to
>> say. Perhaps we have different definitions of "release"?
>
> What I meant is no one said you need to wait for an Emacs release to
> release a package.  The Emacs repository is a public one, like the
> ELPA repository.  So I don't see a difference, really.


Unless I am misunderstanding what you mean by Emacs repo the difference
is that the emacs repo has a developer front end (bzr or git) which ELPA
has a user front end (package.el).

Phil

Re: Adding packages to ELPA

[Eli Zaretskii]

> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,  <emacs-devel@gnu.org>
> Date: Mon, 22 Sep 2014 08:53:35 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> This is obvious, so I must be misunderstanding what you are trying to
> >> say. Perhaps we have different definitions of "release"?
> >
> > What I meant is no one said you need to wait for an Emacs release to
> > release a package.  The Emacs repository is a public one, like the
> > ELPA repository.  So I don't see a difference, really.
> 
> 
> Unless I am misunderstanding what you mean by Emacs repo the difference
> is that the emacs repo has a developer front end (bzr or git) which ELPA
> has a user front end (package.el).

package.el is customizable wrt to where it looks for packages, so its
front end is not specific to ELPA.

If you mean that the Emacs repository lacks the HTTP front end such as
http://elpa.gnu.org/packages/ provides, then setting that up should be
easy.  However, I'm not holding my breath, given the unanimously
negative reaction to my suggestions.

Re: SP? {Spam?} Re: Adding packages to ELPA

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> Unless I am misunderstanding what you mean by Emacs repo the difference
>> is that the emacs repo has a developer front end (bzr or git) which ELPA
>> has a user front end (package.el).
>
> package.el is customizable wrt to where it looks for packages, so its
> front end is not specific to ELPA.
>
> If you mean that the Emacs repository lacks the HTTP front end such as
> http://elpa.gnu.org/packages/ provides, then setting that up should be
> easy.  However, I'm not holding my breath, given the unanimously
> negative reaction to my suggestions.

I think having a consistent interface for packages in core and in elpa
would be a good thing. Then the emacs core would consist of the
interpreter and enough stuff to run package.el.

So, currently, with ELPA I've just tried to install the latest version
of org. This works, but afterwards a call to (require 'org-html) works
but picks up the old version, because it's been removed from the latest.

If org were not installed in core, this wouldn't happen because
org-html would have been deleted.

Phil

Re: Adding packages to ELPA

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Why not add them to Emacs instead?

Some packages belong in Emacs, others in ELPA.  That's a practical
question, depending on what the package does, how popular it is, and
how it relates to other packages.

In general, I think packages that are libraries for other packages
should be in Emacs.

But that is orthogonal to the questions we're discussing here.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Adding packages to ELPA

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Also, I think it's important to include packages in GNU ELPA *early*.

I agree completely.

    So I regularly see answers like "yes, I'd be honored to put this in GNU
    ELPA when it's finished, but it's still being developed" (and this is
    typically for packages that have seen a couple releases announced on
    gnu.emacs.sources).

How about if you respond, "We can defer the actual move if you like,
but to make sure we CAN move it later, could you please sign papers
now, and follow our practices with other contributors"?


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]

On 09/17/2014 04:17 AM, Phillip Lord wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> First, of course we can keep on evolving Elisp on its own.  This has
>> worked OK for the last 30 years, so it's not such a terrible choice.
>> The main problems I see with that:
>> - Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
>>   noticed more often.
> 
> I've been going through some 10 year old elisp of mine recently. The
> thing that surprises me is how many times I mention performance in it. I
> rarely worry about this these days. Elisp performance as is seems rarely
> an issue.
> 
> Where I would say that there is an issue is that too much of Emacs is
> written in C. Having a faster elisp would allow moving more into lisp
> and thus having more of Emacs extensible dynamically.
> 
>> - Lack of some features, most notably FFI and concurrency.
>> - Lack of manpower.
> 
> I'd add a fourth. People who want to extend Emacs for their own purposes
> have to learn it. Having JS extensibility would be an enourmous win.

...until the next fad comes along, at which point JS becomes a
liability. Popular environments come and go. JS is hip now; Lua, Python,
Ruby, and lots of other languages see interest rise and fall according
to fashion, taste, and the Hacker News ranking algorithm. Who knows
which languages will be popular in a few years?

Emacs Lisp has been here and has kept almost complete source
compatibility for decades and has been completely immune to these fads.
Let's maintain this tradition.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Eric Brown]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> One such option is Guile-Emacs.  This presumably would give us a faster
> implementation (at least in theory, their bytecode is significantly
> more efficient), would give us an FFI, and would give us more manpower
> since we'd be benefiting from the work done on Guile.

My vote is for whatever brings emacs closer to an FFI that allows us to
e.g. generate emacs buffer content in whichever language(s) bests suits
the developer and the domain of the subject.

>
> And of course, if Guile's own manpower dries up, Emacs would be forced
> to keep supporting Guile, which is more work than supporting just Elisp.
>

I thought that Guile is the annointed extension language of GNU
projects. Perhaps if more GNU projects evolved into supporting Guile, it
would bring more critical mass and interest to bear on Guile.

I understand the reality that Emacs is developed largely by volunteer
effort. Speaking as a donor/FSF Associate Member, I believe that this
should be considered a strategic goal of FSF/GNU leadership.

A cynical reply to my comments would be "show me the code, contributions
welcome."  But without FSF/GNU's coordination and $upport, I fail to see
how Emacs will ever evolve beyond the efforts of Hemlock, Guilemacs,
Deuce, Climacs, and countless other well-intentioned projects that have
failed to take root due to scale of the effort required to make a port
of equivalent high functionality, quality, and portability.

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

Eric Brown <brown@fastmail.fm> writes:

> [...] I fail to see how Emacs will ever evolve beyond the efforts of
> Hemlock, Guilemacs, Deuce, Climacs, and countless other
> well-intentioned projects [...]

Note that Guile-Emacs runs existing Elisp as-is and is basically a
drop-in replacement.  It's still the GNU Emacs you know; it's not even
like XEmacs.  It only *adds* things to GNU Emacs.  (But currently it's
in what one could call a alpha or even pre-alpha stage.)

More information:
http://www.emacswiki.org/emacs/GuileEmacs

Taylan

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com>
> Date: Wed, 17 Sep 2014 15:33:04 +0200
> Cc: Christopher Allan Webber <cwebber@dustycloud.org>,
> 	Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> Note that Guile-Emacs runs existing Elisp as-is and is basically a
> drop-in replacement.

But that's not really true, is it?  Just look at the changes made in
the Guile-Emacs repo to core Lisp files, and you will see what I mean.

Btw, I admit I don't completely understand what the wiki says about
the problems with t and nil, but if that means you cannot say in
Guile-Emacs something like

   (if (eq foo t) ...

then there are more than 700 places in Emacs that will stop working.

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com>
>> Date: Wed, 17 Sep 2014 15:33:04 +0200
>> Cc: Christopher Allan Webber <cwebber@dustycloud.org>,
>> 	Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
>> 
>> Note that Guile-Emacs runs existing Elisp as-is and is basically a
>> drop-in replacement.
>
> But that's not really true, is it?  Just look at the changes made in
> the Guile-Emacs repo to core Lisp files, and you will see what I mean.

It's not a 100% drop-in replacement, that's true.  I'll take a peek what
changes in Elisp files it did; the only one I remember at the moment is
Gnus directly writing out Elisp bytecode somewhere.

> Btw, I admit I don't completely understand what the wiki says about
> the problems with t and nil, but if that means you cannot say in
> Guile-Emacs something like
>
>    (if (eq foo t) ...
>
> then there are more than 700 places in Emacs that will stop working.

(eq foo t) should always work.  (eq foo nil) will only work if that
`nil' came from Elisp code; if it's a Scheme #f or '() in disguise, then
you need to make that (not foo) or (null foo) (both works in both
cases); anything but (eq foo nil).

Taylan

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I thought that Guile is the annointed extension language of GNU
    projects.

Yes, it is.  Guile is the direction we should go, if at all possible.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Lars Magne Ingebrigtsen]

As Lamba The Ultimate notes in http://lambda-the-ultimate.org/node/5048,
this has been picked up by Hacker News.  I haven't read any of the
comments there myself, so it may or may not be of interest:

https://news.ycombinator.com/item?id=8328206

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Emacs Lisp's future

[Robin Templeton]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The only standard language into which Elisp can evolve, AFAICT, is
> Common Lisp.  [ Now some readers get disappointed, while some others
> become excited.  ]  There are some incompatibilities between the two
> languages, but I can imagine working them out over the years, or even
> living with them without too much trouble, such that we could use
> Common-Lisp libraries in Emacs.
>
> Of course, that's for the language side, but on the implementation side,
> I don't really know what Common-Lisp implementation we could re-use
> (both GNU implementations are dormant, so there's no manpower for us
> tap into).  Still: there are many Common-Lisp implementations out there,
> so there's probably one that could work for us.

There is: Guile-Emacs combined with an improved and extended `cl'
library, plus language extensions needed for Common Lisp compatibility,
many of which have native support in the Guile VM.

I've also considered the possibility of porting Emacs to a Common Lisp
implementation; it does come up in Lisp circles once in a while. I think
that route would entail difficulties that don't apply to Guile-Emacs,
for a few different reasons.

First and most obviously, Guile is also a GNU project, and therefore
shares certain technical and political priorities with Emacs, which
isn't necessarily a given elsewhere.

Consider the Common Lisp implementations that could work for Emacs:
those that are free software and are actively developed with large user
communities. SBCL is one such Lisp. It's a first-rate Lisp
implementation with an excellent compiler, wide adoption in the free
software Lisp community and lots of compatible libraries. But it's not a
good fit for Emacs's needs. First, it's a compiler-only implementation
with a large and complex runtime system including garbage collectors,
native-code compilers, assemblers, etc. that exists largely
independently of the host *nix environment. And as a result, it's
monstrously difficult to port to new platforms. It's also not at all
designed to be an extension language, and the current version of the
manual says: "Calling Lisp functions from C is sometimes possible, but
is extremely hackish and poorly supported as of SBCL 0.7.5."

AFAICT, similar considerations apply to Clozure CL, the only other good
candidate I know of, although it does at least support callbacks from C.

Beyond that, it's true that Common Lisp is more closely related to Elisp
than Scheme is, and Common Lisp and Elisp have a greater degree of
overlap in their semantics and standard libraries. Even so, I believe
that the Guile platform is a superior choice as a foundation for Emacs
and Emacs Lisp.

Common Lisp was the first standardized object-oriented language and was
shaped by decades of experience with traditional Lisp systems, and it
would be beneficial for Elisp to become more compatible with Common
Lisp. Nonetheless, Common Lisp tends to agglutinate logically distinct
features into components which are rather monolithic. That's not a bad
thing in most contexts, but makes things trickier for Emacs and Elisp.
Scheme, on the other hand, is famously designed according to the
principle that

  "Programming languages should be designed not by piling feature on top
   of feature, but by removing the weaknesses and restrictions that make
   additional features appear necessary,"

and this has shaped the design of Guile, due to the primacy of Scheme in
the Guile implementation and community.

A concrete example: Common Lisp and Elisp both include almost identical
facilities for special variables. Guile Scheme doesn't; what it has are
fluids, a library for thread-local dynamic bindings independent of the
core Scheme language, which have a rather different interface. So that
looks like a win for Common Lisp and a point against Guile when it comes
to dynamic binding.

However, once you take into consideration the details of the apparently
slight gap between the languages, the overlap in design begins working
against you. Elisp provides various kinds of context-sensitive
variables, notably buffer-local variables, and thousands of packages
rely on the availability of this facility. But Common Lisp doesn't give
you the tools to implement it. What Common Lisp gives you is something
that has a strong resemblance to the corresponding part of Elisp, but
which isn't actually the same thing. And because it's a fundamental part
of the language, compilers that have been tuned for 20-30 years to
support a single standardized language optimize for a specific special
variable semantics. So, if you want to support buffer-local variables,
you have two basic choices:

(1) You can modify Common Lisp: first figure out exactly how special
variables are implemented in SBCL's compiler and runtime, how to extend
them to support Elisp semantics without breaking any legal Lisp
programs, and how to permit optimization of the use of standard special
variables without breaking the behavior of code using BLVs.

(2) Or you can DIY: implement your own special-variable library that
supports BLVs. Because Common Lisp defines a fixed high-level interface
for special variables, you won't get much help from the host language.
In this case, the apparent similarity between Elisp and Common Lisp
doesn't actually lead to shared functionality, and interaction between
the two languages will be less natural than it could be.

With Guile, the situation is different. When ad hoc extensions to core
functionality are necessary, the situation no worse than with Common
Lisp, and often better because Guile is typically more modular. But ad
hoc extensions often aren't necessary at all, because Guile takes a
fundamentally different approach to language design.

In the case of buffer-local variables, Guile provides delimited
continuations, which are powerful enough to implement both dynamic
binding and BLVs as libraries, without changing the semantics of Scheme
and therefore without disrupting the operation of other programs. Using
delimited continuations and a fluids library, one can implement
context-local bindings in under a hundred lines of code. A
proof-of-concept was presented at GHM 2011; a version compatible with
Elisp BLVs would probably expand to several hundred lines.

Another example: Common Lisp defines a standard package system that may
need extensions or modifications to work well with Elisp, while Guile
provides a module system designed for Scheme but also exposes enough of
the internal functionality to build other namespace systems. Guile-Elisp
makes use of this to compile a Lisp-2 dialect without any special
support from Guile, and it could be used as the basis for an
Elisp-specific module system.

And this pattern repeats over and over in the implementation of Elisp in
the context of a standard Common Lisp system. The small differences turn
out to matter; the family reunion becomes a feud. It's definitely
possible to port Emacs to a Common Lisp implementation; with enough work
and enough time, it could be done and done well. But it won't
necessarily be simple or straightforward, and it's unlikely to be easier
than porting Emacs to the Guile platform, a project which has progressed
from running standalone games of "dunnet.el" to supporting the complete
Emacs system with less than a person-year of work.

Scheme's turn towards minimalism may seem to be something of only
academic appeal, but it affords a freedom of expression that turns out
to be extremely practical. Guile gives you access to the fundamental
abstractions needed to express entire classes of high-level features,
and this makes it uniquely qualified to serve as a platform for multiple
Lisp dialects.

Robin
-- 
Inteligenta persono lernas la lingvon Esperanton rapide kaj facile.
Esperanto estas moderna, kultura lingvo por la mondo. Simpla, fleksebla,
belsona, Esperanto estas la praktika solvo de la problemo de universala
interkompreno. Lernu la interlingvon Esperanton!

Re: Emacs Lisp's future

[Lars Brinkhoff]

Now that we have new maintainers, I'd be curious to hear if they have
any thoughs on this?  (See the original post for more information.)

Stefan Monnier wrote two years ago:
> I've had the opportunity to think a bit more about Emacs Lisp and its
> possible evolution and I'm still not sure what to think about it.
> [...]
> I think that ideally, we'd want to stick to Elisp, or some evolution
> thereof.  Sadly, I don't see how to evolve Elisp into Scheme: they are
> closely related languages, but the differences are large enough that
> it seems hard to reconcile them.
>
> The only standard language into which Elisp can evolve, AFAICT, is
> Common Lisp.  [ Now some readers get disappointed, while some others
> become excited.  ]  There are some incompatibilities between the two
> languages, but I can imagine working them out over the years, or even
> living with them without too much trouble, such that we could use
> Common-Lisp libraries in Emacs.

Re: Emacs Lisp's future

[Nikolay Kudryavtsev]

Is there even a reason to try evolving Elisp into something else? 
Instead, you can cherry-pick the features you want. With cl-lib you get 
almost CL. With dash.el you get your "new cool lisp". And so on.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Emacs Lisp's future

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

> Is there even a reason to try evolving Elisp into something else? Instead,
> you can cherry-pick the features you want. With cl-lib you get almost CL.
> With dash.el you get your "new cool lisp". And so on.


The big benefit I see is being able to replace the lisp engine with a more
standard one, and thus benefit from performance enhancements / bugfixes /
etc from the other engine.

Philippe

[-- Attachment #2: Type: text/html, Size: 651 bytes --]

Re: Emacs Lisp's future

[Nikolay Kudryavtsev]

And that project already exists as Guile Emacs. But here we jump into a 
different territory - the current runtime is relatively well tested, 
polished and stable among all platforms. While Guile is not and I'm not 
sure if there is a free software lisp runtime that is. So, apart from 
being a lot of work just to migrate, for which there is no manpower,  we 
would also introduce a whole slew of new bugs.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Emacs Lisp's future

[Ben]

[-- Attachment #1: Type: text/plain, Size: 700 bytes --]

Why is it that nobody mentions Racket when talking about lisp or scheme?
IMHO it's quite an practical and elegant dialect of lisp/scheme.

On Fri, Oct 7, 2016 at 7:48 PM, Nikolay Kudryavtsev <
nikolay.kudryavtsev@gmail.com> wrote:

> And that project already exists as Guile Emacs. But here we jump into a
> different territory - the current runtime is relatively well tested,
> polished and stable among all platforms. While Guile is not and I'm not
> sure if there is a free software lisp runtime that is. So, apart from being
> a lot of work just to migrate, for which there is no manpower,  we would
> also introduce a whole slew of new bugs.
>
>
> --
> Best Regards,
> Nikolay Kudryavtsev
>
>
>

[-- Attachment #2: Type: text/html, Size: 1073 bytes --]

Re: Emacs Lisp's future

[Stefan Monnier]

> Why is it that nobody mentions Racket when talking about lisp or scheme?

I guess because we still need to run existing Elisp.

The assumption being that Guile can run existing Elisp, and that it
wouldn't be too hard to make an existing Common-Lisp system run existing
Elisp as well.


        Stefan

Re: Emacs Lisp's future

[Stefan Monnier]

> The big benefit I see is being able to replace the lisp engine with a more
> standard one, and thus benefit from performance enhancements / bugfixes /
> etc from the other engine.

That's right.  Any change would be motivated by the desire to benefit
from other people's efforts w.r.t implementation of the language
and of its libraries.


        Stefan

Re: Emacs Lisp's future

[Helmut Eller]

On Fri, Oct 07 2016, Nikolay Kudryavtsev wrote:

> Is there even a reason to try evolving Elisp into something else?

A major pain point of Emacs Lisp is the lack of module system or other
namespace mechanism.  The prefixes all over the place make Elisp code
quite verbose and ugly.  Evolving Elisp into a language with a module
system would be a "good thing".

Helmut

Re: Emacs Lisp's future

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 488 bytes --]

On 2016-10-08 04:23, Helmut Eller wrote:
> On Fri, Oct 07 2016, Nikolay Kudryavtsev wrote:
> 
>> Is there even a reason to try evolving Elisp into something else?
> 
> A major pain point of Emacs Lisp is the lack of module system or other
> namespace mechanism.  The prefixes all over the place make Elisp code
> quite verbose and ugly.  Evolving Elisp into a language with a module
> system would be a "good thing".

The nameless package has made this much better for me :)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 259 bytes --]

Clément Pit--Claudel <clement.pit@gmail.com> writes:

> The nameless package has made this much better for me :)

Indeed, this package is nice, but it's only a visual enhancement.
Having namespaces in Elisp would still be great IMO.

Cheers,
Nico


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Namespace systems do not fit well into Lisp.  Common Lisp's is a total
mess.  It is better to use name prefixes.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 268 bytes --]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

> Namespace systems do not fit well into Lisp.  Common Lisp's is a total
> mess.  It is better to use name prefixes.

Have you had a look at Clojure namespaces?
http://clojure.org/reference/namespaces

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Clojure namespaces (was: Re: Emacs Lisp's future)

[Chad Brown]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]


> On 8Oct, 2016, at 14:40, Nicolas Petton <nicolas@petton.fr> wrote:
> 
>> Namespace systems do not fit well into Lisp.  Common Lisp's is a total
>> mess.  It is better to use name prefixes.
> 
> Have you had a look at Clojure namespaces?
> http://clojure.org/reference/namespaces <http://clojure.org/reference/namespaces>
Thanks for the ref!

Did anyone else find it odd that Clojure’s namespace-related functions use all three namespace affix conventions? For example, name-: (ns-unmap, ns-name), -name: (all-ns, find-ns), and neither: (import, intern). That caught my clojure-ignorant eye.

Thanks,
~Chad

[-- Attachment #2: Type: text/html, Size: 1247 bytes --]

Re: Emacs Lisp's future

[Helmut Eller]

On Sat, Oct 08 2016, Richard Stallman wrote:

> Namespace systems do not fit well into Lisp.

The designers and users of the most important Lisps obviously disagree
with you: Common Lisp, Scheme (since R6RS) and Clojure all have either
namespaces or modules.  Racket even has both.

Helmut

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Namespace systems do not fit well into Lisp.

  > The designers and users of the most important Lisps obviously disagree
  > with you:

They are entitled to their opinions but I am not impressed by them.

	      Common Lisp, Scheme (since R6RS) and Clojure all have either
  > namespaces or modules.  Racket even has both.

I implemented the Common Lisp name space facility on the Lisp machine,
and I found it was inconvenient for actual use.  It is better always
to use the full name.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The reason namespaces systems do not fit well into Lisp
is that they have to operate in 'read', in the choice of which
symbol object you get.

I looked very briefly at the Clojure namespace spec and saw that it
operates as part of evaluation, not as part of reading.  Perhaps that
explains why it is usable.  Namespaces can work reasonably well as
part of evaluation.  However, that doesn't fit Lisp.

Clojure is not Lisp.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[Helmut Eller]

On Mon, Oct 10 2016, Richard Stallman wrote:

> The reason namespaces systems do not fit well into Lisp
> is that they have to operate in 'read', in the choice of which
> symbol object you get.

I would say that namespaces manage symbols.  In contrast, modules manage
bindings.

The annoyance in Common Lisp is mostly that packages have to exist
before READ can intern symbols in a particular package.  That would be
problematic for autoload cookies or customizations in .emacs which are
commonly executed before loading the code that actually defines the
package.

The other annoyance in Common Lisp is that occasionally one gets name
conflict errors when importing a package because a symbol was
accidentally interned in the current package with READ.  People then
often get confused by the error message and quickly jump to the
conclusion that the package system must suck.  It's in fact a pilot
error and the package system just does its job of keeping the namespace
clean.

> I looked very briefly at the Clojure namespace spec and saw that it
> operates as part of evaluation, not as part of reading.  Perhaps that
> explains why it is usable.  Namespaces can work reasonably well as
> part of evaluation.  However, that doesn't fit Lisp.
>
> Clojure is not Lisp.

I guess then Scheme isn't Lisp either.

BTW, Rep, the Lisp dialect used by the Sawfish window manager which
initially looked very similar to Emacs Lisp, also eventually added a
module system.  Apparently without too much trouble.

Helmut

Re: Emacs Lisp's future

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1524 bytes --]

On 11 October 2016 at 15:47, Helmut Eller <eller.helmut@gmail.com> wrote:

> On Mon, Oct 10 2016, Richard Stallman wrote:
>
> > The reason namespaces systems do not fit well into Lisp
> > is that they have to operate in 'read', in the choice of which
> > symbol object you get.
>
> I would say that namespaces manage symbols.  In contrast, modules manage
> bindings.
>

That's an interesting way of putting it.


> The annoyance in Common Lisp is mostly that packages have to exist
> before READ can intern symbols in a particular package.  That would be
> problematic for autoload cookies or customizations in .emacs which are
> commonly executed before loading the code that actually defines the
> package.
>

That can be dealt with by transparently creating a package during READ.
Common Lisp doesn't do this, but it can be argued that it should.


> The other annoyance in Common Lisp is that occasionally one gets name
> conflict errors when importing a package because a symbol was
> accidentally interned in the current package with READ.  People then
> often get confused by the error message and quickly jump to the
> conclusion that the package system must suck.  It's in fact a pilot
> error and the package system just does its job of keeping the namespace
> clean.
>

What I do is simply never importing symbols from other packages. If I need
to refer to a symbol in a different package, I always fully qualify it.

If a package system was implemented in Emacs Lisp, it would make sense to
enforce this.

Elias

[-- Attachment #2: Type: text/html, Size: 2292 bytes --]

Re: Emacs Lisp's future

[Tom Tromey]

>>>>> "RMS" == Richard Stallman <rms@gnu.org> writes:

RMS> The reason namespaces systems do not fit well into Lisp
RMS> is that they have to operate in 'read', in the choice of which
RMS> symbol object you get.

That's pretty much what my module implementation does: it renames
symbols while reading.

https://github.com/tromey/emacs-module

This means the source code uses short names, but the result respects the
Emacs naming conventions.

Tom

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That's pretty much what my module implementation does: it renames
  > symbols while reading.

A priori that sounds bad, but maybe you've found a way of handling this
that is better than what I've seen (in Common Lisp, for instance).

  > https://github.com/tromey/emacs-module

It would take a lot of work for me to try to find the code
by looking at that directory, and then a lot more work for me
to figure out what the package does by reading the code.

Would you please post the documentation here?
-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[Tom Tromey]

RMS> It would take a lot of work for me to try to find the code
RMS> by looking at that directory, and then a lot more work for me
RMS> to figure out what the package does by reading the code.

RMS> Would you please post the documentation here?

I've appended it.

I can send the source if you want as well.  It's short (388 LOC
including comments).  The essence of it is hooking
internal-macroexpand-for-load to rewrite forms.

Tom

This is a simple module system for Emacs Lisp, inspired by [Nic
Ferrier's proposal](https://lists.gnu.org/archive/html/emacs-devel/2013-07/msg00738.html).  It allows short symbol
names in the source code but exports the names using the standard
elisp-style module prefixes.

To define a module, see `define-module`.  After a `define-module`,
some symbols in the current load file will be renamed.  In
particular, symbols coming from `defvar`, `defconst`, `defun` and
the like will be renamed -- but other symbols will not be.  This
feature is critical to keeping normal Elisp working, as the Emacs
core sometimes requires certain symbol names in APIs.

Note that a private symbol must be seen -- declared with `defvar` or
the like -- before any uses.  Otherwise the renamer will not know to
rename the use.  Normally this is not a problem but you must take care
to declare internal functions when you need mutual recursion.  You can
do this like so:

```elisp
(declare-internal-function private-function)
```

When renaming, exported symbols will be given the package prefix, and
internal symbols will use the "--" convention.

E.g., consider:

```elisp
(define-module testmodule :export (somevar))
(defvar somevar nil)
(defvar private nil)
(provide 'testmodule)
```

This defines a module named `testmodule` and two variables, an
"exported" one named `testmodule-somevar`, and a "private" one named
`testmodule--private`.

Symbols from other modules can be imported using `import-module`.
Because module boundaries are purely based on symbol naming
conventions, you can also "import" from modules that do not use
module.el.  These modules are called "implicit" modules.  Importing a
module also `require`s the feature.

```elisp
(define-module testmodule)
;; Import some symbols from M.
(import-module M :symbols (a b c))
```

Sometimes, for an implicit module, the name of the feature and the
name of the module prefix differ.  In this case you can use the
`:prefix` keyword:

```elisp
(import-module cl-macs :prefix cl)
```

A module is closed by calling `provide`.

This module system doesn't have an explicit notion of submodules.
Rather, it piggy-backs on the existing feature of `require`, that lets
a directory separator in the symbol name load a file from a
subdirectory:

```elisp
(require 'feature/subfeature)
```

### Caveats

The renaming is not perfect.  If your code uses `intern` or otherwise
reflects on symbols then you have to be aware of the transforms done
by module.el.

Another issue is that autoload cookies aren't rewritten.  This seems
reasonably hard to fix, since they can refer to any symbol and so the
cookie extraction code would have to duplicate the importing logic.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I have a feeling that problems will arise where a certain name needs
to be used in one way as a variable or function, and another way
for comparisons with eq.  I can't come up with any case that fails,
so this is just a suspicion.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[Georges Ko]

Richard Stallman <rms@gnu.org> writes:

> Namespace systems do not fit well into Lisp.  Common Lisp's is a total
> mess.  It is better to use name prefixes.

A first step could be to provide some simple prefix support for users,
not stuff that comes with Emacs.

Example:

Let's say I want to write some code in sqlited.el to edit SQLite
files. Instead of prefixing everything with sqlited- in sqlited.el, I
would add this to the top of the file:

(set-current-prefix 'sqlited-)

and "magically", everything that is defined (functions, variables,
...) in this file are prefixed with "sqlited-" from the Emacs Lisp
reader point of view: 

(defun file-header (file) ...) 

is read by the reader as:

(defun sqlited-file-header (file) ...)

When looking up for functions, variables, etc., in this file,
the reader would first search with the sqlited- prefix, then the
version without prefix.

"Global" elements (without prefix) can be defined and used through
some virtual prefix such as "emacs:":

(defvar emacs:sqlited-load-hook nil)
(defun emacs:sqlited-edit-file (...) (interactive) ...)

Or it could be defined that anything starting with the prefix means
global, like now:

(defvar sqlited-load-hook nil)
(defun sqlited-edit-file (...) (interactive) ...)

Or the user can set it explicitely:

(set-global-prefix 'g:)
...
(defvar g:sqlited-load-hook nil)
(defun g:sqlited-edit-file (...) (interactive) ...)

When accessing elements using other prefixes:

(require "sqlited-viewer")
(set-prefix-map 'v: 'sqlited-viewer-)
(v:view-file "...")

v:view-file would be read as sqlited-viewer-view-file.
-- 
 Georges Ko                     gko@gko.net                      2016-10-10

Re: Emacs Lisp's future

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 264 bytes --]

On 2016-10-10 00:09, Georges Ko wrote:
> and "magically", everything that is defined (functions, variables,
> ...) in this file are prefixed with "sqlited-" from the Emacs Lisp
> reader point of view: 

Are you familiar with the names package?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Georges Ko]

Clément Pit--Claudel <clement.pit@gmail.com> writes:


>> and "magically", everything that is defined (functions, variables,
>> ...) in this file are prefixed with "sqlited-" from the Emacs Lisp
>> reader point of view: 
>
> Are you familiar with the names package?

It looks exactly what I had in mind :). Thanks!

Georges
-- 
 Georges Ko                     gko@gko.net                      2016-10-10

Re: Emacs Lisp's future

[Marcin Borkowski]

On 2016-10-10, at 15:56, Georges Ko <gko@gko.net> wrote:

>> Are you familiar with the names package?
>
> It looks exactly what I had in mind :). Thanks!

Be sure to check out also the nameless package by the same author!

> Georges

BTW, someone (Raman?) on this thread said that namespaces save typing.
That's not exactly right; that goal is achieved with autocompletion (as
he already noticed).  What namespaces really do is twofold:

1. Help avoid collisions, and

2. (last but not least!) save on reading.

Long function names are really bad.

Elisp is not Java.

Best,

-- 
Marcin Borkowski

Re: Emacs Lisp's future

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 686 bytes --]

On 2016-10-11 00:06, Marcin Borkowski wrote:
> BTW, someone (Raman?) on this thread said that namespaces save typing.
> That's not exactly right; that goal is achieved with autocompletion (as
> he already noticed).  What namespaces really do is twofold:
> 
> 1. Help avoid collisions, and
> 
> 2. (last but not least!) save on reading.
> 
> Long function names are really bad.

I think nameless makes both of these a non-issue.  You get "import x as y" using custom prefixes (such as fl: for font-lock), and for your own code there's no typing (C-- inserts the package prefix) and no wasted space.

Bottom line: I don't see much use for proper namespaces :)

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Marcin Borkowski]

On 2016-10-11, at 06:18, Clément Pit--Claudel <clement.pit@gmail.com> wrote:

> On 2016-10-11 00:06, Marcin Borkowski wrote:
>> BTW, someone (Raman?) on this thread said that namespaces save typing.
>> That's not exactly right; that goal is achieved with autocompletion (as
>> he already noticed).  What namespaces really do is twofold:
>> 
>> 1. Help avoid collisions, and
>> 
>> 2. (last but not least!) save on reading.
>> 
>> Long function names are really bad.
>
> I think nameless makes both of these a non-issue.  You get "import x as y" using custom prefixes (such as fl: for font-lock), and for your own code there's no typing (C-- inserts the package prefix) and no wasted space.
>
> Bottom line: I don't see much use for proper namespaces :)

That's only true to some extent.  Both names and nameless packages (even
though I really appreciate them) are really prosthetics; I'd have to
check it, but how do they behave with tools like xref-find-definitions,
edebug, lispy and others?  I'm pretty sure that names won't cooperate
with them nicely (I vaguely remember trying, though I'm not sure), I'd
have to check nameless.

Best,

-- 
Marcin Borkowski

Re: Emacs Lisp's future

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1375 bytes --]

On 2016-10-11 03:16, Marcin Borkowski wrote:
> On 2016-10-11, at 06:18, Clément Pit--Claudel <clement.pit@gmail.com> wrote:
> 
>> On 2016-10-11 00:06, Marcin Borkowski wrote:
>>> BTW, someone (Raman?) on this thread said that namespaces save typing.
>>> That's not exactly right; that goal is achieved with autocompletion (as
>>> he already noticed).  What namespaces really do is twofold:
>>>
>>> 1. Help avoid collisions, and
>>>
>>> 2. (last but not least!) save on reading.
>>>
>>> Long function names are really bad.
>>
>> I think nameless makes both of these a non-issue.  You get "import x as y" using custom prefixes (such as fl: for font-lock), and for your own code there's no typing (C-- inserts the package prefix) and no wasted space.
>>
>> Bottom line: I don't see much use for proper namespaces :)
> 
> That's only true to some extent.  Both names and nameless packages (even
> though I really appreciate them) are really prosthetics; I'd have to
> check it, but how do they behave with tools like xref-find-definitions,
> edebug, lispy and others?  I'm pretty sure that names won't cooperate
> with them nicely (I vaguely remember trying, though I'm not sure), I'd
> have to check nameless.

I never used names.  Nameless works entirely fine with all of the tools you mentioned, because it only touches font-locking.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 866 bytes --]

On 11 October 2016 at 21:33, Clément Pit--Claudel <clement.pit@gmail.com>
wrote:

>
> I never used names.  Nameless works entirely fine with all of the tools
> you mentioned, because it only touches font-locking.


All of these tools have the problem that it's difficult to tell what part
of a symbol is its "namespace". How do you tell the difference between the
symbols "foo-bar" without a namespace, or "bar" with namespace "foo"?

Also, some Emacs Lisp packages have a dash in the name. How would an
automated tool know that the symbol gnu-apl-interactive-mode is actually
"interactive-mode" with the namespace "gnu-apl"?

I guess I'm just curious as to why a separate symbol isn't used? : sounds
good, and gnu-apl:interactive-mode would be much more clear. Automated
tools would also be able to make more sense out of symbol names.

Elias

[-- Attachment #2: Type: text/html, Size: 1326 bytes --]

Re: Emacs Lisp's future

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1410 bytes --]

On 2016-10-11 10:54, Elias Mårtenson wrote:
> On 11 October 2016 at 21:33, Clément Pit--Claudel
> <clement.pit@gmail.com <mailto:clement.pit@gmail.com>> wrote:
> 
>> I never used names.  Nameless works entirely fine with all of the
>> tools you mentioned, because it only touches font-locking.
> 
> All of these tools have the problem that it's difficult to tell what
> part of a symbol is its "namespace". How do you tell the difference
> between the symbols "foo-bar" without a namespace, or "bar" with
> namespace "foo"?

Indeed.  Have you had this problem in practice, though?

> Also, some Emacs Lisp packages have a dash in the name. How would an
> automated tool know that the symbol gnu-apl-interactive-mode is
> actually "interactive-mode" with the namespace "gnu-apl"?

It's generally sufficient to look at the name of the file that declares that function.  For example, gnu-apl-interactive-mode presumably lives in gnu-apl.el.  If not, then you can set a file-local variable indicating what the library's prefix is.

> I guess I'm just curious as to why a separate symbol isn't used? :
> sounds good, and gnu-apl:interactive-mode would be much more clear.
> Automated tools would also be able to make more sense out of symbol
> names.

nameless uses ':' for display (and '::' for private members), indeed.  Some packages use / (yas/, for example)

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 2401 bytes --]

On 11 October 2016 at 23:10, Clément Pit--Claudel <clement.pit@gmail.com>
wrote:

> On 2016-10-11 10:54, Elias Mårtenson wrote:
>


> > All of these tools have the problem that it's difficult to tell what
> > part of a symbol is its "namespace". How do you tell the difference
> > between the symbols "foo-bar" without a namespace, or "bar" with
> > namespace "foo"?
>
> Indeed.  Have you had this problem in practice, though?
>

That depends on how you define "having the problem". I don't use automated
tools (such as nameless) partly because I find it somewhat hackish. I have
been confused about it though.


> It's generally sufficient to look at the name of the file that declares
> that function.  For example, gnu-apl-interactive-mode presumably lives in
> gnu-apl.el.  If not, then you can set a file-local variable indicating what
> the library's prefix is.
>

It actually lives in a file called gnu-apl-interactive.el.


> > I guess I'm just curious as to why a separate symbol isn't used? :
> > sounds good, and gnu-apl:interactive-mode would be much more clear.
> > Automated tools would also be able to make more sense out of symbol
> > names.
>
> nameless uses ':' for display (and '::' for private members), indeed.
> Some packages use / (yas/, for example)


Presumably they do this because they agree with me that overloading the use
of the - character for both word separation and namespace indicator is not
optimal. The use of multiple different variations is even worse (which is
why I, even though I don't agree with the practice of using - and --,
conform to it).

I'm merely trying to put the attention on this, to gauge just how many
people agree with me. I suspect that not enough does.

I still want to summarise what I'd like to see: Use : (or some other
character other than -) to separate namespace and name. Use :: (or
something else) to indicate private symbols. Then, standardised tooling in
Emacs that would handle this convention could come later (as in something
like nameless, or perhaps an extension to the reader to handle it).

In other words: I think that as long as the character overloading problem
is fixed and a standardised method is promoted, creating tools that
implement pretty much everything people want from a namespace/package
system can be implemented on top of this convention.

Elias

[-- Attachment #2: Type: text/html, Size: 3404 bytes --]

Re: Emacs Lisp's future

[Marcin Borkowski]

On 2016-10-11, at 15:33, Clément Pit--Claudel <clement.pit@gmail.com> wrote:

>>> Bottom line: I don't see much use for proper namespaces :)
>> 
>> That's only true to some extent.  Both names and nameless packages (even
>> though I really appreciate them) are really prosthetics; I'd have to
>> check it, but how do they behave with tools like xref-find-definitions,
>> edebug, lispy and others?  I'm pretty sure that names won't cooperate
>> with them nicely (I vaguely remember trying, though I'm not sure), I'd
>> have to check nameless.
>
> I never used names.  Nameless works entirely fine with all of the tools you mentioned, because it only touches font-locking.

Cool to hear that.  I guess I'll have to start using it.

Best,

-- 
Marcin Borkowski

Re: Emacs Lisp's future

[raman]

Georges Ko <gko@gko.net> writes:


All this would buy would be less typing -- but emacs completion does
most of my typing for me anyway.

another win perhaps might be for shorter lines because of shorter
identifiers -- but for a large file, 
one might easily forget that the prefix has been set at the top --
e.g. when fixing a bug in the middle of a file.
> Richard Stallman <rms@gnu.org> writes:
>
>> Namespace systems do not fit well into Lisp.  Common Lisp's is a total
>> mess.  It is better to use name prefixes.
>
> A first step could be to provide some simple prefix support for users,
> not stuff that comes with Emacs.
>
> Example:
>
> Let's say I want to write some code in sqlited.el to edit SQLite
> files. Instead of prefixing everything with sqlited- in sqlited.el, I
> would add this to the top of the file:
>
> (set-current-prefix 'sqlited-)
>
> and "magically", everything that is defined (functions, variables,
> ...) in this file are prefixed with "sqlited-" from the Emacs Lisp
> reader point of view: 
>
> (defun file-header (file) ...) 
>
> is read by the reader as:
>
> (defun sqlited-file-header (file) ...)
>
> When looking up for functions, variables, etc., in this file,
> the reader would first search with the sqlited- prefix, then the
> version without prefix.
>
> "Global" elements (without prefix) can be defined and used through
> some virtual prefix such as "emacs:":
>
> (defvar emacs:sqlited-load-hook nil)
> (defun emacs:sqlited-edit-file (...) (interactive) ...)
>
> Or it could be defined that anything starting with the prefix means
> global, like now:
>
> (defvar sqlited-load-hook nil)
> (defun sqlited-edit-file (...) (interactive) ...)
>
> Or the user can set it explicitely:
>
> (set-global-prefix 'g:)
> ...
> (defvar g:sqlited-load-hook nil)
> (defun g:sqlited-edit-file (...) (interactive) ...)
>
> When accessing elements using other prefixes:
>
> (require "sqlited-viewer")
> (set-prefix-map 'v: 'sqlited-viewer-)
> (v:view-file "...")
>
> v:view-file would be read as sqlited-viewer-view-file.

-- 

Re: Emacs Lisp's future

[Kelvin White]

On Fri, Oct 7, 2016 at 6:47 AM, Lars Brinkhoff <lars@nocrew.org> wrote:
> Now that we have new maintainers, I'd be curious to hear if they have
> any thoughs on this?  (See the original post for more information.)
>
> Stefan Monnier wrote two years ago:
>> I've had the opportunity to think a bit more about Emacs Lisp and its
>> possible evolution and I'm still not sure what to think about it.
>> [...]
>> I think that ideally, we'd want to stick to Elisp, or some evolution
>> thereof.  Sadly, I don't see how to evolve Elisp into Scheme: they are
>> closely related languages, but the differences are large enough that
>> it seems hard to reconcile them.
>>
>> The only standard language into which Elisp can evolve, AFAICT, is
>> Common Lisp.  [ Now some readers get disappointed, while some others
>> become excited.  ]  There are some incompatibilities between the two
>> languages, but I can imagine working them out over the years, or even
>> living with them without too much trouble, such that we could use
>> Common-Lisp libraries in Emacs.
>
>

I am all for the common-lisp evolution of Emacs-lisp, and willing to
contribute. Just my two cents...

Re: Emacs Lisp's future

[Søren Pilgård]

On Fri, Oct 7, 2016 at 12:47 PM, Lars Brinkhoff <lars@nocrew.org> wrote:
> Now that we have new maintainers, I'd be curious to hear if they have
> any thoughs on this?  (See the original post for more information.)
>
> Stefan Monnier wrote two years ago:
>> I've had the opportunity to think a bit more about Emacs Lisp and its
>> possible evolution and I'm still not sure what to think about it.
>> [...]
>> I think that ideally, we'd want to stick to Elisp, or some evolution
>> thereof.  Sadly, I don't see how to evolve Elisp into Scheme: they are
>> closely related languages, but the differences are large enough that
>> it seems hard to reconcile them.
>>
>> The only standard language into which Elisp can evolve, AFAICT, is
>> Common Lisp.  [ Now some readers get disappointed, while some others
>> become excited.  ]  There are some incompatibilities between the two
>> languages, but I can imagine working them out over the years, or even
>> living with them without too much trouble, such that we could use
>> Common-Lisp libraries in Emacs.
>
>


First of all we need to decide what it really is that is wanted here.
Is it just the underlying lisp engine or what exactly?

Is it basically replacing the C parts with Common lisp code?
That does not seem to gain much.

Is it to discard Emacs lisp entirely (maybe in a longer process) and
replace it with Common lisp
This would discard decades of code and mountains of configurations for
what gain exactly?

Or is the goal to find a mapping of Emacs lisp to Common lisp.
So we can build up the core concepts in common lisp and then map
whatever Emacs code down into Common lisp?
Then people will ask next if in the future can code for Emacs directly
in Common lisp and not just in "legacy" Emacs lisp.

Keeping this distinction in mind is important, especially when talking
to people outside the development group.
A lot of people had very different ideas and interpretations of what
the Emacs-guile project intented to do and did.

Be aware that Emacs lisp is not Common lisp and such a mapping does
have some pitfalls, especially around dynamic scoping and buffer local
variables.
I remember some blogposts about this (from 2012)
http://tromey.com/blog/?p=709 http://tromey.com/blog/?p=751
http://tromey.com/blog/?p=778

Personally I could fear that this mapping will insert certain
restrictions of what can
and can't be done in Common lisp code against Emacs, especially if we open up to
everyday Emacs code in Common lisp. Basically it would be problematic if Common
lisp could violate certain assumptions the Emacs code makes, like
running "single threaded".
Maybe this could be contained by hiding it somehow, but isn't all we
get another internal implementation then?
If the developers prefer to do the internal implementation in Common
Lisp/Emacs Lisp rather than C/Emacs lisp and have
easier access to expose some libraries then that is another thing.


So when people talk about Emacs Lisp transitioning to Common Lisp,
please specify exactly what you want.

Re: Emacs Lisp's future

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 2795 bytes --]

On Fri, Oct 7, 2016 at 12:20 PM Søren Pilgård <fiskomaten@gmail.com> wrote:

> First of all we need to decide what it really is that is wanted here.
> Is it just the underlying lisp engine or what exactly?
>
> Is it basically replacing the C parts with Common lisp code?
> That does not seem to gain much.
>
> Is it to discard Emacs lisp entirely (maybe in a longer process) and
> replace it with Common lisp
> This would discard decades of code and mountains of configurations for
> what gain exactly?
>
> Or is the goal to find a mapping of Emacs lisp to Common lisp.
> So we can build up the core concepts in common lisp and then map
> whatever Emacs code down into Common lisp?
> Then people will ask next if in the future can code for Emacs directly
> in Common lisp and not just in "legacy" Emacs lisp.
>
> Keeping this distinction in mind is important, especially when talking
> to people outside the development group.
> A lot of people had very different ideas and interpretations of what
> the Emacs-guile project intented to do and did.
>
> Be aware that Emacs lisp is not Common lisp and such a mapping does
> have some pitfalls, especially around dynamic scoping and buffer local
> variables.
> I remember some blogposts about this (from 2012)
> http://tromey.com/blog/?p=709 http://tromey.com/blog/?p=751
> http://tromey.com/blog/?p=778
>
> Personally I could fear that this mapping will insert certain
> restrictions of what can
> and can't be done in Common lisp code against Emacs, especially if we open
> up to
> everyday Emacs code in Common lisp. Basically it would be problematic if
> Common
> lisp could violate certain assumptions the Emacs code makes, like
> running "single threaded".
> Maybe this could be contained by hiding it somehow, but isn't all we
> get another internal implementation then?
> If the developers prefer to do the internal implementation in Common
> Lisp/Emacs Lisp rather than C/Emacs lisp and have
> easier access to expose some libraries then that is another thing.
>
>
> So when people talk about Emacs Lisp transitioning to Common Lisp,
> please specify exactly what you want.
>


Exactly my thoughts. My fear is that this supposed transition to Common
Lisp with fragment the emacs user base.. some will stick with elisp and
some will move over to CL. Elisp is working very well for configs and
packages. We should rather stick with it and keep on growing the codebase
in the same language.

(If Elisp is completely discarded, we will not have enough people to
convert *all* existing cool packages discovered and yet to be discovered
from elisp to CL.)

So it is a lose-lose situation whether Elisp and CL co-exist or even if CL
completely replaces Elisp.
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 4404 bytes --]

Re: Emacs Lisp's future

[Tom Tromey]

>>>>> "Søren" == Søren Pilgård <fiskomaten@gmail.com> writes:

Søren> Be aware that Emacs lisp is not Common lisp and such a mapping does
Søren> have some pitfalls, especially around dynamic scoping and buffer local
Søren> variables.
Søren> I remember some blogposts about this (from 2012)
Søren> http://tromey.com/blog/?p=709 http://tromey.com/blog/?p=751
Søren> http://tromey.com/blog/?p=778

I still love this wacky idea, but I pretty much moved on to:

https://github.com/tromey/el-compilador

The basic idea here being to eventually write Emacs in Emacs Lisp.
I no longer think the language needs a radical change.  It can just
evolve in place.

Tom

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The basic idea here being to eventually write Emacs in Emacs Lisp.

It would be a lot of work, for little benefit.
Let's put our time instead into things that will really help users.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

>>>>> "LB" == Lars Brinkhoff <lars@nocrew.org> writes:

LB> Now that we have new maintainers, I'd be curious to hear if they have any
LB> thoughs on this? (See the original post for more information.)

No particular thoughts from me right now. I'm curious to see what happens with
Guile, but I'm not eager to change. It would be nice to have real compilation
of Emacs Lisp after the manner of Common Lisp, and maybe the concurrency
support; but otherwise Emacs Lisp works pretty well for our use cases.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Emacs Lisp's future

[Toon Claes]

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]

On 7 Oct 2016, at 19:07, John Wiegley <jwiegley@gmail.com> wrote:

>  It would be nice to have real compilation
> of Emacs Lisp after the manner of Common Lisp, and maybe the concurrency
> support;


Oh yes, please add concurrency support! 
Would it help if we would start a crowdfunding campaign for this? Because I know a few fellow emacs user who would definitely back it.

Regards,
Toon Claes

[-- Attachment #2: Type: text/html, Size: 1178 bytes --]

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Toon Claes <toon@iotcl.com>
> Date: Sun, 9 Oct 2016 14:03:18 +0200
> Cc: Lars Brinkhoff <lars@nocrew.org>, emacs-devel@gnu.org
> 
> On 7 Oct 2016, at 19:07, John Wiegley <jwiegley@gmail.com> wrote:
> 
>  It would be nice to have real compilation
>  of Emacs Lisp after the manner of Common Lisp, and maybe the concurrency
>  support;
> 
> Oh yes, please add concurrency support! 
> Would it help if we would start a crowdfunding campaign for this? Because I know a few fellow emacs user
> who would definitely back it.

There's already a concurrency branch in the Emacs repository.  It
needs some work before it could be merged to master, so if there are
people here who wants this badly enough, I suggest that they continue
work almost done by Tom Tromey, who developed that branch, instead of
starting anew.

TIA.

Re: Emacs Lisp's future

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 2692 bytes --]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> There's already a concurrency branch in the Emacs repository. It needs some
> work before it could be merged to master, so if there are people here who
> wants this badly enough, I suggest that they continue work almost done by
> Tom Tromey, who developed that branch, instead of starting anew.

I think it needs more than just a little work. I spoke to Tom on the phone
last year, and we both agreed it's not a foregone conclusion that that branch
represents the right way of approaching concurrency for an application like
Emacs.

In general, users don't want programmatic concurrency. What they want is for
Emacs not to freeze up after they've asked it to do something. And there are
other ways of achieving this end -- depending on the nature of the problem --
than making concurrency a first-class citizen, with the innumerable problems
it brings. I fear we'd be debugging subtle interaction issues for the rest of
our lives if we just merged Tom's branch in today. It implements a Java-style
form of concurrency, dependent on mutexes and locking to achieve coherency,
which is fiendishly difficult to get right, even if you confine its use to
just a few small tasks. The hardest bug I ever debugged in my life was a
concurrency bug; the second hardest wasn't even close (and thankfully, I
didn't try to solve it at the same time).

I'd much rather we re-examine the goals we want to achieve with concurrency,
an then ask if there are other ways to get there besides, well, concurrency.
For example, if we had a lightweight forking mechanism with transparent
communication between sub-ordinate processes (think async.el, but in C and
highly efficient), I think we could achieve what users want, without the
downsides developers don't want. Even the popular web browsers are moving in
this direction, since it gives a similar effect to threading, but without as
many of the downsides; or take the popularity of the Actor model, used to
reduce the coherency problem down to just mailboxes.

The reason I love Haskell for its STM concurrency (software transactional
memory) is that it makes certain classes of problems impossible to express. I
believe we would need a mechanism like that for Emacs Lisp, so no one ever has
to hunt down cyclic mutex locks, or reference counts, or why two operations
that should be atomic aren't. I'd rather have a single-threaded Emacs for a
quite a while longer before inviting these sorts problems into our lives.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Concurrency, again (was: Emacs Lisp's future)

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Toon Claes <toon@iotcl.com>,  lars@nocrew.org,  emacs-devel@gnu.org
> Date: Sun, 09 Oct 2016 19:59:09 -0700
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > There's already a concurrency branch in the Emacs repository. It needs some
> > work before it could be merged to master, so if there are people here who
> > wants this badly enough, I suggest that they continue work almost done by
> > Tom Tromey, who developed that branch, instead of starting anew.
> 
> I think it needs more than just a little work. I spoke to Tom on the phone
> last year, and we both agreed it's not a foregone conclusion that that branch
> represents the right way of approaching concurrency for an application like
> Emacs.

We will never know if we don't try it.  We will also never collect the
experience we need to define "the right way" of approaching
concurrency in Emacs, if what's already on the branch is not good
enough.  Meanwhile, chances of someone coming aboard with the kind of
experience and intimate knowledge of Emacs and concurrency that even
approaches that of Tom, and motivated enough to do that kind of
development using some other model, are non-existent.

IOW, saying that the concurrency branch is not good for us without
trying to use it seriously is tantamount to saying that Emacs will
never get any concurrency features at all, and all this sub-thread is
yet another example of issues which never advance anywhere beyond mere
talking about them.  That's too bad, IMO, because I think something
like that could open opportunities for important new features.

> In general, users don't want programmatic concurrency.

AFAIK, the concurrency branch offers programmatic concurrency only as
an opt-in option.  Some concurrency happens automatically without any
code Lisp programmers need to write.  So I see no problem here.  What
we need is for creative Lisp programmers to take this model and
explore how it could be applied to solving real-life problems, with or
without programmatic control of the concurrency, see what are its
limitations and advantages, and generally collect the kind of
experience we sorely need to advance Emacs in that direction.

> I fear we'd be debugging subtle interaction issues for the rest of
> our lives if we just merged Tom's branch in today. It implements a Java-style
> form of concurrency, dependent on mutexes and locking to achieve coherency,
> which is fiendishly difficult to get right, even if you confine its use to
> just a few small tasks. The hardest bug I ever debugged in my life was a
> concurrency bug; the second hardest wasn't even close (and thankfully, I
> didn't try to solve it at the same time).

Did you look at the code, though?  If not, I suggest you do.  The
model is very simple, and its implementation is also simple.  Suffices
it to say that at the time I could port that to MS-Windows and debug
the few problems in a matter of few days work.  When I finished, the
only known problem I left unsolved was the one described here:

  http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00803.html

and in a followup:

  http://lists.gnu.org/archive/html/emacs-devel/2013-09/msg00009.html

For those who want some background, I pointed to long discussions of
the related issues here:

  http://lists.gnu.org/archive/html/emacs-devel/2015-11/msg00068.html

> I'd much rather we re-examine the goals we want to achieve with concurrency,
> an then ask if there are other ways to get there besides, well, concurrency.
> For example, if we had a lightweight forking mechanism with transparent
> communication between sub-ordinate processes (think async.el, but in C and
> highly efficient), I think we could achieve what users want, without the
> downsides developers don't want. Even the popular web browsers are moving in
> this direction, since it gives a similar effect to threading, but without as
> many of the downsides; or take the popularity of the Actor model, used to
> reduce the coherency problem down to just mailboxes.

Sorry for being blunt, but without someone actively working on this,
this is just talk.  Who do we have on board who can implement this in
C in a way that will work on more than a single platform?  On some
important platforms, forking is anything but "lightweight", btw.  And
if async.el is any evidence, it's fiendishly hard to debug and even
understand.  E.g., it took me some time to understand where to plug
the necessary coding stuff, back when we had problems with debbugs
clients related to non-ASCII characters.

In general, debugging processing that is distributed between several
processes is much harder than debugging a single multi-threaded
application.  Since Emacs is mostly in maintenance mode, this aspect
should IMO be significant in any development decision we make.

> The reason I love Haskell for its STM concurrency (software transactional
> memory) is that it makes certain classes of problems impossible to express. I
> believe we would need a mechanism like that for Emacs Lisp, so no one ever has
> to hunt down cyclic mutex locks, or reference counts, or why two operations
> that should be atomic aren't. I'd rather have a single-threaded Emacs for a
> quite a while longer before inviting these sorts problems into our lives.

But we don't even know whether these problems are relevant to what Tom
implemented in the concurrency branch.

We've gone a long way since this issue was first brought up.  We've
changed our sources in significant ways to support concurrency: all
those BVAR and KVAR macros that are all around the C sources were
introduced for that very purpose.  Likewise, the globals.h header
file, and the fact that each variable exposed to Lisp is a member of
some C struct -- all this was for supporting concurrency.  The code
for this is written, debugged, and is only a little ways from being
ready for prime time.  I'm sorry, but in these conditions discarding
all that work without even giving it a serious try makes very little
sense to me.  If we indeed don't intend to use the concurrency branch,
how about removing all that BVAR/KVAR ballast from our sources, to put
our money where our mouth is?

Re: Concurrency, again

[Paul Eggert]

On 10/10/2016 12:17 AM, Eli Zaretskii wrote:
> In general, debugging processing that is distributed between several
> processes is much harder than debugging a single multi-threaded
> application.

That's not my experience. Although both sorts of applications can be 
hard to debug and can have tricky race conditions, I find it easier to 
debug individual single-threaded processes even when they are part of a 
larger multi-process application. No doubt my debugging tools (typically 
GDB on GNU/Linux, for low-level C code) have influenced my experience.

Re: Concurrency, again

[Stefan Monnier]

>> I think it needs more than just a little work. I spoke to Tom on the
>> phone last year, and we both agreed it's not a foregone conclusion
>> that that branch represents the right way of approaching concurrency
>> for an application like Emacs.
> We will never know if we don't try it.

I tend to agree that it's probably not The Right Way.
But I also don't think that The Right Way is anywhere within sight.
Maybe 20 years from now the situation will have changed sufficiently
that something else can be considered, but for now The Right Way
involves something close to a complete rewrite, AFAIK.

IOW, I completely agree with Eli: we should fix the few remaining
issues with the concurrency branch and merge it into master.


        tefan

Re: Concurrency, again

[raman]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Perhaps a first good step might be to cut over  the core elisp modules
to use lexical-binding: t and  fix any/all warnings that the  compiler
reports. >>> I think it needs more than just a little work. I spoke to Tom on the
>>> phone last year, and we both agreed it's not a foregone conclusion
>>> that that branch represents the right way of approaching concurrency
>>> for an application like Emacs.
>> We will never know if we don't try it.
>
> I tend to agree that it's probably not The Right Way.
> But I also don't think that The Right Way is anywhere within sight.
> Maybe 20 years from now the situation will have changed sufficiently
> that something else can be considered, but for now The Right Way
> involves something close to a complete rewrite, AFAIK.
>
> IOW, I completely agree with Eli: we should fix the few remaining
> issues with the concurrency branch and merge it into master.
>
>
>         tefan
>
>

-- 

Re: Concurrency, again

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1778 bytes --]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> IOW, I completely agree with Eli: we should fix the few remaining issues
SM> with the concurrency branch and merge it into master.

I very strongly disagree. I think "merge what we have and make it work" is the
wrong approach. I'd rather not have threading for the next 20 years, than
choose the only option available because it's the only one we have. I'd like
more time to think about this problem myself before making such a hard-to-
revert decision.

Look at what happened with curly quotes: seemingly simple, it got merged into
master "as an experiment". Then witness how many little issues kept popping up
everywhere, how much debate on this list, all for something I'm sure many
thought was totally benign. Now we're talking about doing the same with
something as open-ended as threading?

What could be done to gain experience, as Eli suggests, is maintain a parallel
fork, the way Firefox is doing with their e10s support. That is, build a
version of Emacs with threading enabled and, for a couple of years, release it
in parallel with the main Emacs. Then pick one candidate package, say Gnus,
and have it demonstrate that threading makes everyone's life better, without
making things worse for the rest of us.

If that's too much work to do, my guess is that it's too much work to maintain
threading with the contributors we now have. My prediction is that it will
increase our workload, since a new class of bugs will appear that are much
harder to reproduce and reason about than the sorts of issues we face now.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Concurrency, again

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Wed, 12 Oct 2016 09:42:01 -0700
> Cc: emacs-devel@gnu.org
> 
> Look at what happened with curly quotes: seemingly simple, it got merged into
> master "as an experiment". Then witness how many little issues kept popping up
> everywhere, how much debate on this list, all for something I'm sure many
> thought was totally benign. Now we're talking about doing the same with
> something as open-ended as threading?

Introduction of a significant new feature such as concurrency should
be expected to destabilize Emacs.  It is not a minor feature like
quoting.

OTOH, quotes are pervasive in Emacs; most of the issues we had as
fallout were due to that pervasiveness, something that was not
appreciated at first.  By contrast, concurrency is NOT pervasive: it
only happens if a Lisp application requests that explicitly.  As long
as we don't convert standard Emacs features to use concurrency, it
will not happen in day-to-day operations.

We've had a much tougher decision when bidirectional editing was added
to Emacs.  Unlike concurrency, bidi affects every aspect of Emacs,
because it changed how the display engine works in fundamental ways.
Stefan and Chong, who made that decision, had all the reasons to get
their feet cold, but they made that decision anyway.  I think the risk
was much higher then than it is now.  Given the level of expertise Tom
Tromey has in both Emacs and concurrency, I see no reason to be afraid
now that something terrible will happen that couldn't be fixed as part
of the normal development process.

And if worse comes to worst, and we discover at some point that
concurrency gives us too much trouble for its worth, we can always
remove it from Emacs.

Once again, the concurrency branch is not yet ready for a merge.
Before it is ready, someone should fix the few remaining issues, and
perhaps try using it in a real-life application (which will most
probably expose some more issues).  Unless someone volunteers to do
that, there's no "danger" the concurrency branch will ever be merged.

> What could be done to gain experience, as Eli suggests, is maintain a parallel
> fork, the way Firefox is doing with their e10s support. That is, build a
> version of Emacs with threading enabled and, for a couple of years, release it
> in parallel with the main Emacs. Then pick one candidate package, say Gnus,
> and have it demonstrate that threading makes everyone's life better, without
> making things worse for the rest of us.

I don't think this is practical, what with the resources we have.

> If that's too much work to do, my guess is that it's too much work to maintain
> threading with the contributors we now have. My prediction is that it will
> increase our workload, since a new class of bugs will appear that are much
> harder to reproduce and reason about than the sorts of issues we face now.

I urge you to have a look at the sources on the concurrency branch.
There's no threading, in the usual sense of that word: at any given
time, only a single thread is running.  It's not at all what you seem
to be imagining.  Think about idle timers that run without your having
to do anything to start them: that's what we have there, in a
nutshell, plus a few niceties.

Re: Concurrency, again

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1483 bytes --]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I urge you to have a look at the sources on the concurrency branch. There's
> no threading, in the usual sense of that word: at any given time, only a
> single thread is running. It's not at all what you seem to be imagining.
> Think about idle timers that run without your having to do anything to start
> them: that's what we have there, in a nutshell, plus a few niceties.

More akin to green threading then?

Your point about it not affecting modules that don't opt-in is well taken.
lexical-binding was introduced as major change, but didn't affect any old code
because it had to be enabled explicitly. This allowed us to gradually phase in
its use over several years. If something like that could be done for
threading, some of my fears would be alleviated.

I'd also like to point out a thread that occurred on reddit recently,
attempting to widen this discussion:

https://www.reddit.com/r/emacs/comments/56q8ih/john_wiegley_id_much_rather_we_reexamine_the/

It's interesting to note how many people agree with caution, or don't claim to
need threading, vs. those calling for its inclusion. I know many users who
don't want Emacs locking up on them. This is the problem that needs solving.
Is threading in Emacs Lisp the answer? I still don't know.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Concurrency, again

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 12 Oct 2016 11:01:35 -0700
> 
> https://www.reddit.com/r/emacs/comments/56q8ih/john_wiegley_id_much_rather_we_reexamine_the/
> 
> It's interesting to note how many people agree with caution, or don't claim to
> need threading, vs. those calling for its inclusion.

Most of that thread is irrelevant, because most, if not all the
participants obviously have no idea what the concurrency branch tries
to do, what are its scope and limits, and what will be its impact on
Emacs.

People who want to make up their own mind are encouraged to do

  git diff ...origin/concurrency

and read the diffs.  They are not large, which already should speak
volumes about the potential impact.  They also include very nice
documentation, so C-challenged readers don't need to wade through C
code to get the feeling of what it is about.  There are also ERT tests
that could serve as simple examples of Lisp usage.

For those who do decide to read the diffs, I have a small quiz of 2
questions, which are supposed to summarize what you have learned:

  1) What will be different behavior-wise in a concurrency-enabled
     Emacs that runs Lisp code which never creates any additional
     threads?

  2) If a Lisp application does create other threads, where in the
     code you reviewed will thread switch happen, and by what
     mechanism?

Enjoy your reading.

Re: Concurrency, again

[Stefan Monnier]

>> I urge you to have a look at the sources on the concurrency branch. There's
>> no threading, in the usual sense of that word: at any given time, only a
>> single thread is running. It's not at all what you seem to be imagining.
>> Think about idle timers that run without your having to do anything to start
>> them: that's what we have there, in a nutshell, plus a few niceties.
> More akin to green threading then?

Right.  It does introduce the possibility to have new interleavings, so
new bugs, but 99% of the bugs it uncovers should be reproducible (at
least in theory; tho maybe only in convoluted circumstances) in the
non-concurrent branch as well.

> It's interesting to note how many people agree with caution, or don't claim to
> need threading, vs. those calling for its inclusion.  I know many users who
> don't want Emacs locking up on them.  This is the problem that needs solving.
> Is threading in Emacs Lisp the answer? I still don't know.

I think there are many different needs/desires related to "concurrency".
A typical one (for me anyway) is:

- I want to use Gnus, but I want to be able to edit files while Gnus
  consults the IMAP server to see if there are new messages, or while
  Gnus is waiting for the damn NNTP server to answer.
  Currently I solve this problem by running Gnus in a separate Emacs
  session (i.e. I basically always have 2 Emacs processes).  I lived
  with it for the last 20 years, so it's not that bad, but
  it's inconvenient.

  The concurrency branch should be able to solve this problem.

  We could probably solve this problem also using a cps-translation like
  the one in generator.el, to take the synchronous code using
  accept-process-output, and turn it into code which runs
  asynchronously, from the process filters&sentinels.  It's likely to be
  somewhat brittle, tho.

- I want semantic-mode to scan my buffers in the background and to scan
  several buffers at the same time, to make use of my 8 cores.
  The concurrency branch doesn't really try to solve this problem.
  I'm not sure what's the best way to solve this problem, and I don't
  even know what would be a "simple" way to solve this problem.


-- Stefan

Re: Concurrency, again

[Perry E. Metzger]

On Wed, 12 Oct 2016 15:43:12 -0400 Stefan Monnier
<monnier@IRO.UMontreal.CA> wrote:
> - I want semantic-mode to scan my buffers in the background and to
> scan several buffers at the same time, to make use of my 8 cores.
>   The concurrency branch doesn't really try to solve this problem.
>   I'm not sure what's the best way to solve this problem, and I
> don't even know what would be a "simple" way to solve this problem.

One possibility: move to a multiple processes paradigm where each
buffer has an associated process, and global variable values are
propagated via message passing to a master process. Think Hoare's CSP
or Go or Erlang for precedents, though perhaps with more sugaring to
make it easy for Emacs users to not think much about global state.

This sort of model might be especially important if Emacs starts
being used as a serious web browser by lots of its community, given
how much CPU modern browsers can eat.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Concurrency, again

[Søren Pilgård]

On Wed, Oct 12, 2016 at 10:59 PM, Perry E. Metzger <perry@piermont.com> wrote:
> On Wed, 12 Oct 2016 15:43:12 -0400 Stefan Monnier
> <monnier@IRO.UMontreal.CA> wrote:
>> - I want semantic-mode to scan my buffers in the background and to
>> scan several buffers at the same time, to make use of my 8 cores.
>>   The concurrency branch doesn't really try to solve this problem.
>>   I'm not sure what's the best way to solve this problem, and I
>> don't even know what would be a "simple" way to solve this problem.
>
> One possibility: move to a multiple processes paradigm where each
> buffer has an associated process, and global variable values are
> propagated via message passing to a master process. Think Hoare's CSP
> or Go or Erlang for precedents, though perhaps with more sugaring to
> make it easy for Emacs users to not think much about global state.
>
> This sort of model might be especially important if Emacs starts
> being used as a serious web browser by lots of its community, given
> how much CPU modern browsers can eat.
>

The thing is all current code expects to be able to access any buffer
they like without interferrence from other code.
So to make this new model work we would need to support both paradigms.
If any "old-style" code started execution, by being called by a
function or from a timer, then the entire world would have to halt so
the old-style code could execute with its assumption of sequential
access.
And then each "new-style" code currently executing would have to
"finish" its execution and somehow mark that now its buffer and
bufferlocal variables are in a safe state for the old-style code to
execute.
When all current new-style code is marked "safe" then the old-style
code can execute and when done emacs can continue execution of all the
new-style code.
Does this make sense?

Re: Concurrency, again

[Perry E. Metzger]

On Wed, 12 Oct 2016 23:20:21 +0200 S__ren Pilg__rd
<fiskomaten@gmail.com> wrote:
> > One possibility: move to a multiple processes paradigm where each
> > buffer has an associated process, and global variable values are
> > propagated via message passing to a master process. Think Hoare's
> > CSP or Go or Erlang for precedents, though perhaps with more
> > sugaring to make it easy for Emacs users to not think much about
> > global state.
> >
> > This sort of model might be especially important if Emacs starts
> > being used as a serious web browser by lots of its community,
> > given how much CPU modern browsers can eat.
> >  
> 
> The thing is all current code expects to be able to access any
> buffer they like without interferrence from other code.
> So to make this new model work we would need to support both
> paradigms. If any "old-style" code started execution, by being
> called by a function or from a timer, then the entire world would
> have to halt so the old-style code could execute with its
> assumption of sequential access.

Well, the code expects to *access* buffers, but it doesn't expect to
interact with other executing code. So for "new" code, you would need
to allow code to be able to get to a buffer's contents by some
protocol (possibly even transparent to that process), but it need not
(necessarily) have low level shared data structures the way a
pthreads-like model allows.

I have to admit I haven't thought through all the implications here,
that was an impulse post on my part and I somewhat regret making it
without thinking longer. OTOH, I've become fairly convinced over time
that what is preferable are paradigms where there is zero shared low
level state between executing processes. You just lose all your hair
otherwise yanking it out when you hit bugs.

That said, given the fact that Emacs concurrency would have to evolve
from the current way of doing things, it might be completely
infeasible to implement such a paradigm.

> And then each "new-style" code currently executing would have to
> "finish" its execution and somehow mark that now its buffer and
> bufferlocal variables are in a safe state for the old-style code to
> execute.
> When all current new-style code is marked "safe" then the old-style
> code can execute and when done emacs can continue execution of all
> the new-style code.
> Does this make sense?

That's certainly one possible way of doing things that would allow
migration to a new paradigm without breaking old code.

Concurrency is a giant tarpit of subtleties. :(

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Concurrency, again

[Stefan Monnier]

>> > One possibility: move to a multiple processes paradigm where each
>> > buffer has an associated process, and global variable values are
>> > propagated via message passing to a master process.

Yes, I also like this paradigm, and it might work for a significant part
of existing Elisp code.  But that means (with-current-buffer <buf> <code>)
is turned into something like

    (send <buf> (lambda ()
                  (let ((result <code>))
                    (send <current-buffer> (lambda () <continuation>)))))

which in turns pretty much implies that

    (eq (w-c-b buf (point)) (w-c-b buf (point)))

might not always be t since some other code might be executed between
the two w-c-b.  So it breaks some assumptions which current code might
(and sometimes does) rely on.

To avoid this, we'd have to make it so that with-current-buffer takes
a lock on the buffer, but that gets us into the world of deadlocks.

> OTOH, I've become fairly convinced over time that what is preferable
> are paradigms where there is zero shared low level state between
> executing processes.

Amen.

BTW, in the case of Emacs, something like STM would be great: its
potential lackluster performance wouldn't be too much of an issue.
The problem is that it fundamentally needs to adorn *all* code that
changes shared state (and all code which might have irreversible
side-effects), which would require an enormous amount of work.


        Stefan

Re: Concurrency, again

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> > One possibility: move to a multiple processes paradigm where each >
>>> buffer has an associated process, and global variable values are >
>>> propagated via message passing to a master process.

SM> Yes, I also like this paradigm, and it might work for a significant part
SM> of existing Elisp code.

You can try out the paradigm today with async.el, since that's exactly what it
does.

SM> BTW, in the case of Emacs, something like STM would be great: its
SM> potential lackluster performance wouldn't be too much of an issue. The
SM> problem is that it fundamentally needs to adorn *all* code that changes
SM> shared state (and all code which might have irreversible side-effects),
SM> which would require an enormous amount of work.

If we distinguish between pure and impure forms in the evaluator, we could
enforce that STM transactions are only allowed using pure forms, plus
interactions with transactional variables. This is all that Haskell lets you
to do, but it's quite enough for effective inter-thread communication.

However, if the concurrency we allow is cooperative, all we need to do is
disallow yielding within blocks marked as atomic.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Concurrency, again

[Eli Zaretskii]

> Date: Wed, 12 Oct 2016 17:33:14 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel@gnu.org
> 
> > The thing is all current code expects to be able to access any
> > buffer they like without interferrence from other code.
> > So to make this new model work we would need to support both
> > paradigms. If any "old-style" code started execution, by being
> > called by a function or from a timer, then the entire world would
> > have to halt so the old-style code could execute with its
> > assumption of sequential access.
> 
> Well, the code expects to *access* buffers, but it doesn't expect to
> interact with other executing code.

That's not accurate, since all kinds of hooks triggered by buffer
access could execute code that accesses other buffers and changes
their contents.

> Concurrency is a giant tarpit of subtleties. :(

And Emacs is another giant tarpit.

Re: Concurrency, again

[Perry E. Metzger]

On Thu, 13 Oct 2016 08:55:55 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > Concurrency is a giant tarpit of subtleties. :(  
> 
> And Emacs is another giant tarpit.

Here's another possibility, which I'll admit is even more half baked
(at best) and probably even more excessively ambitious: architect
something entirely new, allow concurrency only for new code, maintain
the old elisp interpreter as single thread only for backward
compatibility, and subsystems that want concurrency can slowly migrate
in to the new part of the system over a period of years.

But, allowing ourselves to ponder the possibility for a minute, this
could solve several problems:

1) There's a perceived desire to move to a new extension language.
2) There's a perceived desire for concurrency, and doing concurrency
well (such that one doesn't end up with data corruption issues here
and there) is hard to retrofit.
3) There's a lot of technical debt in emacs, and a chance to
incrementally move to a new architecture would be a chance to correct
a lot of it.

So the notion (and again, don't take me too seriously, this is very
half baked) would be that one would have a second extension language
with a new design and its own execution engine running alongside the
execution engine for the existing elisp, with relatively minimal
interaction between them to avoid concurrency issues. Over a period of
many years, the codebase would be converted over bit by bit. The new
language should presumably be a lisp that isn't *too* far from elisp,
and a particularly ambitious goal might be to build semi-automated
tools for converting over straightforward code without much human
intervention.

However, all this depends on something I don't know offhand, which is
what the right design for such a new, native-concurrent lisp would be,
or even if a "correct" design is obvious, or even if all of this is
possible. Again, I'm doing the wrong thing here, which is writing a
note long before I've thought about the problem in depth.

I do know that systems that avoid shared memory and do only message
passing seem to be a lot easier to work with than pthreads style
designs where the programmer has to worry that any global state is
potentially in play.

I also know that in the not that distant future processors with
hundreds or thousands of cores are going to be common, so there will
only be more demand with time for the ability to do real concurrency
for performance. Solving the parallel execution issue may be a matter
of importance if (in the future) people are going to do things like
using Emacs as a primary web browser and development environment.

But again, I won't pretend I know how to solve all of this.


Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Concurrency, again

[Stefan Monnier]

> I also know that in the not that distant future processors with
> hundreds or thousands of cores are going to be common, so there will
> only be more demand with time for the ability to do real concurrency
> for performance. Solving the parallel execution issue may be a matter
> of importance if (in the future) people are going to do things like
> using Emacs as a primary web browser and development environment.

"Luckily" here, it seems that (usefully) increasing the number of cores
is not that easy, so this future is still a bit further.  But I'll note
also that concurrency is not solution for parallelism.

IOW if we want to take advantage of tens of cores in Emacs, we have to
think very differently.  A language with STM or with "communicating
agents" won't cut it, I think.


        Stefan

Re: Concurrency, again

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 2326 bytes --]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> IOW if we want to take advantage of tens of cores in Emacs, we have to
SM> think very differently. A language with STM or with "communicating agents"
SM> won't cut it, I think.

I can't think of too many cases where existing Emacs Lisp would immediately
benefit from being massively parallel, either. Too much code is written in an
imperative manner (manipulating variables) than a functional one (generating
new values from inputs). If Lars had written much of Gnus using a map/reduce
pattern, I could see it suddenly speeding up if we parallelized those
primitives. But I'm guessing a fair bit of rewrite would be needed to take
advantage of extra cores.

Note too that all of this perceived benefit is still theoretical. We've yet to
answer the main question -- in my mind, the only question when it comes to
whether we should consider including this type of support: Will it solve the
annoyance people have of Emacs blocking when they ask it to do certain things?

Examples of my own, coffee-break inducing moments:

 - Asking for 10,000 back articles in Gnus, with sorting and threading on
 - Applying a keyboard macro to manipulate 1000 file contents in dired
 - Hitting 'g' in Magit in a repository with tons of stashes, pull requests,
   pending changes, etc.
 - Waiting for ERC to replay message logs from ZNC in 20 channels
 - Asking Proof General to check a file that depends on 30 other files

And yet, this list is *about it* for me, and I practically live in Emacs. I'm
not even completely sure yet that solving these would appreciably benefit my
workflow. I don't mind the occasional coffee break.

I just want to make sure we're not try to solve the technical problem just
because it's solvable; we should only solve it if it's a real problem that is
getting in people's way, or is stopping us from doing the Next Cool Thing.

So, thanks to Eli I'm open to considering the concurrency branch for merging.
But I'm also in no hurry whatsoever, and I'd prefer to avoid the extra
complexity if it doesn't actually represent a significant improvement.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Concurrency, again

[Stefan Monnier]

> I can't think of too many cases where existing Emacs Lisp would immediately
> benefit from being massively parallel, either.

I think the benefit only exists if we want to increasing the
functionality along with the available compute power.

AFAIK single-thread performance has reached a plateau about ten years
ago and there's no sign of its end, and Emacs is "stuck" in
this plateau.  Luckily for us, Emacs is far from the only one: very few
programs nowadays know how to take advantage of the extra compute power.

But I see no way to make Emacs take much advantage of parallelism, other
than in completely separate tasks written in other languages
(e.g. a separate executable scanning all a project's files in parallel).

All I was pointing out is that this discussion is about concurrency and
not parallelism.

> answer the main question -- in my mind, the only question when it comes to
> whether we should consider including this type of support: Will it solve the
> annoyance people have of Emacs blocking when they ask it to do certain things?

Indeed.

> I just want to make sure we're not try to solve the technical problem just
> because it's solvable; we should only solve it if it's a real problem that is
> getting in people's way, or is stopping us from doing the Next Cool Thing.

AFAIK the issue is fairly simple: writing async code (with process
filters/sentinels) in Elisp is rather painful, which is why we have
those blocking situations (which could all be fixed by a "bit" of
rewriting).

So don't think of it as "how can we avoid those breaks" but rather "how
can we make it easier for the programmer to write non-blocking code".

I'm sure it'll break code.  But I think it's worth a try, if only to
convince ourselves that Elisp will never ever move away from the
single-thread world.


        Stefan

Re: Concurrency, again

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I thought that "concurrency" and "parallelism" were the same.
I don't understand the distinction you are making.

I think the main use of parallelism in Emacs would be to make some
commands asynchronous.  Currently, the only way a command can be
asynchronous is if it runs in a separate process.  With some rather
unsophisticated parallelism, commands written in Lisp could 
be made asynchronous.

The easy way to support this would be to provide special constructs
that can be used to make a particular program able to run
asynchronously.  Such programs could run in parallel with any and all
ordinary Lisp programs.

It would not be necessary to allow two ordinary Lisp programs to run
in parallel.  In other words, it would be ok if Emacs allowed only one
ordinary program at any time -- alongside any number of special
asynchronous programs.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Concurrency, again

[Stefan Monnier]

> I thought that "concurrency" and "parallelism" were the same.

They're related and they overlap.

If you use use a pseudo-parallel system (such as the `concurrency`
branch), then you have concurrency but not parallelism.
Vector processing (or GPUs nowadays) on the other hand, offers
parallelism but not concurrency.

To take another example, from the world of memory management:

- A concurrent GC is one where the mutator is not stopped while the
  collector does its job.
- A parallel GC is one where the collector divides its job into chunks
  that can be performed at the same time by different CPUs.  It may be
  concurrent or not.


        Stefan

Re: Concurrency, again

[John Wiegley]

>>>>> "RS" == Richard Stallman <rms@gnu.org> writes:

RS> I thought that "concurrency" and "parallelism" were the same. I don't
RS> understand the distinction you are making.

"Concurrency is when two or more tasks can start, run, and complete in
overlapping time periods. It doesn't necessarily mean they'll ever both be
running at the same instant. Eg. multitasking on a single-core machine.

Parallelism is when tasks literally run at the same time, eg. on a multicore
processor." -- Stack Overflow

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Concurrency, again

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Fri, 14 Oct 2016 17:01:24 -0400
> Cc: emacs-devel@gnu.org
> 
> It would not be necessary to allow two ordinary Lisp programs to run
> in parallel.  In other words, it would be ok if Emacs allowed only one
> ordinary program at any time -- alongside any number of special
> asynchronous programs.

That's what the code on the "concurrency" branch of the Emacs
repository does -- except that (1) it doesn't allow you to create
special asynchronous programs at all, and (2) it is capable of
switching to another ordinary Lisp program when the current one is
waiting for something (e.g., keyboard input or subprocess output), or
explicitly yields to another.

Re: Concurrency, again

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It would not be necessary to allow two ordinary Lisp programs to run
  > > in parallel.  In other words, it would be ok if Emacs allowed only one
  > > ordinary program at any time -- alongside any number of special
  > > asynchronous programs.

  > That's what the code on the "concurrency" branch of the Emacs
  > repository does -- except that (1) it doesn't allow you to create
  > special asynchronous programs at all, and (2) it is capable of
  > switching to another ordinary Lisp program when the current one is
  > waiting for something (e.g., keyboard input or subprocess output), or
  > explicitly yields to another.

(2) is a source of possible conflicts between various programs; I
suspect that lots of bugs could result.  Are such bugs why the
"concurrency" branch is not ready?

To limit concurrency to specially designated programs
might make it easier to avoid those problems.  Those programs
could conceivably be written following certain special rules
that avoid the bugs.



-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Concurrency, again

[John Wiegley]

>>>>> "RS" == Richard Stallman <rms@gnu.org> writes:

RS> (2) is a source of possible conflicts between various programs; I suspect
RS> that lots of bugs could result. Are such bugs why the "concurrency" branch
RS> is not ready?

RS> To limit concurrency to specially designated programs might make it easier
RS> to avoid those problems. Those programs could conceivably be written
RS> following certain special rules that avoid the bugs.

I'd like to note that the danger of such bugs really depends on how heavily a
given software package plays tricks with mutexes and yields.

For counter-example: Eshell today contains a function called `eshell-do-eval',
whose sole purpose is to evaluate a Lisp form up to a point where it would
otherwise block; save the current execution stack manually as a continuation
lambda in a global; then resume execution where it left off the next time
evaluation is attempted.

This code -- which is by far the most complex aspect of Eshell -- would simply
disappear once `thread-yield` became available, since that's exactly what
Eshell needs to do: yield execution until a condition is met, provided the
user isn't busily engaged doing something else.

So, for some use cases, threads as they've been proposed will simplify code
and reduce bugs, not make things worse. It depends on *how* it is used, what
damage it will do. I suppose the question then is: How much rope are we giving
new programmers to hang themselves with?

There have been other suggestions made to offer a better interface on top of
the threading primitives -- based on the actor model, or message passing, or
transactions, or futures, etc. -- such that none of the failure modes we fear
are possible through such an interface. This would restrict what users can do,
but might not restrict them so much that they can't achieve what is needed to
make Emacs less blocking.

So, even if we merged the concurrency branch today, this doesn't immediately
invite the hellhounds. It's just an implementation strategy, after all, for
allowing interleaved evaluation. I'm sure we can devise bulletproof ways to
make use of that functionality to achieve our end goal. If not, we revert.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Concurrency, again

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> Date: Sat, 15 Oct 2016 18:03:15 -0400
> 
>   > That's what the code on the "concurrency" branch of the Emacs
>   > repository does -- except that (1) it doesn't allow you to create
>   > special asynchronous programs at all, and (2) it is capable of
>   > switching to another ordinary Lisp program when the current one is
>   > waiting for something (e.g., keyboard input or subprocess output), or
>   > explicitly yields to another.
> 
> (2) is a source of possible conflicts between various programs; I
> suspect that lots of bugs could result.

I believe Tom Tromey, who wrote the code, took precautions so that
such bugs won't happen "by design".  Some design principles and
limitations towards this end are:

 . thread switching happens either when Emacs is waiting for input, or
   by explicit request of a thread to yield, or when a thread blocks
   because it waits for a mutex or to join another thread

 . global variables are shared between all threads, but local variable
   let-bindings are specific to the thread that made them

 . each thread has its own current buffer and its own match data

 . by default, a process is locked to the thread that created it,
   which means output from the process can only be accepted by that
   thread; however, a Lisp program can specify to which thread a
   process is locked, or unlock the process, in which case any thread
   can accept its output.  In any case, once some thread starts
   waiting for output of a process, the process becomes temporarily
   locked to that thread, so only that thread will eventually receive
   the output, unless it yields.  If a thread exits, all its processes
   are unlocked.

> Are such bugs why the "concurrency" branch is not ready?

We don't really know, because the branch was never seriously used.

> To limit concurrency to specially designated programs
> might make it easier to avoid those problems.  Those programs
> could conceivably be written following certain special rules
> that avoid the bugs.

That's what the branch does: if a Lisp program doesn't create any
threads, it will run normally, as in today's Emacs, using a single
main thread.

Re: Concurrency, again

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Are such bugs why the "concurrency" branch is not ready?

  > We don't really know, because the branch was never seriously used.

We could do something about that.  Can we modify them so that
you need to set a variable or you get the current default
behavior?  Then it could be merged in, and people could try it.

  > That's what the branch does: if a Lisp program doesn't create any
  > threads, it will run normally, as in today's Emacs, using a single
  > main thread.

Maybe the flag (or flags) could control whether these applications
create threads.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Concurrency, again

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> Date: Sun, 16 Oct 2016 20:43:14 -0400
> 
>   > > Are such bugs why the "concurrency" branch is not ready?
> 
>   > We don't really know, because the branch was never seriously used.
> 
> We could do something about that.  Can we modify them so that
> you need to set a variable or you get the current default
> behavior?  Then it could be merged in, and people could try it.
> 
>   > That's what the branch does: if a Lisp program doesn't create any
>   > threads, it will run normally, as in today's Emacs, using a single
>   > main thread.
> 
> Maybe the flag (or flags) could control whether these applications
> create threads.

That could probably be done, but it still needs someone motivated
enough to work on that.  Volunteers are welcome.

Re: Concurrency, again

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1659 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Do., 13. Okt. 2016 um
20:15 Uhr:

> > I can't think of too many cases where existing Emacs Lisp would
> immediately
> > benefit from being massively parallel, either.
>
> I think the benefit only exists if we want to increasing the
> functionality along with the available compute power.
>
> AFAIK single-thread performance has reached a plateau about ten years
> ago and there's no sign of its end, and Emacs is "stuck" in
> this plateau.  Luckily for us, Emacs is far from the only one: very few
> programs nowadays know how to take advantage of the extra compute power.
>
> But I see no way to make Emacs take much advantage of parallelism, other
> than in completely separate tasks written in other languages
> (e.g. a separate executable scanning all a project's files in parallel).
>
> All I was pointing out is that this discussion is about concurrency and
> not parallelism.
>

And that's a very important distinction to make. I think most of the
resistance to the concurrency branch in Emacs comes from the fact that it
uses OS parallelism primitives such as threads and mutexes to implement
cooperative concurrency.
Parallelism indeed appears to be impossible for the foreseeable future; it
would require almost a complete rewrite of Emacs. Cooperative concurrency,
on the other hand, has a much more limited impact.
I'd suggest to adapt the terminology accordingly: "thread" *can* of course
mean "cooperative userland thread", but for most programmers it has come to
mean "OS-level parallel thread with preemptive kernel-level scheduling".
Should we avoid this term and use "coroutine" instead?

[-- Attachment #2: Type: text/html, Size: 2388 bytes --]

Re: Concurrency, again

[Dmitry Gutov]

On 25.10.2016 16:34, Philipp Stephani wrote:

> I'd suggest to adapt the terminology accordingly: "thread" *can* of
> course mean "cooperative userland thread", but for most programmers it
> has come to mean "OS-level parallel thread with preemptive kernel-level
> scheduling". Should we avoid this term and use "coroutine" instead?

Or "fiber" [0], which might be more familiar to at least some Ruby, D 
and .NET programmers [1].

[0] https://en.wikipedia.org/wiki/Fiber_(computer_science)
[1] 
https://en.wikipedia.org/wiki/Coroutine#Implementation_in_the_.NET_Framework_as_fibers

Re: Concurrency, again

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 155 bytes --]

On 2016-10-13 13:25, John Wiegley wrote:
>  - Asking Proof General to check a file that depends on 30 other files

Isn't proof general asynchronous?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Concurrency, again

[John Wiegley]

>>>>> "CP" == Clément Pit--Claudel <clement.pit@gmail.com> writes:

PC> On 2016-10-13 13:25, John Wiegley wrote:
>> - Asking Proof General to check a file that depends on 30 other files

CP> Isn't proof general asynchronous?

Sometimes it is, sometimes my UI gets frozen.  I haven't identified under
which circumstances this happens yet.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Concurrency, again

[Ted Zlatanov]

On Thu, 13 Oct 2016 10:25:26 -0700 John Wiegley <jwiegley@gmail.com> wrote: 

JW> Examples of my own, coffee-break inducing moments:

JW>  - Asking for 10,000 back articles in Gnus, with sorting and threading on
JW>  - Applying a keyboard macro to manipulate 1000 file contents in dired
JW>  - Hitting 'g' in Magit in a repository with tons of stashes, pull requests,
JW>    pending changes, etc.
JW>  - Waiting for ERC to replay message logs from ZNC in 20 channels
JW>  - Asking Proof General to check a file that depends on 30 other files

A few from my experience:

- byte-compiling
- waiting for dynamic data sources like Helm sources
- Tramp to a slow system
- eww or Gnus pausing while rendering a page/article
- gnus-registry and Gnus searching (lots of data, not all local)
- syntax highlighting (debugging it is... tough and the internals are scary)
- Javascript, Java, C++... parsing complex languages in general

I hope that's useful.

Ted

Re: Concurrency, again

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> A few from my experience:
>
> - Tramp to a slow system

The point of Tramp is not to use additional power of sleeping cores, but
waiting for slow connections. One could already make some of its
handlers asynchronously as of today, but a proper API is missing. All
callers of (for example) `copy-file' expect, that the function has
finished once it returns. We would need another
`copy-file-asynchronously', which offers also a handler to be called
once the copying has finished.

Sure, concurrency in Elisp would help to implement a respective handler
in Tramp. But I do expect much more work for all uses of `copy-file',
which must change to the other semantics of `copy-file-asynchronously'.

> Ted

Best regards, Michael.

Re: Concurrency, again

[John Wiegley]

>>>>> "MA" == Michael Albinus <michael.albinus@gmx.de> writes:

MA> The point of Tramp is not to use additional power of sleeping cores, but
MA> waiting for slow connections. One could already make some of its handlers
MA> asynchronously as of today, but a proper API is missing. All callers of
MA> (for example) `copy-file' expect, that the function has finished once it
MA> returns. We would need another `copy-file-asynchronously', which offers
MA> also a handler to be called once the copying has finished.

Btw, this is just what I did in dired-async, and it works well enough there.
The code looks roughly like this (somewhat simplified):

      (async-start
       ;; lambda form executed in the child process
       `(lambda ()
          (require 'cl-lib)
          (require 'dired-aux)
          (require 'dired-x)
          ,(async-inject-variables dired-async-env-variables-regexp)
          (condition-case err
              (copy-file from to ok dired-copy-preserve-time)
            (file-date-error
             (dired-log "Can't set date on %s:\n%s\n" from err)))
          ,(dired-async-maybe-kill-ftp))

       ;; the callback
       (lambda (&optional _ignore)
         (dired-async-after-file-create
          total (list operation (length async-fn-list)) failures skipped)))

So, you can do what you mentioned today. Maybe a better solution would be to
move this support into C and support direct transmission of values into the
child process over IPC, rather than streaming them as text across a file
handle, as it does now.

Further, I'm not sure if the concurrency branch would actually execute
`copy-file` asynchronously; Emacs might still block at some point during the
copy, since there's a GIL preventing threads from executing at the same time.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Concurrency, again

[John Wiegley]

>>>>> "MA" == Michael Albinus <michael.albinus@gmx.de> writes:

MA> The point of Tramp is not to use additional power of sleeping cores, but
MA> waiting for slow connections. One could already make some of its handlers
MA> asynchronously as of today, but a proper API is missing. All callers of
MA> (for example) `copy-file' expect, that the function has finished once it
MA> returns. We would need another `copy-file-asynchronously', which offers
MA> also a handler to be called once the copying has finished.

Btw, this is just what I did in dired-async, and it works well enough there.
The code looks roughly like this (somewhat simplified):

      (async-start
       ;; lambda form executed in the child process
       `(lambda ()
          (require 'cl-lib)
          (require 'dired-aux)
          (require 'dired-x)
          ,(async-inject-variables dired-async-env-variables-regexp)
          (condition-case err
              (copy-file from to ok dired-copy-preserve-time)
            (file-date-error
             (dired-log "Can't set date on %s:\n%s\n" from err)))
          ,(dired-async-maybe-kill-ftp))

       ;; the callback
       (lambda (&optional _ignore)
         (dired-async-after-file-create
          total (list operation (length async-fn-list)) failures skipped)))

So, you can do what you mentioned today. Maybe a better solution would be to
move this support into C and support direct transmission of values into the
child process over IPC, rather than streaming them as text across a file
handle, as it does now.

Further, I'm not sure if the concurrency branch would actually execute
`copy-file` asynchronously; Emacs might still block at some point during the
copy, since there's a GIL preventing threads from executing at the same time.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Concurrency, again

[Ted Zlatanov]

On Fri, 14 Oct 2016 17:47:21 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
>> A few from my experience:
>> 
>> - Tramp to a slow system

MA> The point of Tramp is not to use additional power of sleeping cores, but
MA> waiting for slow connections. One could already make some of its
MA> handlers asynchronously as of today

You are right that in this use case (unlike some others I gave),
a non-blocking API is all that's needed.

I'm also not sure if Tramp would be asynchronous when a Tramp method
used just core functions, no process.

Ted

Re: Concurrency, again

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I'm also not sure if Tramp would be asynchronous when a Tramp method
> used just core functions, no process.

There are Tramp methods like "scp", which use already a second
asynchronous process for copying files internally. This could be exposed
to the user easily, once we have an asynchronous `copy-file-*'
interface.

> Ted

Best regards, Michael.

Re: Concurrency, again

[Ted Zlatanov]

On Sat, 15 Oct 2016 09:16:11 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I'm also not sure if Tramp would be asynchronous when a Tramp method
>> used just core functions, no process.

MA> There are Tramp methods like "scp", which use already a second
MA> asynchronous process for copying files internally. This could be exposed
MA> to the user easily, once we have an asynchronous `copy-file-*'
MA> interface.

I have a suggestion, crossing into implementation. I hope that's
somewhat useful.

Rather than a special function, the new copy-file could simply return a
Future of some sort. Sebastian Weisner suggested that general approach;
I think Futures are a good API based on my experience. Here's how I
would implement it.

The usage could be like this:

1) fully backwards compatible: (use value of (copy-file ...)) ->
collects the Future, blocking until it's available

2) (with-future (copy-file ...) -> makes local functions available
inside the macro to check if the value is available, like
(future-is-ready) or whatever. This can provide voluntary yields based
on the availability of the future.

3) (setq ((f (future (copy-file ...))))) -> capture Future in f, do
voluntary yields on it, etc.

The key is that unless you are aware of Futures, you'll get the
collected value. I think that's the easiest way to provide the async
functionality without breaking code or adding new API functions to every
library out there.

Ted

Re: Concurrency, again

[Ken Raeburn]

We’re getting quite a list of “coffee-break inducing moments”. :-)

To what’s been posted so far, I’ll add a couple of cases that I’ve run into trouble with, which might be helped in part with multiple threads:

- regexp scanning in a huge file that took so long my IM sessions would time out because the Emacs-based client code couldn’t respond to the server (which scanning also turned out not to be interruptible with ^G)

- file operations hanging for minutes at a time when an NFS session stalls due to network lossage, freezing the UI, causing IM session timeouts, etc; I’d love for auto-revert-mode type timestamp checking for buffers I’m not editing at the moment to not lock up my entire Emacs session in that situation, and it’d be a huge amount of work, but being able to cancel or abort a hung “save” or find-file operation would be pretty nice

Letting the IM client code use a helper thread for callbacks could help, as long as regexp scanning and file operations release the global lock.  Having the UI not freeze up during file operations is probably more complicated, especially if we want to let the user hit ^G and go do something else, since NFS operations tend (depending on implementation) to be uninterruptible.

In the latter case (and probably some others, like Perry’s video playback in an eww buffer), C-level threads mostly invisible to Lisp might suffice, but I think there are a number of cases where we do want concurrent evaluation of multiple bodies of Lisp code, which code is often most logically written in structured or functional programming style, not “massive pile of callbacks” style.

Re: Concurrency, again

[Stefan Monnier]

> - regexp scanning in a huge file that took so long my IM sessions would time
> out because the Emacs-based client code couldn’t respond to the server
> (which scanning also turned out not to be interruptible with ^G)

FWIW, this is an algorithmic problem.  The better to way to fix this is
to avoid backtracking in the regexp engine.

Also to avoid this problem, we'd need to spawn a new thread and since
regexp-matching is normally not expected to take a long time, that means
we'd need to spawn a new thread either "all the time" (just in case it
proves necessary), or "lazily" (i.e. if the current command takes too
long, then user can use a special handshake that spawns a new thread
and lets the current command finish asynchronously in that thread).


        Stefan

Re: Concurrency, again

[Stefan Huchler]

Hello guys,

I want to add something to that conversation,

John Wiegley <jwiegley@gmail.com> writes:

> And yet, this list is *about it* for me, and I practically live in Emacs. I'm
> not even completely sure yet that solving these would appreciably benefit my
> workflow. I don't mind the occasional coffee break.

I gone from gnome to various tiling wms to finaly when it came up land
in Exwm, for those who dont know it a emacs based window manager.

It improved my workflow cause I dont have to handle a 2 layer
applcation/buffer layout, I dont have to think that complex do I know
change to other program like lets say thunderbird, or do I only have to
change to another buffer like gnus (or *Groups*).

I also dont have to switch first to emacs and then in a second step to
gnus.

I am shure many of you know think how does that relate to that topic,
well if you use exwm, emacs is basicly your complete desktop. So
concurency or parralism is much of a bigger problem. If you use a
external browser you can do things there if emacs freezes, in exwm you
cant.

Also its nearly impossible for me to start a 2nd emacs for gnus inside exwm at
least with the same configuration, cause it would eat the shortcuts to
switch back to the "primary" emacs session.

But gnus has on the one hand great usability and features on the other
hand is a pain in the ass, using it on 2 different machines is difficult
to setup, also the window configuration with constant resizing or
creating many new buffers causes problems at least for my usecase, when
I then switch to another buffer it gets not resised to full screen
automaticly and when I switch back to gnus it does not rearange the
buffers (or windows).

So I question the quality of that package itself, and maybe hacking gnus
itself would be a better way to make things better than rewrite emacs
with some multithread magic.

But yes I want a emacs browser and also instead of running this monstres
proprietary javascript programs that makes us want to use 8 cpu cores, I
would rather have better ways to extract the data from websites, and
beeing able to answer to comments on websites with something like gnus
groups. 

Also even I use keyboard driven browsers like conkeror or qutebrowser a
native emacs browser would still be better.

I maybe give a bit a mixed message, but my main points are that having a
second emacs instance is not always a (easy) option, 2nd people like me
want to use emacs as desktop, I think there locking is even a bigger
issue.

Maybe a middle step would be to integrate XELB/EXWM and just have more
than one instance of emacs running but INSIDE emacs. so that you dont
have 2 windows and then make them play better together with each other?
Communicate between them?

Maybe have single-progamm emacs instances tat then is a buffer in the
master-emacs, that would also solve the problem with rearanging windows
in gnus.

Hope I did not talk to much unrelated stuff. But it gives you maybe a
different perspective of a slightly different use case.


> I just want to make sure we're not try to solve the technical problem just
> because it's solvable; we should only solve it if it's a real problem that is
> getting in people's way, or is stopping us from doing the Next Cool
> Thing

Yes I agree to that, there would be other places where a bit love would
help. solving the browser-mess as example (luckily we see here movement
with ewb and emacs xwidget/webkit. 

Re: Concurrency, again

[Perry E. Metzger]

On Thu, 13 Oct 2016 10:25:26 -0700 John Wiegley <jwiegley@gmail.com>
wrote:
> >>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:  
> Examples of my own, coffee-break inducing moments:
>
>  - Asking for 10,000 back articles in Gnus, with sorting and
> threading on
>  - Applying a keyboard macro to manipulate 1000 file contents in
> dired
>  - Hitting 'g' in Magit in a repository with tons of stashes, pull
> requests, pending changes, etc.
>  - Waiting for ERC to replay message logs from ZNC in 20 channels
>  - Asking Proof General to check a file that depends on 30 other
> files
>
> And yet, this list is *about it* for me,

Imagine, though, if in the not too distant future Emacs is used by
many of us as our web browser thanks to Webkit integration. That might
change the game a lot.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Concurrency, again

[Eli Zaretskii]

> Date: Mon, 17 Oct 2016 10:53:45 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> > And yet, this list is *about it* for me,
> 
> Imagine, though, if in the not too distant future Emacs is used by
> many of us as our web browser thanks to Webkit integration. That might
> change the game a lot.

Asynchronous network communications are supported already on master,
so I'm not sure what problems you see in that direction.

Re: Concurrency, again

[Perry E. Metzger]

On Mon, 17 Oct 2016 19:07:33 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Mon, 17 Oct 2016 10:53:45 -0400
> > From: "Perry E. Metzger" <perry@piermont.com>
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> >   
> > > And yet, this list is *about it* for me,  
> > 
> > Imagine, though, if in the not too distant future Emacs is used by
> > many of us as our web browser thanks to Webkit integration. That
> > might change the game a lot.  
> 
> Asynchronous network communications are supported already on master,
> so I'm not sure what problems you see in that direction.
> 

Every open browser tab (should I say "window", this being emacs?) can
potentially be doing all sorts of computation in the background via
JavaScript, and that could potentially momentarily hang the editor if
they're sharing an underlying thread.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Concurrency, again

[Eli Zaretskii]

> Date: Mon, 17 Oct 2016 12:34:59 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: jwiegley@gmail.com, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> On Mon, 17 Oct 2016 19:07:33 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > > Date: Mon, 17 Oct 2016 10:53:45 -0400
> > > From: "Perry E. Metzger" <perry@piermont.com>
> > > Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> > >   
> > > > And yet, this list is *about it* for me,  
> > > 
> > > Imagine, though, if in the not too distant future Emacs is used by
> > > many of us as our web browser thanks to Webkit integration. That
> > > might change the game a lot.  
> > 
> > Asynchronous network communications are supported already on master,
> > so I'm not sure what problems you see in that direction.
> 
> Every open browser tab (should I say "window", this being emacs?) can
> potentially be doing all sorts of computation in the background via
> JavaScript, and that could potentially momentarily hang the editor if
> they're sharing an underlying thread.

So you are saying the problem is only with URLs that invoke
JavaScript?  Good.

Re: Concurrency, again

[Perry E. Metzger]

On Mon, 17 Oct 2016 19:57:37 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > > > Imagine, though, if in the not too distant future Emacs is
> > > > used by many of us as our web browser thanks to Webkit
> > > > integration. That might change the game a lot.    
> > > 
> > > Asynchronous network communications are supported already on
> > > master, so I'm not sure what problems you see in that
> > > direction.  
> > 
> > Every open browser tab (should I say "window", this being emacs?)
> > can potentially be doing all sorts of computation in the
> > background via JavaScript, and that could potentially momentarily
> > hang the editor if they're sharing an underlying thread.  
> 
> So you are saying the problem is only with URLs that invoke
> JavaScript?  Good.

Well, active content like video or audio too. For good or ill, though,
active content is now most of the web. Also, the line between
JavaScript and active content (like HTML5 video) is often thin. If one
wanted to have Emacs be a truly effective browser, it would need the
ability to deal with this. Someone might want to watch HTML5 video in
one Emacs window while reading their mail in another, and why
shouldn't they want to be able to do that? The video might even have
been embedded in one of their mail messages.

I think having Emacs be a truly effective browser is a very useful
goal btw., especially since Webkit, which is already free software,
can do all the hard stuff. The idea of being able to isearch inside
the web page of a blog, mark and copy a short program out of a
posting, c-x o, and yank it, is really tantalizing. I love the idea of
never having to leave emacs again.

Of course, one could do something like wrapping the Webkit instances
in subprocesses, which might not be awful from an isolation
viewpoint. Still, that leaves the question of how to handle the
interaction with such a thing.

I suppose part of the problem here is that the existing Emacs
implementation goes back to an era before such things could have even
been contemplated. This might be what one would want out of Emacs but
it is not what the current code was built to do.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Concurrency, again

[Lars Ingebrigtsen]

"Perry E. Metzger" <perry@piermont.com> writes:

> Of course, one could do something like wrapping the Webkit instances
> in subprocesses, which might not be awful from an isolation
> viewpoint. 

Have you tried using the webkit widgets?  Emacs does not hang while,
say, playing video, so presumably it's running its own thread.  It is,
in any case, irrelevant to the discussion of Emacs Lisp concurrency:
The Emacs "event loop" is doing anything inside the widgets.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Concurrency, again

[Stefan Huchler]

Lars Ingebrigtsen <larsi@gnus.org> writes:
>
> Have you tried using the webkit widgets?  Emacs does not hang while,
> say, playing video, so presumably it's running its own thread.  It is,
> in any case, irrelevant to the discussion of Emacs Lisp concurrency:
> The Emacs "event loop" is doing anything inside the widgets.

As far as I understand the xwidget webkit browser is "only" some sort of
a Client-server thing where you send messages from emacs to another
process and it then does stuff and sends maybe answers back.

More interesting is where does ewb go. Cause I doubt that the Xwidget
thing will be very well hackable, like emacs can show pictures and use
that to show pdfs, but it dont directly interact with them on a pixel
level or can select the text in this pictures or manipulate it with
functions directly.

And I could see ewb be the real hacker browser in the future. Cause we
dont get any api/standarts in the web besides imap and pop3 webpages and
as example comment sections cant be threated like nntp messages or as
gnus group. So if you wanna interact with such sites you need a
browser.

But (at least in a normal browser) they force you to use their interface
and some sites even force shortcuts on you (like github). So a browser
that would make it easy to format sites the way you like and use emacs
shortcuts in it would be really nice.

Like the readability mode, does start it, with some sort of basic
heuristic to change the optics of web sites.

Sorry I go OT, but the question is, does ewb need some
Concurrency? I guess not really cause the javascript stuff is more a
pain than a gain for most cases in such environment.

Maybe if you wanna have 10 20 ewb tabs open and some sort of automatic
refresh in the background?

Re: Concurrency, again

[Lars Ingebrigtsen]

Stefan Huchler <stefan.huchler@mail.de> writes:

> As far as I understand the xwidget webkit browser is "only" some sort of
> a Client-server thing where you send messages from emacs to another
> process and it then does stuff and sends maybe answers back.

Yes.  It's pretty opaque.  I think it's possible to see a way to
interact with these objects more naturally by creating a Javascript
bridge that would expose the DOM fully to Emacs, and then you could
rewrite all Emacs commands to do things to that DOM.  Like `C-t'
(transpose-chars) with "point" inside one of these objects would look at
the DOM around point, figure out the necessary changes, and then update
the DOM inside the widget.

And so on.

But it would be...  er...  a major undertaking, and it would always be a
toy.

And it's irrelevant to the concurrency discussion, really.  :-)

> Sorry I go OT, but the question is, does ewb need some
> Concurrency? I guess not really cause the javascript stuff is more a
> pain than a gain for most cases in such environment.

I'm assuming you're referring to eww, and it sure does need concurrency.
Computing complicated layouts can be slow, and it would be nice if it
could take without stopping the user from doing other things.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Concurrency, again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 18 Oct 2016 16:34:18 +0200
> Cc: emacs-devel@gnu.org
> 
> I'm assuming you're referring to eww, and it sure does need concurrency.
> Computing complicated layouts can be slow, and it would be nice if it
> could take without stopping the user from doing other things.

It could start by only doing layout of the visible portion, and doing
the rest from an idle timer.  Or does it already do that?

Re: Concurrency, again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> It could start by only doing layout of the visible portion, and doing
> the rest from an idle timer.  Or does it already do that?

That doesn't work when trying to figure out how to lay out a deeply
nested multi-columnar layout.  It's complicated.

And the current implementation sucks: It basically combinatorally tries
all possibilities to find the best layout.  There are probably shortcuts
that can be taken, but any I've tried makes common table-based layouts
awful.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Concurrency, again

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefan.huchler@mail.de,  emacs-devel@gnu.org
> Date: Tue, 18 Oct 2016 17:05:41 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It could start by only doing layout of the visible portion, and doing
> > the rest from an idle timer.  Or does it already do that?
> 
> That doesn't work when trying to figure out how to lay out a deeply
> nested multi-columnar layout.

In that case, no, so you cannot win there.  But in other cases, yes.
So this will allow faster display of at least some pages, I think.

An in any case, you can stop when you reach the end of the current
top-level element that is beyond the visible portion.

> And the current implementation sucks: It basically combinatorally tries
> all possibilities to find the best layout.  There are probably shortcuts
> that can be taken, but any I've tried makes common table-based layouts
> awful.

That's unrelated to the issue at hand, I think.  Algorithmic
improvements should be done regardless.

Re: Concurrency, again

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> In that case, no, so you cannot win there.  But in other cases, yes.
> So this will allow faster display of at least some pages, I think.

Simple layouts take little time to lay out.  You'd need a pretty
gigantic text for layout time to be noticeable, so it's not worth the
added complexity, in my experience.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Concurrency, again

[joakim]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Huchler <stefan.huchler@mail.de> writes:
>
>> As far as I understand the xwidget webkit browser is "only" some sort of
>> a Client-server thing where you send messages from emacs to another
>> process and it then does stuff and sends maybe answers back.
>
> Yes.  It's pretty opaque.  I think it's possible to see a way to
> interact with these objects more naturally by creating a Javascript
> bridge that would expose the DOM fully to Emacs, and then you could
> rewrite all Emacs commands to do things to that DOM.  Like `C-t'
> (transpose-chars) with "point" inside one of these objects would look at
> the DOM around point, figure out the necessary changes, and then update
> the DOM inside the widget.
>
> And so on.
>
> But it would be...  er...  a major undertaking, and it would always be a
> toy.

The original idea was to implement a gobject introspection bridge
between elisp and gobjects. The beginings of such a bridge is in the
main xwidget branch. The main flaw with it atm is that it doesnt handle
complex types very well. That would require some work. Anyway, if the
bridge could be made to  work, you could use all functions of the gobject.

If I ever have the time to get back to working on the gobject
introspection bridge, I would try implementing it as an emacs module I
think, which some only minor modification to emacs core(I think you need to
do some initialization early on in emacs startup)

>
> And it's irrelevant to the concurrency discussion, really.  :-)

Yes, sorry.

>
>> Sorry I go OT, but the question is, does ewb need some
>> Concurrency? I guess not really cause the javascript stuff is more a
>> pain than a gain for most cases in such environment.
>
> I'm assuming you're referring to eww, and it sure does need concurrency.
> Computing complicated layouts can be slow, and it would be nice if it
> could take without stopping the user from doing other things.
-- 
Joakim Verona
joakim@verona.se
+46705459454

Re: Concurrency, again

[Stefan Huchler]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Sorry I go OT, but the question is, does ewb need some
>> Concurrency? I guess not really cause the javascript stuff is more a
>> pain than a gain for most cases in such environment.
>
> I'm assuming you're referring to eww, and it sure does need concurrency.
> Computing complicated layouts can be slow, and it would be nice if it
> could take without stopping the user from doing other things.

I would prefer to NOT compute/have the layouts at all and only extract
the data and then have client side a own interface, but you would need
to do that for every website seperatly sadly, cause html stinks and we
have no imap for websites/forums/blogs other standart formats. So it
will stay probably a wet dream from me.

Something like the proprietary tapatalk but for emacs and free. There
is the Shimbun module, sadly its in a pretty bad state and depends on
w3m. I would love something like that based on eww.

https://www.emacswiki.org/emacs/WThreeMShimbun

But as starting point some reformating and shortcuts for "next"
(qutebrowser has that) and emacs theming and maybe some macro editing of
websites which you could then maybe save and maybe run as website-hooks
would be pretty nice.

Anyway, you probable cant fix the web completly in a emacs client at
least the absurd processor requirements to transfer 99% text only (with
ads) and some mostly evil or at best pretty useless javascript blobs
that are 99% proprietary.

So therefor such concurrency may be still very important for eww I
guess. So go for it :)

On the locking front if we would have concurrency how would you end
background processes, some sort of proced for emacs processes? Would it
not be better for debugging a "hanging" or in this case a process that
never finish or at least take very long to?

You could probable even have a backlog how long different actions did
take and then look into the ones that took longer.

Theoreticaly that should be possible without concurency right? But it
would be easier to run that logger in the background?

Or does that emacs anyway and I did just not notice it so far?

btw even full fledged external browsers tend to hang/crash,
like conkeror and qutebroweser. Which if you use them inside emacs over
exwm, can make emacs hang and I belive even crash emacs.

So web is today a monster and if emacs could be used to hack around the
website to make them sane "again" it would be a blessing.

Browsers inside Emacs (was Re: Concurrency, again)

[Perry E. Metzger]

On Tue, 18 Oct 2016 16:34:18 +0200 Lars Ingebrigtsen <larsi@gnus.org>
wrote:
> Yes.  It's pretty opaque.  I think it's possible to see a way to
> interact with these objects more naturally by creating a Javascript
> bridge that would expose the DOM fully to Emacs, and then you could
> rewrite all Emacs commands to do things to that DOM.  Like `C-t'
> (transpose-chars) with "point" inside one of these objects would
> look at the DOM around point, figure out the necessary changes, and
> then update the DOM inside the widget.
> 
> And so on.
> 
> But it would be...  er...  a major undertaking, and it would always
> be a toy.
> 
> And it's irrelevant to the concurrency discussion, really.  :-)

It is irrelevant to the concurrency discussion, but I don't think it
is irrelevant overall. I think the ability to do things like edit web
pages inside emacs in WYSYWIG mode with simultaneous changes in the
real web page buffer would be *amazing*, and totally worthwhile, and
it need not be a toy.

There are real applications here beyond just editing web pages and
having Emacs be a really competitive mail reader again even for people
who deal with people that use HTML email all day.

The problem is, though, as I've said, it would be enormous work to do
this, and emacs as it stands wasn't built for this.

Another problem, btw, is that currently there is only Webkit
integration for X users. I get that Windows and Mac are lower
priorities because they aren't free software but it would be important
to have something that is cross platform and not very X
dependent. (Hell, in the medium term, X is getting replaced by Wayland
on GNU/Linux as well.)

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Concurrency, again

[Eli Zaretskii]

> Date: Mon, 17 Oct 2016 13:53:53 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: jwiegley@gmail.com, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Well, active content like video or audio too.

Isn't that already asynch in nature?  IOW, isn't it right that you
write the stream to some system API, which then plays it
asynchronously at its own leisure?

> For good or ill, though, active content is now most of the
> web. Also, the line between JavaScript and active content (like
> HTML5 video) is often thin. If one wanted to have Emacs be a truly
> effective browser, it would need the ability to deal with this.

Emacs can deal with that if the service which performs the job can
present a 'pselect'able interface.  That could be an API or a
subprocess (of any type we support, including a socket).

> I think having Emacs be a truly effective browser is a very useful
> goal btw.

I don't.  I think it's good to have Web browsing facilities in Emacs,
including EWW, but making Emacs a browser that can compete with the
likes of Firefox and Chrome should not be our goal, because we will
never succeed (and we don't have to).

> I love the idea of never having to leave emacs again.

You are "leaving" it already: for compiling, for grepping, for talking
to MTA, for spell-checking, etc.  Emacs cannot do everything, it can
only ever hope to have an interface to everything.  And browse-url is
as good an interface as any.

> Of course, one could do something like wrapping the Webkit instances
> in subprocesses, which might not be awful from an isolation
> viewpoint. Still, that leaves the question of how to handle the
> interaction with such a thing.

The way we always do: through pselect.  Which is a proper interface
for handling multiple streams of data.

> I suppose part of the problem here is that the existing Emacs
> implementation goes back to an era before such things could have even
> been contemplated.

What problem is that?

Re: Concurrency, again

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Well, active content like video or audio too.

Please don't use the word "content" to refer to works or publications.
The term disparages whatever works it is applied to.

See http://gnu.org/philosophy/words-to-avoid.html#Content.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Concurrency, again

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Every open browser tab (should I say "window", this being emacs?) can
  > potentially be doing all sorts of computation in the background via
  > JavaScript,

We should not make Emacs a platform for running nonfree software
sent from servers designed to snoop on people.

An ethical web browser requires lots of work.  We made have IceCat by
changing Firefox so that
(1) it doesn't by default run nonfree programs, and
(2) it blocks various kinds of schemes to track or fingerprint users.

If Emacs is going to have a web browser powerful enough to be vulnerable
to these things, then it too has to be changed to defend against them.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Web browsing (was Re: Concurrency, again)

[Perry E. Metzger]

On Tue, 18 Oct 2016 15:47:43 -0400 Richard Stallman <rms@gnu.org>
wrote:
>   > Every open browser tab (should I say "window", this being
>   > emacs?) can potentially be doing all sorts of computation in
>   > the background via JavaScript,
>
> We should not make Emacs a platform for running nonfree software
> sent from servers designed to snoop on people.

I agree, but that's predicated on first answering the question of how
to make Emacs into a good browser.  Defensive systems (like the open
source Brave browser) have shown that there are mechanisms that can be
used to block bad javascript and links, so the technology is
understood at least in terms of what sorts of things one needs to
block etc.

I'm concerned, however, about the vast amount of work needed to get
Emacs browsing to the point where anti-spyware measures are useful, as
unless Emacs gets powerful enough to really browse the modern web
well, there won't be any issue of people being spied on via Emacs
since there will be no users. This involves things like thinking hard
about the architectural problems involved.

> An ethical web browser requires lots of work.

Sure, and I run loads of anti-spying goop on my browser right now that
I would want in Emacs too. Again, though, it is only possible to add
anti-spying measures to an Emacs that is capable of browsing the web
at all. So I think a good first step is figuring out how that might be
achieved.

> If Emacs is going to have a web browser powerful enough to be
> vulnerable to these things, then it too has to be changed to defend
> against them.

No argument whatsoever.

Again, I think one wants Emacs to be able to do all this (to provide a
really good integrated work environment), but there is a *staggering*
amount of work that would need to be done to make all this happen. I'm
not even sure the dream is realistic. I would like to see it though.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Web browsing (was Re: Concurrency, again)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > We should not make Emacs a platform for running nonfree software
  > > sent from servers designed to snoop on people.

  > I agree, but that's predicated on first answering the question of how
  > to make Emacs into a good browser.

Yes.  But while this ordering of issues is a valid logical sequence,
it would not be a good idea to address them sequentially.

People are talking now about scenarios that would be instances of the
problem, so it is time now to bring the problem into the discussion.



-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Web browsing (was Re: Concurrency, again)

[Perry E. Metzger]

On Wed, 19 Oct 2016 15:57:40 -0400 Richard Stallman <rms@gnu.org>
wrote:
>   > > We should not make Emacs a platform for running nonfree
>   > > software sent from servers designed to snoop on people.  
> 
>   > I agree, but that's predicated on first answering the question
>   > of how to make Emacs into a good browser.  
> 
> Yes.  But while this ordering of issues is a valid logical sequence,
> it would not be a good idea to address them sequentially.
> 
> People are talking now about scenarios that would be instances of
> the problem, so it is time now to bring the problem into the
> discussion.

Again, I won't disagree, but I think doing what is needed is
relatively straightforward. State of the art is blacklisting or
whitelisting various pages and scripts, and I think it will be pretty
easy to put in such mechanisms given all the hooks the rest of the
functionality needs. So read that as "violently agreed, but I'm not
worried that it will be the difficult part of the overall project".

(In general, I'm worried that this is a very very big thing to
attempt. Not the user safety stuff which I feel confident about, but
getting the thing really working at more than a toy level.)

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Web browsing (was Re: Concurrency, again)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > relatively straightforward. State of the art is blacklisting or
  > whitelisting various pages and scripts, and I think it will be pretty
  > easy to put in such mechanisms given all the hooks the rest of the
  > functionality needs.

I hope it is relatively straightforward -- but we shouldn't put it off!
We should not release an Emacs browser which is vulnerable to this
sort of mistreatment; rather, we should attend to this issue
in the first release where it is needed.

What about EWW?  Is it vulnerable to any of the tracking tags?


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Concurrency, again

[Ricardo Wurmus]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Every open browser tab (should I say "window", this being emacs?) can
>   > potentially be doing all sorts of computation in the background via
>   > JavaScript,
>
> We should not make Emacs a platform for running nonfree software
> sent from servers designed to snoop on people.
>
> An ethical web browser requires lots of work.  We made have IceCat by
> changing Firefox so that
> (1) it doesn't by default run nonfree programs, and
> (2) it blocks various kinds of schemes to track or fingerprint users.
>
> If Emacs is going to have a web browser powerful enough to be vulnerable
> to these things, then it too has to be changed to defend against them.

Emacs has the potential for a fully functional browser since the
addition of the xwidget feature.  (The current implementation is not yet
very usable but it doesn’t take much work to make it usable enough for
common browsing tasks.)  We can disable execution of *any* JavaScript by
default (by changing the initial settings in Webkit) and/or add hooks to
run Elisp procedures that decide whether or not to run JavaScript,
e.g. by inspecting license information.

With the Webkit xwidget we can also send our own JavaScript to the
widget to have it executed *instead* of whatever the web page loads.
This would work similar to what the dotjs extension does for
Firefox/Icecat.  I’d like to prepare some patches to implement this.  (I
was told that my copyright assignment is now complete.)

(Blocking fingerprinting and tracking attempts is harder and I don’t know
how to approach this.)

~~ Ricardo

Emacs as browser (was Re: Concurrency, again)

[Perry E. Metzger]

On Wed, 19 Oct 2016 10:52:02 +0200 Ricardo Wurmus
<rekado@elephly.net> wrote:
> Emacs has the potential for a fully functional browser since the
> addition of the xwidget feature.  (The current implementation is
> not yet very usable but it doesn’t take much work to make it usable
> enough for common browsing tasks.)

Just an aside: that only works for X users as things stand. There's
no Webkit integration for users of other windowing platforms at the
moment.


Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs as browser (was Re: Concurrency, again)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Just an aside: that only works for X users as things stand.

I doubt that we can make a graphical browser work on a text terminal.
Aside from that and X, aren't there only secondary platforms such as
nonfree operating systems?

We should not hold back from improving the free world for the sake of
improving the very systems that we aim to eliminate.

, which are less important as targets.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs as browser (was Re: Concurrency, again)

[Perry E. Metzger]

On Wed, 19 Oct 2016 16:02:21 -0400 Richard Stallman <rms@gnu.org>
wrote:
>   > Just an aside: that only works for X users as things stand.  
> 
> I doubt that we can make a graphical browser work on a text
> terminal. Aside from that and X, aren't there only secondary
> platforms such as nonfree operating systems?

Wayland is on the horizon for Linux. X doesn't have an infinite
life ahead of it, it dates from the 1980s and the free OS world
is working to move past it (thus Wayland). I suspect that simply
adopting Webkit (which is LGPL already) directly will probably fix
most potential portability problems, since the Webkit people have to
support whatever is used on most platforms already.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs as browser (was Re: Concurrency, again)

[Eli Zaretskii]

> Date: Wed, 19 Oct 2016 16:38:06 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: rekado@elephly.net, jwiegley@gmail.com, eliz@gnu.org,
>  monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Wayland is on the horizon for Linux. X doesn't have an infinite
> life ahead of it, it dates from the 1980s and the free OS world
> is working to move past it (thus Wayland).

Mostly irrelevant for Emacs, since we have no experts on that on
board, and no development being done in that direction AFAIK.  Without
such development, we will be forever tied to X (and die when it dies),
and any attempts to break those ties will end up like the Cairo build
did, unfortunately, even though the person who worked on that was an
excellent expert (he just left too soon).

IOW, all these nice developments are just pipe dreams, as long as no
one works seriously on adapting Emacs to them.  IMO, working on that
is much more important for the future of Emacs than any other
improvements, including, but not limited to, the "future of Emacs
Lisp" discussions, the "feature/integrated-elpa" discussions, etc.
Developing Emacs without first-class experts on X on board makes no
sense to me.

Re: Emacs as browser (was Re: Concurrency, again)

[Perry E. Metzger]

On Thu, 20 Oct 2016 10:33:52 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > Wayland is on the horizon for Linux. X doesn't have an infinite
> > life ahead of it, it dates from the 1980s and the free OS world
> > is working to move past it (thus Wayland).
>
> Mostly irrelevant for Emacs, since we have no experts on that on
> board, and no development being done in that direction AFAIK.
> Without such development, we will be forever tied to X (and die
> when it dies), and any attempts to break those ties will end up
> like the Cairo build did, unfortunately, even though the person who
> worked on that was an excellent expert (he just left too soon).
>
> IOW, all these nice developments are just pipe dreams, as long as no
> one works seriously on adapting Emacs to them.

I have never really looked at the terminal layer stuff. (There are
four front ends now, right? MS Windows, X, NextStep and tty, yes?)

Is there any documentation about the internal interfaces between the
terminal layer and the back end? I doubt I'm skilled enough in
something like Wayland to do this work but I'd like to get a bit of a
sense of how awful the work is.

And what *did* happen to the Cairo stuff? Cairo would make Wayland
easy if I recall correctly.

> IMO, working on that is much more important for the future of Emacs
> than any other improvements, including, but not limited to, the
> "future of Emacs Lisp" discussions, the "feature/integrated-elpa"
> discussions, etc.  Developing Emacs without first-class experts on X
> on board makes no sense to me.

Perhaps no first class experts in the area are aware that Emacs needs
the help?

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs as browser (was Re: Concurrency, again)

[Eli Zaretskii]

> Date: Thu, 20 Oct 2016 11:15:27 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: rekado@elephly.net, jwiegley@gmail.com, emacs-devel@gnu.org,
>  rms@gnu.org, monnier@iro.umontreal.ca
> 
> (There are four front ends now, right? MS Windows, X, NextStep and
> tty, yes?)

Yes.  Although the TTY back-end has some quirks in the MS-Windows
build, because termcap/terminfo is not supported by the Windows
console (or wasn't until Windows 10.1).

> Is there any documentation about the internal interfaces between the
> terminal layer and the back end?

Look at 'struct redisplay_interface' (for X it gets populated around
line 12470 of xterm.c) and at hooks in 'struct terminal' (populated
for X in x_create_terminal).  TTYs don't have 'struct
redisplay_interface' (for historical reasons), but instead call the
corresponding functions directly.

> I doubt I'm skilled enough in something like Wayland to do this work
> but I'd like to get a bit of a sense of how awful the work is.

Thanks for trying.

> And what *did* happen to the Cairo stuff?

I has bugs that we cannot fix because no one knowns enough about Cairo
drawing.  The person who wrote the code left the Emacs development
short time after merging the code.

> Cairo would make Wayland easy if I recall correctly.

That was the idea behind introducing it.

> > IMO, working on that is much more important for the future of Emacs
> > than any other improvements, including, but not limited to, the
> > "future of Emacs Lisp" discussions, the "feature/integrated-elpa"
> > discussions, etc.  Developing Emacs without first-class experts on X
> > on board makes no sense to me.
> 
> Perhaps no first class experts in the area are aware that Emacs needs
> the help?

Perhaps.

Cairo bugs (was Re: Emacs as browser (was Re: Concurrency, again))

[Perry E. Metzger]

On Thu, 20 Oct 2016 18:42:07 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > And what *did* happen to the Cairo stuff?  
> 
> I has bugs that we cannot fix because no one knowns enough about
> Cairo drawing.  The person who wrote the code left the Emacs
> development short time after merging the code.
> 
> > Cairo would make Wayland easy if I recall correctly.  
> 
> That was the idea behind introducing it.
> 
> > > IMO, working on that is much more important for the future of
> > > Emacs than any other improvements, including, but not limited
> > > to, the "future of Emacs Lisp" discussions, the
> > > "feature/integrated-elpa" discussions, etc.  Developing Emacs
> > > without first-class experts on X on board makes no sense to
> > > me.  
> > 
> > Perhaps no first class experts in the area are aware that Emacs
> > needs the help?  
> 
> Perhaps.

So, I believe Keith Packard (who co-created Cairo and is a very
expert GUI developer) uses Emacs. Perhaps if one of the Emacs
maintainers appealed to him, he might be able to find someone able to
help?

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs as browser (was Re: Concurrency, again)

[Dov Grobgeld]

[-- Attachment #1: Type: text/plain, Size: 2559 bytes --]

For whatever it is worth, I checked out the branch
origin/old-branches/cairo compiled it and it seemed to work just fine under
Fedora 24. I couldn't reproduce either of the bugs #20997 nor #23925.

In any case these two bugs both discuss refresh issues, which is outside of
the scope of cairo which is a rendering engine.

My feeling is that the only issue in the bug tracker that you may need an
"cairo expert" for, is the memory leak in #22961. The rest of the bugs are
more related to exposure triggered redraws and interaction with the window
manager. These are certainly related to the cairo branch, but have nothing
to do with cairo per se.

Regards,
Dov


On Thu, Oct 20, 2016 at 6:42 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Thu, 20 Oct 2016 11:15:27 -0400
> > From: "Perry E. Metzger" <perry@piermont.com>
> > Cc: rekado@elephly.net, jwiegley@gmail.com, emacs-devel@gnu.org,
> >  rms@gnu.org, monnier@iro.umontreal.ca
> >
> > (There are four front ends now, right? MS Windows, X, NextStep and
> > tty, yes?)
>
> Yes.  Although the TTY back-end has some quirks in the MS-Windows
> build, because termcap/terminfo is not supported by the Windows
> console (or wasn't until Windows 10.1).
>
> > Is there any documentation about the internal interfaces between the
> > terminal layer and the back end?
>
> Look at 'struct redisplay_interface' (for X it gets populated around
> line 12470 of xterm.c) and at hooks in 'struct terminal' (populated
> for X in x_create_terminal).  TTYs don't have 'struct
> redisplay_interface' (for historical reasons), but instead call the
> corresponding functions directly.
>
> > I doubt I'm skilled enough in something like Wayland to do this work
> > but I'd like to get a bit of a sense of how awful the work is.
>
> Thanks for trying.
>
> > And what *did* happen to the Cairo stuff?
>
> I has bugs that we cannot fix because no one knowns enough about Cairo
> drawing.  The person who wrote the code left the Emacs development
> short time after merging the code.
>
> > Cairo would make Wayland easy if I recall correctly.
>
> That was the idea behind introducing it.
>
> > > IMO, working on that is much more important for the future of Emacs
> > > than any other improvements, including, but not limited to, the
> > > "future of Emacs Lisp" discussions, the "feature/integrated-elpa"
> > > discussions, etc.  Developing Emacs without first-class experts on X
> > > on board makes no sense to me.
> >
> > Perhaps no first class experts in the area are aware that Emacs needs
> > the help?
>
> Perhaps.
>
>

[-- Attachment #2: Type: text/html, Size: 4075 bytes --]

Re: Emacs as browser (was Re: Concurrency, again)

[Paul Eggert]

On 10/21/2016 12:31 AM, Dov Grobgeld wrote:
> My feeling is that the only issue in the bug tracker that you may need 
> an "cairo expert" for, is the memory leak in #22961. The rest of the 
> bugs are more related to exposure triggered redraws and interaction 
> with the window manager. These are certainly related to the cairo 
> branch, but have nothing to do with cairo per se.

I assume you meant Bug#22691, not Bug#22961.

Did you build the origin/old-branches/cairo branch with ImageMagick as 
well? The master-branch Cairo problems I ran into involved interaction 
with ImageMagick (Bug#21110, Bug#22442). Or is that interaction also a 
refresh issue? (Please bear with me, as I know little about Cairo.)

Re: Emacs as browser (was Re: Concurrency, again)

[Dov Grobgeld]

[-- Attachment #1: Type: text/plain, Size: 1393 bytes --]

Yes, 22691.

And I was indeed able to reproduce the png display bug with ImageMagick.
I'll try to look in to it.

And regarding cairo, from a top level view it is quite simple. It receives
a memory buffer and draws pixels in the buffer. The memory buffer may be
either a "normal" memory area, or together with gdk an GdkWindow. Its
painting model is PostScript (or svg) like, e.g. moveto, lineto, curveto,
stroke, fill, and draw glyphs. It is simpler than than the X11 drawing
model as there are e.g. no XOR modes nor 8-bit index modes, etc. The rest
is details. :-)

Regards,
Dov



On Fri, Oct 21, 2016 at 10:43 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:

> On 10/21/2016 12:31 AM, Dov Grobgeld wrote:
>
>> My feeling is that the only issue in the bug tracker that you may need an
>> "cairo expert" for, is the memory leak in #22961. The rest of the bugs are
>> more related to exposure triggered redraws and interaction with the window
>> manager. These are certainly related to the cairo branch, but have nothing
>> to do with cairo per se.
>>
>
> I assume you meant Bug#22691, not Bug#22961.
>
> Did you build the origin/old-branches/cairo branch with ImageMagick as
> well? The master-branch Cairo problems I ran into involved interaction with
> ImageMagick (Bug#21110, Bug#22442). Or is that interaction also a refresh
> issue? (Please bear with me, as I know little about Cairo.)
>
>

[-- Attachment #2: Type: text/html, Size: 2361 bytes --]

Re: Emacs as browser (was Re: Concurrency, again)

[Dov Grobgeld]

[-- Attachment #1: Type: text/plain, Size: 2456 bytes --]

I had a look at the image magick bug and I'm afraid that I don't understand
what is going on, but it seems like it has nothing to do with cairo.

This is what I have found.

   - The ImageMagick image is loaded in
   image.c:image_create_x_image_and_pixmap() which causes XCreateImage() to be
   called. The image is then populated in imagemagick_load_image() through
   calls to XPutPixel() with data from the imagemagick image.
   - The image is then displayed through a call to XCopyArea() in xterm.c ,
   but instead of showing the original image, garbage (copies of various
   rectangles pieces of the display) is shown instead.

My guess is that either something went wrong with the creation of the ximg
or it was released before XCopyArea() was called.

I'd be happy to hear if someone has any idea of what is going on and how to
debug this.

(Btw, how do you print a Lisp_Object in gdb? Probably a FAQ...)

Regards,

Dov



On Sun, Oct 23, 2016 at 9:23 AM, Dov Grobgeld <dov.grobgeld@gmail.com>
wrote:

> Yes, 22691.
>
> And I was indeed able to reproduce the png display bug with ImageMagick.
> I'll try to look in to it.
>
> And regarding cairo, from a top level view it is quite simple. It receives
> a memory buffer and draws pixels in the buffer. The memory buffer may be
> either a "normal" memory area, or together with gdk an GdkWindow. Its
> painting model is PostScript (or svg) like, e.g. moveto, lineto, curveto,
> stroke, fill, and draw glyphs. It is simpler than than the X11 drawing
> model as there are e.g. no XOR modes nor 8-bit index modes, etc. The rest
> is details. :-)
>
> Regards,
> Dov
>
>
>
> On Fri, Oct 21, 2016 at 10:43 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
>
>> On 10/21/2016 12:31 AM, Dov Grobgeld wrote:
>>
>>> My feeling is that the only issue in the bug tracker that you may need
>>> an "cairo expert" for, is the memory leak in #22961. The rest of the bugs
>>> are more related to exposure triggered redraws and interaction with the
>>> window manager. These are certainly related to the cairo branch, but have
>>> nothing to do with cairo per se.
>>>
>>
>> I assume you meant Bug#22691, not Bug#22961.
>>
>> Did you build the origin/old-branches/cairo branch with ImageMagick as
>> well? The master-branch Cairo problems I ran into involved interaction with
>> ImageMagick (Bug#21110, Bug#22442). Or is that interaction also a refresh
>> issue? (Please bear with me, as I know little about Cairo.)
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 3927 bytes --]

Re: Emacs as browser (was Re: Concurrency, again)

[Paul Eggert]

Dov Grobgeld wrote:
> (Btw, how do you print a Lisp_Object in gdb? Probably a FAQ...)

source .gdbinit

then "pp lispobj".

Re: Emacs as browser (was Re: Concurrency, again)

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

>
> > Wayland is on the horizon for Linux. X doesn't have an infinite
> > life ahead of it, it dates from the 1980s and the free OS world
> > is working to move past it (thus Wayland).
>
> Mostly irrelevant for Emacs, since we have no experts on that on
> board, and no development being done in that direction AFAIK.  Without
> such development, we will be forever tied to X (and die when it dies),
> and any attempts to break those ties will end up like the Cairo build
> did, unfortunately, even though the person who worked on that was an
> excellent expert (he just left too soon).
>

Just putting it out there, maybe it's irrelevant or does not make sense
but: how about using Qt?

From my own experience of using it, it's the clear winner when you want
something simple (yet powerful) & portable. That way you are not tied to X
or the win32 api or whatever (tho I agree you are then tied to Qt).

Also, the community is huge and I'd expect much more people willing to
fiddle with Qt than with X.

My 0.02$
Philippe

[-- Attachment #2: Type: text/html, Size: 1405 bytes --]

Re: Emacs as browser (was Re: Concurrency, again)

[Perry E. Metzger]

On Thu, 20 Oct 2016 20:15:11 +0200 Philippe Vaucher
<philippe.vaucher@gmail.com> wrote:
> Just putting it out there, maybe it's irrelevant or does not make
> sense but: how about using Qt?

Emacs already has multiple "terminals" for different platforms. Qt is
really C++ oriented too, not to mention that it uses its own weird
preprocessor in addition, and Emacs is really a C app. It's also not
truly multiplatform — a Qt app doesn't look as good as native toolkits
do on platforms other than X.

Though I doubt anyone would be upset about a Qt front end being added
by someone if they wanted to work on it.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs as browser (was Re: Concurrency, again)

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

>
> Emacs already has multiple "terminals" for different platforms. Qt is
> really C++ oriented too, not to mention that it uses its own weird
> preprocessor in addition, and Emacs is really a C app.


I agree that adding a C++ dependency is probably complicated in a project
like Emacs.



> It's also not truly multiplatform — a Qt app doesn't look as good as
> native toolkits
> do on platforms other than X.


I think that is somewhat of an argument of the past. Qt styles (and ability
to use css stylesheets) really allow for widgets to look just like native.
See http://doc.qt.io/qt-5/stylesheet.html or
http://doc.qt.io/qt-5/qstyle.html#details for screenshots examples (see
QMacStyle. QWindowsStyle & friends).

Also, I'm not sure that "not looking as good as native toolkits" is really
important if it simplify the implementation, but it's a big if :-)

But I disgress, as Eli said this is a pipe dream in its current form :-)

Thank you for your input.
Philippe

[-- Attachment #2: Type: text/html, Size: 1631 bytes --]

Re: Emacs as browser (was Re: Concurrency, again)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Please note that Qt is a competitor to a GNU program, GTK+.
So our primary effort should be supporting GTK+.

It is ok to support Qt also.  It is free software and there
is nothing wrong with it.  We simply should not prefer it over GTK+.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs as browser (was Re: Concurrency, again)

[Eli Zaretskii]

> From: Philippe Vaucher <philippe.vaucher@gmail.com>
> Date: Thu, 20 Oct 2016 20:15:11 +0200
> Cc: "Perry E. Metzger" <perry@piermont.com>, rekado@elephly.net, jwiegley@gmail.com, 
> 	Emacs developers <emacs-devel@gnu.org>, rms@gnu.org, 
> 	Stefan Monnier <monnier@iro.umontreal.ca>
> 
> Just putting it out there, maybe it's irrelevant or does not make sense but: how about using Qt?

If someone is willing to work on that, sure.  Otherwise, it's just
another pipe dream.

(Sorry for being blunt, but I had my share of endless discussions
about these issues that led nowhere.)

Qt is C++, so the job is more complex, btw.

Re: Emacs as browser (was Re: Concurrency, again)

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 758 bytes --]

>
> > Just putting it out there, maybe it's irrelevant or does not make sense
> but: how about using Qt?
>
> If someone is willing to work on that, sure.  Otherwise, it's just
> another pipe dream.
>
> (Sorry for being blunt, but I had my share of endless discussions
> about these issues that led nowhere.)
>

Understandable.

I was curious because to me, writing the GUI part for each platforms is
much more work than writing it in Qt and maybe adapting some parts for
special cases... and given I saw a discussion about targeting Wayland, I
thought it'd be better to target something that was designed to be
cross-platform.

Anyway, you are right about the fact that it is useless to discuss this
without showing some work in that direction :-)

Philippe

[-- Attachment #2: Type: text/html, Size: 1111 bytes --]

Re: Emacs as browser (was Re: Concurrency, again)

[Eli Zaretskii]

> From: Philippe Vaucher <philippe.vaucher@gmail.com>
> Date: Thu, 20 Oct 2016 22:33:38 +0200
> Cc: "Perry E. Metzger" <perry@piermont.com>, rekado@elephly.net, jwiegley@gmail.com, 
> 	Emacs developers <emacs-devel@gnu.org>, rms@gnu.org, 
> 	Stefan Monnier <monnier@iro.umontreal.ca>
> 
> I was curious because to me, writing the GUI part for each platforms is much more work than writing it in Qt
> and maybe adapting some parts for special cases...

You seem to have in mind programming at raw Xlib level.  Such a
"toolkit-less" configuration indeed exists in Emacs, but it is no
longer developed, and rarely used.  The other GUI configurations all
use some kind of toolkit, and toolkits generally present similar APIs
for similar things.  So adapting Emacs to some toolkit is not harder
than adapting it to Qt.  The only advantage of Qt is that it's
multi-platform, but GTK is also multi-platform.

Tookits (was Re: Emacs as browser)

[Perry E. Metzger]

On Fri, 21 Oct 2016 09:53:15 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> The only advantage of Qt is that it's multi-platform, but GTK is
> also multi-platform.

Well, to be precise, it seems to be "mostly" multiplatform. By this I
mean that it appears that it is possible for features that use the API
to build on the X side but not on other systems. For example,
apparently the GTK based xwidgets stuff doesn't build on OS X, or at
least doesn't currently.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Tookits (was Re: Emacs as browser)

[Eli Zaretskii]

> Date: Fri, 21 Oct 2016 08:39:26 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Philippe Vaucher <philippe.vaucher@gmail.com>, rekado@elephly.net,
>  jwiegley@gmail.com, emacs-devel@gnu.org, rms@gnu.org,
>  monnier@iro.umontreal.ca
> 
> On Fri, 21 Oct 2016 09:53:15 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > The only advantage of Qt is that it's multi-platform, but GTK is
> > also multi-platform.
> 
> Well, to be precise, it seems to be "mostly" multiplatform. By this I
> mean that it appears that it is possible for features that use the API
> to build on the X side but not on other systems. For example,
> apparently the GTK based xwidgets stuff doesn't build on OS X, or at
> least doesn't currently.

Only because no one wrote the Emacs supporting code for the other
platforms.

GTK is available on all platforms we support with GUI frames, so it's
"only" a matter of integrating it into Emacs on the other platforms.

Re: Tookits (was Re: Emacs as browser)

[Perry E. Metzger]

On Fri, 21 Oct 2016 15:57:44 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Fri, 21 Oct 2016 08:39:26 -0400
> > From: "Perry E. Metzger" <perry@piermont.com>
> > Cc: Philippe Vaucher <philippe.vaucher@gmail.com>,
> > rekado@elephly.net, jwiegley@gmail.com, emacs-devel@gnu.org,
> > rms@gnu.org, monnier@iro.umontreal.ca
> > 
> > On Fri, 21 Oct 2016 09:53:15 +0300 Eli Zaretskii <eliz@gnu.org>
> > wrote:  
> > > The only advantage of Qt is that it's multi-platform, but GTK is
> > > also multi-platform.  
> > 
> > Well, to be precise, it seems to be "mostly" multiplatform. By
> > this I mean that it appears that it is possible for features that
> > use the API to build on the X side but not on other systems. For
> > example, apparently the GTK based xwidgets stuff doesn't build on
> > OS X, or at least doesn't currently.  
> 
> Only because no one wrote the Emacs supporting code for the other
> platforms.
> 
> GTK is available on all platforms we support with GUI frames, so
> it's "only" a matter of integrating it into Emacs on the other
> platforms.

Well, exactly. That was my point. It isn't perfectly portable, you
need to write some support code for it to work on each platform.

(To my knowledge, the only "really" multiplatform GUIs right now are
written for Electron and systems like it, which is to say, web apps
running on the desktop. *Shudder*.)

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs as browser (was Re: Concurrency, again)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Wayland is on the horizon for Linux.

Is Wayland tied specifically to Linux?

					 X doesn't have an infinite
  > life ahead of it, it dates from the 1980s and the free OS world
  > is working to move past it (thus Wayland). I suspect that simply
  > adopting Webkit (which is LGPL already) directly will probably fix
  > most potential portability problems, since the Webkit people have to
  > support whatever is used on most platforms already.

That seems plausible to me -- provided you really meant "GNU/Linux" in
the first line.  If it was correct to write "Linux", that implies it
is not so portable.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs as browser (was Re: Concurrency, again)

[Perry E. Metzger]

On Thu, 20 Oct 2016 14:04:15 -0400 Richard Stallman <rms@gnu.org>
wrote:
>   > Wayland is on the horizon for Linux.  
> 
> Is Wayland tied specifically to Linux?

Wayland is sort of an X successor based on much more modern
principles. It will run on other free operating systems as well as
GNU/Linux.

> 					 X doesn't have an infinite
>   > life ahead of it, it dates from the 1980s and the free OS world
>   > is working to move past it (thus Wayland). I suspect that simply
>   > adopting Webkit (which is LGPL already) directly will probably
>   > fix most potential portability problems, since the Webkit
>   > people have to support whatever is used on most platforms
>   > already.  
> 
> That seems plausible to me -- provided you really meant "GNU/Linux"
> in the first line.

I did.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs as browser (was Re: Concurrency, again)

[Paul Eggert]

On 10/20/2016 11:04 AM, Richard Stallman wrote:
> Is Wayland tied specifically to Linux?

No in theory, mostly yes in practice so far. Although Wayland is 
intended to be portable among Unix-like systems, it's currently mostly 
limited to the Linux kernel because its reference implementation is 
being developed on GNU/Linux.

Wayland itself is a protocol. Its reference implementation Weston relies 
on a compositor that in native mode currently requires Kernel Mode 
Setting (KMS) and the evdev input event interface, which were originally 
specific to Linux although there are efforts to port them to other 
kernels. Although it's not intended that Weston be specific to Linux in 
the long run (e.g., there is a DragonFlyBSD port) almost all Wayland and 
Weston users are running on GNU/Linux right now.

Also, Weston can run in backward compatibility mode attached to an X 
server, and in this mode it runs on kernels other than Linux (e.g., 
there is a unsupported port to FreeBSD). I don't think this approach is 
much used outside of GNU/Linux either, though.

Re: Emacs as browser (was Re: Concurrency, again)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No in theory, mostly yes in practice so far. Although Wayland is 
  > intended to be portable among Unix-like systems, it's currently mostly 
  > limited to the Linux kernel because its reference implementation is 
  > being developed on GNU/Linux.

Do you think there is a great difficulty in making other kernels
(BSD, Hurd) support Wayland?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs as browser (was Re: Concurrency, again)

[Perry E. Metzger]

On Thu, 20 Oct 2016 21:45:59 -0400 Richard Stallman <rms@gnu.org>
wrote:
>   > No in theory, mostly yes in practice so far. Although Wayland
>   > is intended to be portable among Unix-like systems, it's
>   > currently mostly limited to the Linux kernel because its
>   > reference implementation is being developed on GNU/Linux.  
> 
> Do you think there is a great difficulty in making other kernels
> (BSD, Hurd) support Wayland?

So it depends upon what you mean by Wayland. As with X, Wayland
is partially a suite of protocols and part a set of reference
implementations of those protocols. The current reference
implementations are fairly Linux (kernel) dependent. That will
doubtless change, because all free OSes are ultimately going to be
under pressure to have working ports.

Regardless, I think it would be good to recruit some of the Cairo
devs (some of whom are Emacs users) to help clean up the Cairo code
if they can be persuaded to assist, because that would make Emacs work
better on both X and Wayland.


Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Concurrency, again

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Do., 13. Okt. 2016 um
17:07 Uhr:

> > I also know that in the not that distant future processors with
> > hundreds or thousands of cores are going to be common, so there will
> > only be more demand with time for the ability to do real concurrency
> > for performance. Solving the parallel execution issue may be a matter
> > of importance if (in the future) people are going to do things like
> > using Emacs as a primary web browser and development environment.
>
> "Luckily" here, it seems that (usefully) increasing the number of cores
> is not that easy, so this future is still a bit further.  But I'll note
> also that concurrency is not solution for parallelism.
>
> IOW if we want to take advantage of tens of cores in Emacs, we have to
> think very differently.  A language with STM or with "communicating
> agents" won't cut it, I think.
>

These approaches are widely used in practice, probably much more widely
than Emacs. They are good enough for Emacs.

[-- Attachment #2: Type: text/html, Size: 1642 bytes --]

Re: Concurrency, again

[joakim]

"Perry E. Metzger" <perry@piermont.com> writes:

> On Wed, 12 Oct 2016 15:43:12 -0400 Stefan Monnier
> <monnier@IRO.UMontreal.CA> wrote:
>> - I want semantic-mode to scan my buffers in the background and to
>> scan several buffers at the same time, to make use of my 8 cores.
>>   The concurrency branch doesn't really try to solve this problem.
>>   I'm not sure what's the best way to solve this problem, and I
>> don't even know what would be a "simple" way to solve this problem.
>
> One possibility: move to a multiple processes paradigm where each
> buffer has an associated process, and global variable values are
> propagated via message passing to a master process. Think Hoare's CSP
> or Go or Erlang for precedents, though perhaps with more sugaring to
> make it easy for Emacs users to not think much about global state.

It would be useful if this kind of model could be utilized over a
network connection. For instance to share buffers between emacsen
running on different machines.

>
> This sort of model might be especially important if Emacs starts
> being used as a serious web browser by lots of its community, given
> how much CPU modern browsers can eat.
>
> Perry

-- 
Joakim Verona
joakim@verona.se

Re: Concurrency, again

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

I just had a nice side discussion with Eli that has allayed many of my fears
at the prospect of merging this branch.

What we're talking about here is not the kind of concurrency that leads to
nightmares involving corruption of global state, but cooperative concurrency.
Think of it as "timers, done right", if that helps. It would make a library
like emacs-deferred trivial to write, for example, which today is tortuously
implemented on top of the hackish timers we have now.

Tom, if I can find some volunteers to help, would you be willing to see the
work of merging this into master through to completion?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Concurrency, again

[SAKURAI Masashi]

At Wed, 12 Oct 2016 11:19:11 -0700,
John Wiegley wrote:
> :
> What we're talking about here is not the kind of concurrency that leads to
> nightmares involving corruption of global state, but cooperative concurrency.
> Think of it as "timers, done right", if that helps. It would make a library
> like emacs-deferred trivial to write, for example, which today is tortuously
> implemented on top of the hackish timers we have now.

emacs-deferred is built on tick-timer function (as browser's Promise
implementations do), and provides async APIs not only to run async
tasks but also to connect them without callback continuation.

I have made many programs on it and built async RPC to use desired
functions of other language environments from emacs. In my experience
so far, emacs-deferred has worked well. (I know some issues on github,
but now I have no time to revise it, such as lexbind and cl-lib.)

Then, I have learned that race bug, dead-lock and not-reproducible bug
are terrible even in the single-thread model, and I realized we should
choose concurrent programming without shared mutable state or locking.

So, I think multi-process model with IPC messaging is better for
emacs, like browser's web-worker. I think they have helpful knowledge
about retrofitted concurrency: why did they design the isolated thread
with messaging, not simple thread API?

--
SAKURAI, Masashi (family, given)
m.sakurai@kiwanami.net

Re: Concurrency, again

[Perry E. Metzger]

On Sat, 15 Oct 2016 13:45:58 +0900 SAKURAI Masashi
<m.sakurai@kiwanami.net> wrote:
> Then, I have learned that race bug, dead-lock and not-reproducible
> bug are terrible even in the single-thread model, and I realized we
> should choose concurrent programming without shared mutable state
> or locking.
>
> So, I think multi-process model with IPC messaging is better for
> emacs,

This is also my experience. However, figuring out how to make Emacs
function in such an environment is non-trivial. That said, it might
well be worth it.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Concurrency, again

[Tom Tromey]

>>>>> "John" == John Wiegley <jwiegley@gmail.com> writes:

John> Tom, if I can find some volunteers to help, would you be willing
John> to see the work of merging this into master through to completion?

I can help to some degree, but I definitely can't do it all myself.

There are two main issues.

First, it has to be rebased.  There have been a lot of changes since it
was written -- I tried a rebase last year but it was pretty painful, I
think I stopped while fixing up the process changes.  The branch is also
kind of messy, since I think I mostly didn't write ChangeLogs and there
have been several merges from master.  So I suppose it would need some
cleanups as well.

Second, last time this was discussed there were a reasonably large
number of comments on features that had to be done in order to merge.  I
haven't worked on any of those.

BTW Ken Raeburn was also interested in this branch.

Tom

Re: Concurrency, again

[Eli Zaretskii]

> From: Tom Tromey <tom@tromey.com>
> Date: Mon, 17 Oct 2016 21:14:55 -0600
> Cc: emacs-devel@gnu.org
> 
> >>>>> "John" == John Wiegley <jwiegley@gmail.com> writes:
> 
> John> Tom, if I can find some volunteers to help, would you be willing
> John> to see the work of merging this into master through to completion?
> 
> I can help to some degree, but I definitely can't do it all myself.

Thanks in advance.

> There are two main issues.
> 
> First, it has to be rebased.  There have been a lot of changes since it
> was written -- I tried a rebase last year but it was pretty painful, I
> think I stopped while fixing up the process changes.

Was it before or after Ken Raeburn did his rebase?

> Second, last time this was discussed there were a reasonably large
> number of comments on features that had to be done in order to merge.

Can you point to this discussion?

> BTW Ken Raeburn was also interested in this branch.

Yes, indeed.

Re: Concurrency, again

[Ken Raeburn]

> On Oct 17, 2016, at 23:14, Tom Tromey <tom@tromey.com> wrote:
> 
>>>>>> "John" == John Wiegley <jwiegley@gmail.com> writes:
> 
> John> Tom, if I can find some volunteers to help, would you be willing
> John> to see the work of merging this into master through to completion?
> 
> I can help to some degree, but I definitely can't do it all myself.
> 
> There are two main issues.
> 
> First, it has to be rebased.  There have been a lot of changes since it
> was written -- I tried a rebase last year but it was pretty painful, I
> think I stopped while fixing up the process changes.

I’ve recently been looking at doing another merge, actually.  And yeah, process.c is the piece I haven’t finished.

>  The branch is also
> kind of messy, since I think I mostly didn't write ChangeLogs and there
> have been several merges from master.  So I suppose it would need some
> cleanups as well.

Yeah, the lack of documentation on specifics made my last merge (about a year ago) challenging.  In fact, in order to get a better understanding of it, I set up a private repository where I broke up the concurrency branch changes — not each commit, but the overall diff from the branch point — into little pieces, separating straightforward but pervasive changes from more subtle changes so I could figure out what the latter were doing.  The result of my merge that I checked in was in fact the result of rebasing those little changes onto the latest master branch, fixing them up as necessary as I went along.  (But the resulting body of code was checked in as a merge commit so as to preserve history of the work on the branch — what change was put in, who fixed what bugs, etc; I wasn’t about to unilaterally throw that away.  It was just a slow, tedious, manual merge process instead of an automated one.)

> Second, last time this was discussed there were a reasonably large
> number of comments on features that had to be done in order to merge.  I
> haven't worked on any of those.

I collected some notes from those past discussions, though it was often unclear whether there was consensus on certain things being needed or whether they were just ideas being kicked around.  My list, aside from the note to go back and review the discussions again :-) …

 * collapse systhread and thread, adding w32threads.c to emulate the pthread interface
 * header inclusion order requirement is weird; can generating one big header help?
 * field names and faking globals with macros: maybe change m_ prefix, maybe add BVAR-like macro
 * one thread per terminal?
 * file notifications and such shouldn’t go through same queue as keyboard events
 * interaction of SIGCHLD handling and threads?
 * handle errors in lower-level routines like sys_cond_init
 * ns_select needs fixing

There are also uses of jmp_buf in places that should be examined carefully, like stack overflow handling, keyboard.c:getcjmp, and image handling code.

Ken

Re: Concurrency, again

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Tue, 18 Oct 2016 03:58:04 -0400
> Cc: emacs-devel@gnu.org
> 
> I collected some notes from those past discussions, though it was often unclear whether there was consensus on certain things being needed or whether they were just ideas being kicked around.  My list, aside from the note to go back and review the discussions again :-) …
> 
>  * collapse systhread and thread, adding w32threads.c to emulate the pthread interface

This can be done later, it's not a critical issue, IMO.

>  * header inclusion order requirement is weird; can generating one big header help?

I'm not sure I understand what inclusion order is being alluded to
here.  Can you elaborate?

>  * field names and faking globals with macros: maybe change m_ prefix, maybe add BVAR-like macro

Again, not critical.

>  * one thread per terminal?

Why?

>  * file notifications and such shouldn’t go through same queue as keyboard events

Why?

>  * interaction of SIGCHLD handling and threads?

Details?

>  * handle errors in lower-level routines like sys_cond_init
>  * ns_select needs fixing
> 
> There are also uses of jmp_buf in places that should be examined carefully, like stack overflow handling, keyboard.c:getcjmp, and image handling code.

I'd say, insert appropriate FIXME comments where these issues pop up,
and leave this for later, unless the solution is already known.

Thanks.

Re: Concurrency, again

[Ken Raeburn]

> On Oct 18, 2016, at 05:22, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Tue, 18 Oct 2016 03:58:04 -0400
>> Cc: emacs-devel@gnu.org
>> 
>> I collected some notes from those past discussions, though it was often unclear whether there was consensus on certain things being needed or whether they were just ideas being kicked around.  My list, aside from the note to go back and review the discussions again :-) …
>> 
>> * collapse systhread and thread, adding w32threads.c to emulate the pthread interface
> 
> This can be done later, it's not a critical issue, IMO.
> 
>> * header inclusion order requirement is weird; can generating one big header help?
> 
> I'm not sure I understand what inclusion order is being alluded to
> here.  Can you elaborate?

I think it’s the ordering and recursive inclusion involving thread.h relative to the other headers.  For example lisp.h includes thread.h which includes sysselect.h which includes lisp.h; thread.h uses struct vectorlike_header, so it has to be included in lisp.h after that structure is defined but before struct thread_state gets used; thread.h also includes regex.h which includes lisp.h.  You commented at one point, “We now have an unfortunate situation whereby lisp.h cannot be included before some of the other headers, due to this.”

> 
>> * field names and faking globals with macros: maybe change m_ prefix, maybe add BVAR-like macro
> 
> Again, not critical.


>> * one thread per terminal?
> 
> Why?
> 
>> * file notifications and such shouldn’t go through same queue as keyboard events
> 
> Why?

Stefan’s message: http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00755.html

> 
>> * interaction of SIGCHLD handling and threads?
> 
> Details?

Your message http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00738.html raised questions.  If they were ever satisfactorily answered, I overlooked it when putting together my notes….

> 
>> * handle errors in lower-level routines like sys_cond_init
>> * ns_select needs fixing
>> 
>> There are also uses of jmp_buf in places that should be examined carefully, like stack overflow handling, keyboard.c:getcjmp, and image handling code.
> 
> I'd say, insert appropriate FIXME comments where these issues pop up,
> and leave this for later, unless the solution is already known.

It’s easy enough to disable stack overflow checking when enabling thread support.  If only one thread is allowed into the image processing code at a time (i.e., don’t release the global lock for that code) then that’s probably fine for now, and there’s probably other state there that different threads shouldn’t be mucking around with in parallel.  The keyboard.c one is the only one I’m a bit concerned about, in part because I haven’t looked at it.

Re: Concurrency, again

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Tue, 18 Oct 2016 06:08:49 -0400
> Cc: tom@tromey.com,
>  emacs-devel@gnu.org
> 
> >> * header inclusion order requirement is weird; can generating one big header help?
> > 
> > I'm not sure I understand what inclusion order is being alluded to
> > here.  Can you elaborate?
> 
> I think it’s the ordering and recursive inclusion involving thread.h relative to the other headers.  For example lisp.h includes thread.h which includes sysselect.h which includes lisp.h; thread.h uses struct vectorlike_header, so it has to be included in lisp.h after that structure is defined but before struct thread_state gets used; thread.h also includes regex.h which includes lisp.h.  You commented at one point, “We now have an unfortunate situation whereby lisp.h cannot be included before some of the other headers, due to this.”

This could be solved by moving parts of thread.h into lisp.h, with the
appropriate #ifdef guards.  I think Lisp objects should be in lisp.h
anyway, even if they are optional; anything else is confusing.

> >> * one thread per terminal?
> > 
> > Why?
> > 
> >> * file notifications and such shouldn’t go through same queue as keyboard events
> > 
> > Why?
> 
> Stefan’s message: http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00755.html

I don't see that as a critical problem, perhaps because we don't yet
realize how serious it can be.  The whole purpose of trying to merge
the concurrency branch is to collect practical experience as to what
should and shouldn't be in this kind of Emacs feature.  So I'd tend to
let this be, until we find out we can't, and why.

> >> * interaction of SIGCHLD handling and threads?
> > 
> > Details?
> 
> Your message http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00738.html raised questions.  If they were ever satisfactorily answered, I overlooked it when putting together my notes….

That should be easy: since a subprocess is locked to a single thread,
SIGCHLD should be delivered to that thread.  If we don't have that
already, we should add that, it doesn't sound hard, given the
infrastructure we already have (deliver_thread_signal etc.).

> It’s easy enough to disable stack overflow checking when enabling thread support.

Or add some simple code in the stack overflow handler to check if we
are in the main thread, and if not, punt (i.e. crash).

> If only one thread is allowed into the image processing code at a time (i.e., don’t release the global lock for that code) then that’s probably fine for now, and there’s probably other state there that different threads shouldn’t be mucking around with in parallel.

Redisplay runs in the main thread anyway, right?  If so, there's no
problem.

> The keyboard.c one is the only one I’m a bit concerned about, in part because I haven’t looked at it.

What part(s) of keyboard.c, exactly?

Thanks.

Re: Concurrency, again

[Ken Raeburn]

> On Oct 18, 2016, at 06:41, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Tue, 18 Oct 2016 06:08:49 -0400
>> Cc: tom@tromey.com,
>> emacs-devel@gnu.org
>> 
>>>> * header inclusion order requirement is weird; can generating one big header help?
>>> 
>>> I'm not sure I understand what inclusion order is being alluded to
>>> here.  Can you elaborate?
>> 
>> I think it’s the ordering and recursive inclusion involving thread.h relative to the other headers.  For example lisp.h includes thread.h which includes sysselect.h which includes lisp.h; thread.h uses struct vectorlike_header, so it has to be included in lisp.h after that structure is defined but before struct thread_state gets used; thread.h also includes regex.h which includes lisp.h.  You commented at one point, “We now have an unfortunate situation whereby lisp.h cannot be included before some of the other headers, due to this.”
> 
> This could be solved by moving parts of thread.h into lisp.h, with the
> appropriate #ifdef guards.  I think Lisp objects should be in lisp.h
> anyway, even if they are optional; anything else is confusing.

For big, specialized objects like buffers, I think the modularity can help keep things organized, but where we’ve got a structure with all the per-thread state from random parts of the program, like condition handlers and regex state, we don’t have the luxury of separating things.  But even if we move most of thread.h into lisp.h, I think there may still be mutual recursion between it and the other headers.

Perhaps if the thread state contained struct pointers instead of structures, we could forward-declare some of the types and not have to pull in the other headers in such a fashion.


> 
>>>> * one thread per terminal?
>>> 
>>> Why?
>>> 
>>>> * file notifications and such shouldn’t go through same queue as keyboard events
>>> 
>>> Why?
>> 
>> Stefan’s message: http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00755.html
> 
> I don't see that as a critical problem, perhaps because we don't yet
> realize how serious it can be.  The whole purpose of trying to merge
> the concurrency branch is to collect practical experience as to what
> should and shouldn't be in this kind of Emacs feature.  So I'd tend to
> let this be, until we find out we can't, and why.
> 
>>>> * interaction of SIGCHLD handling and threads?
>>> 
>>> Details?
>> 
>> Your message http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00738.html raised questions.  If they were ever satisfactorily answered, I overlooked it when putting together my notes….
> 
> That should be easy: since a subprocess is locked to a single thread,

by default, but if that thread exits, that lock disappears

> SIGCHLD should be delivered to that thread.  If we don't have that
> already, we should add that, it doesn't sound hard, given the
> infrastructure we already have (deliver_thread_signal etc.).

It’s not completely trivial.  Under POSIX, we get no control over which thread receives SIGCHLD due to a subprocess exiting, except if we want certain threads to block receiving the signal completely.  We can use pthread_kill to explicitly send SIGCHLD to a specific thread, if it’s not blocking the signal, but we don’t know which thread until we’ve fetched the child’s pid and looked it up in some data structure; we do that by calling waitpid() but then the kernel discards the child process’s status info, so the “correct” thread can no longer respond to the signal by making another waitpid() call to collect the status info.  We’d have to save the info the first time we call waitpid().  But doing it from within the signal handler could be tricky, because in that context we’re limited to async-signal-safe functions, and helpful routines like malloc() and pthread_mutex_lock() aren’t on the list.

On the other hand, perhaps we can create one special thread to do all the waitpid() calls and pass info to the Lisp-running threads.  If that’s all it’s doing, the non-signal-handler portion of the thread’s code can loop calling waitpid and locking mutexes and updating data structures, and the Lisp-running threads can check in their thread-state structures for messages from the child-reaping thread.  I’m not sure how safe it would be to block SIGCHLD in threads that might call into the various system UI libraries and such, but we can probably just use a simple SIGCHLD handler to let any thread wake the reaper thread, which can then do its waitpid() calls.


> 
>> It’s easy enough to disable stack overflow checking when enabling thread support.
> 
> Or add some simple code in the stack overflow handler to check if we
> are in the main thread, and if not, punt (i.e. crash).
> 
>> If only one thread is allowed into the image processing code at a time (i.e., don’t release the global lock for that code) then that’s probably fine for now, and there’s probably other state there that different threads shouldn’t be mucking around with in parallel.
> 
> Redisplay runs in the main thread anyway, right?  If so, there's no
> problem.

If some random thread calls (redisplay) or (sit-for …)?  I think it’ll run in whichever Lisp-running thread triggers it.  But, it’ll be the one holding the lock.

> 
>> The keyboard.c one is the only one I’m a bit concerned about, in part because I haven’t looked at it.
> 
> What part(s) of keyboard.c, exactly?

Anything looking at getcjmp; that means read_event_from_main_queue and read_char.  Like I said, I haven’t looked very closely; if the static storage isn’t ever used across a point where the global lock could be released to allow a thread switch, it may be fine.

Re: Concurrency, again

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Wed, 19 Oct 2016 06:18:18 -0400
> Cc: Emacs-Devel devel <emacs-devel@gnu.org>
> 
> > This could be solved by moving parts of thread.h into lisp.h, with the
> > appropriate #ifdef guards.  I think Lisp objects should be in lisp.h
> > anyway, even if they are optional; anything else is confusing.
> 
> For big, specialized objects like buffers, I think the modularity can help keep things organized, but where we’ve got a structure with all the per-thread state from random parts of the program, like condition handlers and regex state, we don’t have the luxury of separating things.  But even if we move most of thread.h into lisp.h, I think there may still be mutual recursion between it and the other headers.

We already have that, and deal with it.

> >> Stefan’s message: http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00755.html
> > 
> > I don't see that as a critical problem, perhaps because we don't yet
> > realize how serious it can be.  The whole purpose of trying to merge
> > the concurrency branch is to collect practical experience as to what
> > should and shouldn't be in this kind of Emacs feature.  So I'd tend to
> > let this be, until we find out we can't, and why.
> > 
> >>>> * interaction of SIGCHLD handling and threads?
> >>> 
> >>> Details?
> >> 
> >> Your message http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00738.html raised questions.  If they were ever satisfactorily answered, I overlooked it when putting together my notes….
> > 
> > That should be easy: since a subprocess is locked to a single thread,
> 
> by default, but if that thread exits, that lock disappears

And the process gets locked to some other thread.

> 
> > SIGCHLD should be delivered to that thread.  If we don't have that
> > already, we should add that, it doesn't sound hard, given the
> > infrastructure we already have (deliver_thread_signal etc.).
> 
> It’s not completely trivial.  Under POSIX, we get no control over which thread receives SIGCHLD due to a subprocess exiting, except if we want certain threads to block receiving the signal completely.  We can use pthread_kill to explicitly send SIGCHLD to a specific thread, if it’s not blocking the signal, but we don’t know which thread until we’ve fetched the child’s pid and looked it up in some data structure; we do that by calling waitpid() but then the kernel discards the child process’s status info, so the “correct” thread can no longer respond to the signal by making another waitpid() call to collect the status info.  We’d have to save the info the first time we call waitpid().  But doing it from within the signal handler could be tricky, because in that context w
 e’re limited to async-signal-safe functions, and helpful routines like malloc() and pthread_mutex_lock() aren’t on the list.

So you are saying this problem was never encountered before in any
other program out there, and doesn't already have a solution?  I find
that hard to believe.

> On the other hand, perhaps we can create one special thread to do all the waitpid() calls and pass info to the Lisp-running threads.

Sounds like an unnecessary complication, but if that's how others solve
this problem, so shall we.

> >> It’s easy enough to disable stack overflow checking when enabling thread support.
> > 
> > Or add some simple code in the stack overflow handler to check if we
> > are in the main thread, and if not, punt (i.e. crash).
> > 
> >> If only one thread is allowed into the image processing code at a time (i.e., don’t release the global lock for that code) then that’s probably fine for now, and there’s probably other state there that different threads shouldn’t be mucking around with in parallel.
> > 
> > Redisplay runs in the main thread anyway, right?  If so, there's no
> > problem.
> 
> If some random thread calls (redisplay) or (sit-for …)?  I think it’ll run in whichever Lisp-running thread triggers it.  But, it’ll be the one holding the lock.

No, sit-for causes a thread switch.

> >> The keyboard.c one is the only one I’m a bit concerned about, in part because I haven’t looked at it.
> > 
> > What part(s) of keyboard.c, exactly?
> 
> Anything looking at getcjmp; that means read_event_from_main_queue and read_char.  Like I said, I haven’t looked very closely; if the static storage isn’t ever used across a point where the global lock could be released to allow a thread switch, it may be fine.

That should already be solved, or else threads cannot receive keyboard
input safely.

Re: Concurrency, again

[Ken Raeburn]

On Oct 19, 2016, at 07:57, Eli Zaretskii <eliz@gnu.org> wrote:

>>> That should be easy: since a subprocess is locked to a single thread,
>> 
>> by default, but if that thread exits, that lock disappears
> 
> And the process gets locked to some other thread.

Not that I can see, unless you explicitly call set-process-thread; update_processes_for_thread_death sets the process thread field to Qnil.


> 
>> 
>>> SIGCHLD should be delivered to that thread.  If we don't have that
>>> already, we should add that, it doesn't sound hard, given the
>>> infrastructure we already have (deliver_thread_signal etc.).
>> 
>> It’s not completely trivial.  […]
> 
> So you are saying this problem was never encountered before in any
> other program out there, and doesn't already have a solution?  I find
> that hard to believe.

Not at all, just that it may take some work.  Though, I would expect that most such programs we might go look at are committed to a multi-threaded design, and aren’t written to support both multi-threaded and single-threaded environments.

> 
>> On the other hand, perhaps we can create one special thread to do all the waitpid() calls and pass info to the Lisp-running threads.
> 
> Sounds like an unnecessary complication, but if that's how others solve
> this problem, so shall we.

I don’t know.  Aside from checking a few man pages while writing the previous email, I haven’t researched it.  I think it’s probably the way I’d be most inclined to do such a thing, if the ability to use such helper threads could be assumed; it can’t be in this case.

I just took a quick look at some bits of a Thunderbird source tree.  In the Netscape Portable Runtime (thunderbird-49.0b1/mozilla/nsprpub/pr/src/md/unix/uxproces.c) they’ve got a “waitpid daemon thread”; whichever thread gets SIGCHLD delivered to it writes a byte to a pipe to wake up that daemon thread, which then calls waitpid until there are no more child process status updates to process.  It updates global (mutex-protected) data structures and optionally uses a condition variable to notify some other thread of the process change.

> 
>>>> It’s easy enough to disable stack overflow checking when enabling thread support.
>>> 
>>> Or add some simple code in the stack overflow handler to check if we
>>> are in the main thread, and if not, punt (i.e. crash).
>>> 
>>>> If only one thread is allowed into the image processing code at a time (i.e., don’t release the global lock for that code) then that’s probably fine for now, and there’s probably other state there that different threads shouldn’t be mucking around with in parallel.
>>> 
>>> Redisplay runs in the main thread anyway, right?  If so, there's no
>>> problem.
>> 
>> If some random thread calls (redisplay) or (sit-for …)?  I think it’ll run in whichever Lisp-running thread triggers it.  But, it’ll be the one holding the lock.
> 
> No, sit-for causes a thread switch.

I believe that’s when it’s checking for input, after calling redisplay, though.

I just tried a test with a breakpoint in redisplay_internal; it can get called from threads other than the main thread.  As far as I know, we can’t have multiple threads in the redisplay code at the same time, though.

> 
>>>> The keyboard.c one is the only one I’m a bit concerned about, in part because I haven’t looked at it.
>>> 
>>> What part(s) of keyboard.c, exactly?
>> 
>> Anything looking at getcjmp; that means read_event_from_main_queue and read_char.  Like I said, I haven’t looked very closely; if the static storage isn’t ever used across a point where the global lock could be released to allow a thread switch, it may be fine.
> 
> That should already be solved, or else threads cannot receive keyboard
> input safely.

I hope so.  I haven’t convinced myself one way or the other yet, though.

Ken

Re: Concurrency, again

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 20 Oct 2016 02:08:14 -0400
> Cc: emacs-devel@gnu.org
> 
> I just tried a test with a breakpoint in redisplay_internal; it can get called from threads other than the main thread.  As far as I know, we can’t have multiple threads in the redisplay code at the same time, though.

But redisplay_internal will return immediately without doing anything
if it's already running in some other thread.  (And this "running in
some other thread could only happen if that other thread somehow
succeeded to block or yield in the middle of redisplay, which I think
is only possible if it calls Lisp.)

RE: Concurrency, again

[Herring, Davis]

> we do that by calling waitpid() but then the kernel discards the child
> process’s status info, so the "correct" thread can no longer respond
> to the signal by making another waitpid() call to collect the status
> info.  We’d have to save the info the first time we call waitpid().
> But doing it from within the signal handler could be tricky, because
> in that context we’re limited to async-signal-safe functions, and
> helpful routines like malloc() and pthread_mutex_lock() aren’t on the
> list.

waitid() supports the WNOWAIT flag that peeks at the child information without destroying it: http://pubs.opengroup.org/onlinepubs/9699919799/functions/waitid.html

Davis

Re: Concurrency, again

[Ken Raeburn]

> On Oct 20, 2016, at 03:12, Herring, Davis <herring@lanl.gov> wrote:
> 
>> we do that by calling waitpid() but then the kernel discards the child
>> process’s status info, so the "correct" thread can no longer respond
>> to the signal by making another waitpid() call to collect the status
>> info.  We’d have to save the info the first time we call waitpid().
>> But doing it from within the signal handler could be tricky, because
>> in that context we’re limited to async-signal-safe functions, and
>> helpful routines like malloc() and pthread_mutex_lock() aren’t on the
>> list.
> 
> waitid() supports the WNOWAIT flag that peeks at the child information without destroying it: http://pubs.opengroup.org/onlinepubs/9699919799/functions/waitid.html

Ah, yes, thanks!  I wasn’t aware of that one.  I think the last time I had to deal with the wait* interfaces in portable code was back before POSIX support could be assumed…. :-)

Still, sending signals back and forth between threads so each can wait on its own child processes (and also somehow dealing with the ones not bound to any specific thread) seems messy compared to the waitpid daemon thread approach of NSPR.

Ken

Re: Concurrency, again

[Paul Eggert]

On 10/20/2016 12:55 AM, Ken Raeburn wrote:
> I think the last time I had to deal with the wait* interfaces in portable code was back before POSIX support could be assumed

Emacs doesn't assume full POSIX support, and in practice waitid is 
problematic. The Gnulib manual says:

    This function is missing on some platforms: Mac OS X 10.4, FreeBSD 
6.0, NetBSD 5.0, OpenBSD 3.8, Minix 3.1.8, Cygwin, mingw, MSVC 9, 
Interix 3.5, BeOS. As of 2005, no system is known on which waitid with 
flag WNOWAIT works correctly.

Although WNOWAIT is probably in better shape now, I wouldn't assume it 
works on all current Emacs targets.

Re: Concurrency, again

[Alan Third]

[-- Attachment #1: Type: text/plain, Size: 182 bytes --]

On 18 October 2016 at 08:58, Ken Raeburn <raeburn@raeburn.org> wrote:

>  * ns_select needs fixing


Do you have any more information about what's to be fixed here?


-- 
Alan Third

[-- Attachment #2: Type: text/html, Size: 582 bytes --]

Re: Concurrency, again

[Ken Raeburn]

[-- Attachment #1: Type: text/plain, Size: 1504 bytes --]


> On Oct 18, 2016, at 06:46, Alan Third <alan@idiocy.org> wrote:
> 
> 
> On 18 October 2016 at 08:58, Ken Raeburn <raeburn@raeburn.org <mailto:raeburn@raeburn.org>> wrote:
>  * ns_select needs fixing
> 
> Do you have any more information about what's to be fixed here?
> 
> 
> -- 
> Alan Third


From my notes I thought that came out of the 2013 discussion, but I can’t find anything in the archives at the moment.  Looks like I put it in the wrong sections of my notes.

Last November I put in a patch (commit b87000208) changing a few the select calls because of locking issues; the global lock still needed to be held while xg_select called routines like unblock_input, so I pushed the thread_select wrapper (which releases and re-acquires the lock) calls to a lower level.

In ns_select it wasn’t clear to me where to put the calls.  It’s also clearly using static storage for fd arrays, which is a no-no if multiple threads will be able to call into that routine concurrently.  So I added a “FIXME” statement to nsterm.m to prevent compilation of a version known to be wrong.

Depending how the NS version needs to change, possibly the “thread_select(pselect,…)” calls should be replaced with a wrapper specifically for pselect-using platforms, and the NS code manage the global lock separately.  Or perhaps we’ll wind up with an NS-specific pselect-like routine that we pass to thread_select on the Mac, keeping the global lock management in one place.

Ken

[-- Attachment #2: Type: text/html, Size: 2615 bytes --]

Re: Concurrency, again

[Tom Tromey]

>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:

Ken> In fact, in order to get a better
Ken> understanding of it, I set up a private repository where I broke up
Ken> the concurrency branch changes — not each commit, but the overall diff
Ken> from the branch point — into little pieces, separating straightforward
Ken> but pervasive changes from more subtle changes so I could figure out
Ken> what the latter were doing.  The result of my merge that I checked in
Ken> was in fact the result of rebasing those little changes onto the
Ken> latest master branch, fixing them up as necessary as I went along.
Ken> (But the resulting body of code was checked in as a merge commit so as
Ken> to preserve history of the work on the branch — what change was put
Ken> in, who fixed what bugs, etc; I wasn’t about to unilaterally throw
Ken> that away.  It was just a slow, tedious, manual merge process instead
Ken> of an automated one.)

Your branch sounds promising.

Ken>  * one thread per terminal?

This was something I thought would be useful, but not a must-have.

Tom

Re: Concurrency, again

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

John Wiegley <jwiegley@gmail.com> schrieb am Mi., 12. Okt. 2016 um
18:43 Uhr:

> >>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> SM> IOW, I completely agree with Eli: we should fix the few remaining
> issues
> SM> with the concurrency branch and merge it into master.
>
> I very strongly disagree. I think "merge what we have and make it work" is
> the
> wrong approach.


I agree, there should at least be some more discussion about the design we
want.
You could group the concurrency problem into a couple of subproblems:
- Context switching
- Scheduling
- Communication
- Synchronization
IIUC the concurrency branch uses OS threads plus custom code for context
switching and scheduling, global variables for communication, and OS
mutexes/condvars for synchronization. My experiment uses setcontext for
context switching, a user mode scheduler for scheduling, and CSP without
shared state for both communication and synchronization. I think CSP has
the advantage of being higher-level and avoiding low-level bugs due to
misuse of synchronization primitives such as mutexes.

[-- Attachment #2: Type: text/html, Size: 1674 bytes --]

Re: Concurrency, again

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Tue, 25 Oct 2016 13:27:39 +0000
> 
>  I very strongly disagree. I think "merge what we have and make it work" is the
>  wrong approach.

(Since then, John changed his mind.)

> I agree, there should at least be some more discussion about the design we want.

IMO, we don't have Emacs-specific experience for such discussions.  So
these discussions will most probably lead nowhere.

My suggestion is to "learn by doing", collect experience, and _then_
discuss the design issues based on that experience.

Re: Concurrency, again

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

>> I very strongly disagree. I think "merge what we have and make it work" is the
>> wrong approach.

EZ> (Since then, John changed his mind.)

More accurately, Eli changed John's mind, by helping me see that (a) having
the support affects no one but modules that choose to use it, and (b) we can
call it experimental and subject to removal if the experiment fails. At the
very least we'd gain some experience by the attempt, rather than confining
ourselves to thought exercises.

EZ> IMO, we don't have Emacs-specific experience for such discussions. So
EZ> these discussions will most probably lead nowhere.

Yes, this.

EZ> My suggestion is to "learn by doing", collect experience, and _then_
EZ> discuss the design issues based on that experience.

I'm open to this. I'd *like* to see exploration of alternative paradigms for
expressing concurrency, so merging in the concurrency branch shouldn't be seen
as closing the door, or setting anything in stone.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Concurrency, again

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

John Wiegley <jwiegley@gmail.com> schrieb am Di., 25. Okt. 2016 um
18:45 Uhr:

> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> EZ> My suggestion is to "learn by doing", collect experience, and _then_
> EZ> discuss the design issues based on that experience.
>
> I'm open to this. I'd *like* to see exploration of alternative paradigms
> for
> expressing concurrency, so merging in the concurrency branch shouldn't be
> seen
> as closing the door, or setting anything in stone.
>
>
That might be the intention, but I expect the outcome will be that interest
in alternative paradigms gets lost (unless such alternative paradigms would
also be merged and be available in parallel).

[-- Attachment #2: Type: text/html, Size: 1247 bytes --]

Re: Concurrency, again

[Stefan Huchler]

Philipp Stephani <p.stephani2@gmail.com> writes:

> That might be the intention, but I expect the outcome will be that
> interest in alternative paradigms gets lost (unless such alternative
> paradigms would also be merged and be available in parallel). 

In hope to bring the discussion further without knowing to much about
the topic, you talked about "in your experiment" so do you have some
sort of proof-of-concept code working with emacs?

Are you willing to invest much time in implementing your solution? It seems
to be similar to the commercial world, where people say "put your money
where your mouth is", just in this case "put your code where your mouth
is".

I think people would be more convinced if there is some commitment,
cause else you can formulate the nicest thing and nothing will happen.

I dont want to attack you but maybe help you to "sell" your idea better.

But maybe I get the situation wrong, just looks for me like thats the
major motivation why people now push for that mostly done solution
instead of pie in the sky?

Re: Concurrency, again

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1890 bytes --]

Stefan Huchler <stefan.huchler@mail.de> schrieb am Di., 25. Okt. 2016 um
19:06 Uhr:

> Philipp Stephani <p.stephani2@gmail.com> writes:
>
> > That might be the intention, but I expect the outcome will be that
> > interest in alternative paradigms gets lost (unless such alternative
> > paradigms would also be merged and be available in parallel).
>
> In hope to bring the discussion further without knowing to much about
> the topic, you talked about "in your experiment" so do you have some
> sort of proof-of-concept code working with emacs?
>

Yes, I'd be happy to upload it to my Github repo or to a new branch in the
Savannah repo if there's interest.


>
> Are you willing to invest much time in implementing your solution?


I think this is important, so I'd be willing to spend some significant
amount of time on it. If it turns out to be a huge time sink, we'd still
have the existing concurrency branch, so we wouldn't be worse off than we
are now.


> It seems
> to be similar to the commercial world, where people say "put your money
> where your mouth is", just in this case "put your code where your mouth
> is".
>
>
That's a good principle.


> I think people would be more convinced if there is some commitment,
> cause else you can formulate the nicest thing and nothing will happen.
>
> I dont want to attack you but maybe help you to "sell" your idea better.
>
>
I don't see it as an attack. I'm happy to experiment with the various
options and participate in concrete design discussions.


> But maybe I get the situation wrong, just looks for me like thats the
> major motivation why people now push for that mostly done solution
> instead of pie in the sky?
>
>
>
My branch can run all examples from the Go tutorial, so I wouldn't call it
"pie in the sky". There are obviously open questions (such as Windows
support), but I think it's a good start for a discussion.

[-- Attachment #2: Type: text/html, Size: 3472 bytes --]

Re: Concurrency, again

[John Wiegley]

>>>>> "PS" == Philipp Stephani <p.stephani2@gmail.com> writes:

>> Are you willing to invest much time in implementing your solution?

PS> I think this is important, so I'd be willing to spend some significant
PS> amount of time on it. If it turns out to be a huge time sink, we'd still
PS> have the existing concurrency branch, so we wouldn't be worse off than we
PS> are now.

This sounds great, Philipp, I look forward to your experiment.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Concurrency, again

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

John Wiegley <jwiegley@gmail.com> schrieb am Di., 25. Okt. 2016 um
23:46 Uhr:

> >>>>> "PS" == Philipp Stephani <p.stephani2@gmail.com> writes:
>
> >> Are you willing to invest much time in implementing your solution?
>
> PS> I think this is important, so I'd be willing to spend some significant
> PS> amount of time on it. If it turns out to be a huge time sink, we'd
> still
> PS> have the existing concurrency branch, so we wouldn't be worse off than
> we
> PS> are now.
>
> This sounds great, Philipp, I look forward to your experiment.
>
>
I've pushed the experimental branch to 'concurrency-libtask'. It's
essentially a simple wrapper around libtask, which implements CSP based on
setcontext. I've also implemented a Windows equivalent based on Windows
native fibers, but haven't tried that yet.

[-- Attachment #2: Type: text/html, Size: 1466 bytes --]

Re: Concurrency, again

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Tue, 25 Oct 2016 23:28:51 +0000
> 
> I've pushed the experimental branch to 'concurrency-libtask'. It's essentially a simple wrapper around libtask,
> which implements CSP based on setcontext. I've also implemented a Windows equivalent based on
> Windows native fibers, but haven't tried that yet. 

Thanks.

Could you perhaps summarize the relative advantages and disadvantages
of the two concurrency branches?  Your branch doesn't have any
documentation besides doc strings of the new primitives, so it's not
easy to grasp the high-level picture by looking at the details.

One issue that bothers me is whether it's wise to use libtask here,
because the changes you did there seem to imply that we will have to
maintain the library (which is pretty low-level stuff) as part of
Emacs.  Isn't using system threads better?

(In any case, libtask shouldn't go under lib/, that directory is for
Gnulib and related stuff.  It should be a peer directory to lwlib, I
think.)

I'd also like to hear other people's opinions about the way
communications with coroutines is exposed to Lisp, it seemed a bit
tedious and low-level to me.

As for Windows implementation, fibers are only available since XP,
whereas the 32-bit build of Emacs on Windows still attempts to support
older systems.  So using threads here would be better, IMO.

Re: Concurrency, again

[Daniel Colascione]

On 10/27/2016 10:27 AM, Eli Zaretskii wrote:
>> From: Philipp Stephani <p.stephani2@gmail.com>
>> Date: Tue, 25 Oct 2016 23:28:51 +0000
>>
>> I've pushed the experimental branch to 'concurrency-libtask'. It's essentially a simple wrapper around libtask,
>> which implements CSP based on setcontext. I've also implemented a Windows equivalent based on
>> Windows native fibers, but haven't tried that yet.
>
> Thanks.
>
> Could you perhaps summarize the relative advantages and disadvantages
> of the two concurrency branches?  Your branch doesn't have any
> documentation besides doc strings of the new primitives, so it's not
> easy to grasp the high-level picture by looking at the details.
>
> One issue that bothers me is whether it's wise to use libtask here,
> because the changes you did there seem to imply that we will have to
> maintain the library (which is pretty low-level stuff) as part of
> Emacs.  Isn't using system threads better?

Agreed on system threads vs libtask. Fibers of the sort libtask provides 
(similar to GNU Pth) have some claimed advantages in efficiency in large 
programs with lots of concurrent tasks, but for us, I vote for 
Python-style use of OS threads with a Python-style GIL that we can 
release to do long-running computations and recover from parallelism. (I 
believe the existing concurrency branch runs under this model already, 
but I'll have to take a closer look.) System threads also have much 
better integration with the debugger and other development tools; fibers 
have no offsetting advantages for us.

Re: Concurrency, again

[Christopher Allan Webber]

Daniel Colascione writes:

> On 10/27/2016 10:27 AM, Eli Zaretskii wrote:
>>> From: Philipp Stephani <p.stephani2@gmail.com>
>>> Date: Tue, 25 Oct 2016 23:28:51 +0000
>>>
>>> I've pushed the experimental branch to 'concurrency-libtask'. It's essentially a simple wrapper around libtask,
>>> which implements CSP based on setcontext. I've also implemented a Windows equivalent based on
>>> Windows native fibers, but haven't tried that yet.
>>
>> Thanks.
>>
>> Could you perhaps summarize the relative advantages and disadvantages
>> of the two concurrency branches?  Your branch doesn't have any
>> documentation besides doc strings of the new primitives, so it's not
>> easy to grasp the high-level picture by looking at the details.
>>
>> One issue that bothers me is whether it's wise to use libtask here,
>> because the changes you did there seem to imply that we will have to
>> maintain the library (which is pretty low-level stuff) as part of
>> Emacs.  Isn't using system threads better?
>
> Agreed on system threads vs libtask. Fibers of the sort libtask provides 
> (similar to GNU Pth) have some claimed advantages in efficiency in large 
> programs with lots of concurrent tasks, but for us, I vote for 
> Python-style use of OS threads with a Python-style GIL that we can 
> release to do long-running computations and recover from parallelism.

Python's GIL + threading is probably not the best model to work off
of... while the GIL is largely misunderstood, there have been a lot of
challenges with getting this model to work nice.  At one point, it
turned out that the GIL + threads would result in constant CPU
thrashing.  This isn't the case any more, but it was once, and was a
challenging thing to fix:

  http://dabeaz.com/python/UnderstandingGIL.pdf

Even now, my understanding is that Python does a ton of context
switching, and while it's better than it was, you don't really get a lot
of the performance you would over threads being used in other systems,
because it's hard for Python to really run things concurrently.  Thus
the rise of things like asyncio, where instead of parallelizing, you
break things into a lot of coroutines which don't block on any IO.

Other mechanisms like green threading + coroutines for most IO bound
async stuff and, in the case that you need CPU bound stuff to be
optimized, something that spawns independent processes (or okay, maybe
threads) that communicate with message passing is probably much better
than a GIL + system threads route, I'd think?

Re: Concurrency, again

[Daniel Colascione]

On October 27, 2016 1:55:02 PM PDT, Christopher Allan Webber <cwebber@dustycloud.org> wrote:
>Daniel Colascione writes:
>
>> On 10/27/2016 10:27 AM, Eli Zaretskii wrote:
>>>> From: Philipp Stephani <p.stephani2@gmail.com>
>>>> Date: Tue, 25 Oct 2016 23:28:51 +0000
>>>>
>>>> I've pushed the experimental branch to 'concurrency-libtask'. It's
>essentially a simple wrapper around libtask,
>>>> which implements CSP based on setcontext. I've also implemented a
>Windows equivalent based on
>>>> Windows native fibers, but haven't tried that yet.
>>>
>>> Thanks.
>>>
>>> Could you perhaps summarize the relative advantages and
>disadvantages
>>> of the two concurrency branches?  Your branch doesn't have any
>>> documentation besides doc strings of the new primitives, so it's not
>>> easy to grasp the high-level picture by looking at the details.
>>>
>>> One issue that bothers me is whether it's wise to use libtask here,
>>> because the changes you did there seem to imply that we will have to
>>> maintain the library (which is pretty low-level stuff) as part of
>>> Emacs.  Isn't using system threads better?
>>
>> Agreed on system threads vs libtask. Fibers of the sort libtask
>provides 
>> (similar to GNU Pth) have some claimed advantages in efficiency in
>large 
>> programs with lots of concurrent tasks, but for us, I vote for 
>> Python-style use of OS threads with a Python-style GIL that we can 
>> release to do long-running computations and recover from parallelism.
>
>Python's GIL + threading is probably not the best model to work off
>of... while the GIL is largely misunderstood, there have been a lot of
>challenges with getting this model to work nice.  At one point, it
>turned out that the GIL + threads would result in constant CPU
>thrashing.  This isn't the case any more, but it was once, and was a
>challenging thing to fix:
>
>  http://dabeaz.com/python/UnderstandingGIL.pdf
>
>Even now, my understanding is that Python does a ton of context
>switching, and while it's better than it was, you don't really get a
>lot
>of the performance you would over threads being used in other systems,
>because it's hard for Python to really run things concurrently.  Thus
>the rise of things like asyncio, where instead of parallelizing, you
>break things into a lot of coroutines which don't block on any IO.

Of course they block on IO, in their own terms. On a different level of abstraction, they're non blocking. They are exactly as blocking as OS's threads, except you're doing the work the OS already does, and probably better, at least at Emacs scale.

I favor maintainability and developer familiarity over raw performance. If we're merely as successful as Python, I don't think I'll be terribly unhappy 

>
>Other mechanisms like green threading + coroutines for most IO bound
>async stuff and, in the case that you need CPU bound stuff to be
>optimized, something that spawns independent processes (or okay, maybe
>threads) that communicate with message passing is probably much better
>than a GIL + system threads route, I'd think

The numpy approach helps for the odd truly compute bound task. You can implement message passing on top of a GIL system.

I understand you as claiming that a green thread system is more powerful than a real thread system. Can you see how? From where I'm standing, it has the same limitations *and* it's harder to release the GIL and do something heavy like JPEG decoding.

Re: Concurrency, again

[Eli Zaretskii]

> From: Christopher Allan Webber <cwebber@dustycloud.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, Philipp Stephani <p.stephani2@gmail.com>, stefan.huchler@mail.de, emacs-devel@gnu.org
> Date: Thu, 27 Oct 2016 15:55:02 -0500
> 
> Python's GIL + threading is probably not the best model to work off
> of... while the GIL is largely misunderstood, there have been a lot of
> challenges with getting this model to work nice.  At one point, it
> turned out that the GIL + threads would result in constant CPU
> thrashing.  This isn't the case any more, but it was once, and was a
> challenging thing to fix:
> 
>   http://dabeaz.com/python/UnderstandingGIL.pdf

It sounds like you are talking about problems inherent to Python
threads, and not about what the concurrency branch in Emacs attempts
to do.

The problematic issues described in the above presentation are not
relevant to the Emacs concurrency branch, because (1) it doesn't have
the thread preemption for CPU-bound threads, and (2) it doesn't switch
threads on _any_ I/O, just on I/O with an external process or user
input, i.e. when Emacs calls pselect or its equivalent to see whether
any input has arrived.

So the performance penalties discussed in that presentation have
nothing to do with what the Emacs concurrency branch implements.

Re: Concurrency, again

[Tom Tromey]

>>>>> "Philipp" == Philipp Stephani <p.stephani2@gmail.com> writes:

Philipp> I've pushed the experimental branch to
Philipp> 'concurrency-libtask'. It's essentially a simple wrapper around
Philipp> libtask, which implements CSP based on setcontext. I've also
Philipp> implemented a Windows equivalent based on Windows native
Philipp> fibers, but haven't tried that yet.

I took a quick look at it.

I think one thing to know is that the existing concurrency branch has
two distinct parts.  One part is changing various bits of Emacs
internals to cope with multiple threads.  The other part is the design
and implementation of the thread API that is exposed to Emacs Lisp.

IMO the concurrency branch's approach to the internals changes is
generally preferable here.  There are a few issues I saw:

* AFAICT concurrency-libtask doesn't correctly handle updating let
  bindings when task switching;
* It doesn't make the match data per-task;
* Nor the current buffer;
* It doesn't handle the case where multiple tasks select on an fd
  at the same time (the concurrency branch also adds process locking,
  which I thought was handy)

These could be solved of course -- they're already all solved on the
concurrency branch.  There may be more things there that I've forgotten
about as well.

If channels are preferred to locks, it doesn't seem hard to either
simply make channels the API on the concurrency branch; or just write
channels as some lisp around the current code.

Tom

Re: Concurrency, again

[Ted Zlatanov]

On Tue, 25 Oct 2016 19:04:42 +0200 Stefan Huchler <stefan.huchler@mail.de> wrote: 

SH> Are you willing to invest much time in implementing your solution? It seems
SH> to be similar to the commercial world, where people say "put your money
SH> where your mouth is", just in this case "put your code where your mouth
SH> is".

SH> I think people would be more convinced if there is some commitment,
SH> cause else you can formulate the nicest thing and nothing will happen.

Commitment and personal investment are not enough to make a code change
convincing.

Please don't forget that writing code is a small (10-15%) part of the
overall cost. Documentation, backwards portability, handling bugs
(including bugs caused by poor documentation or poor API design!),
technical debt, and ongoing maintenance are costs that must be
considered by the Emacs maintainers for any proposed change. The scope,
therefore, matters: the wider the scope of the change, the more risk and
cost it may incur.

In that light, I think Philipp's proposed libtask wrapper may be a good
direction because the risk is low (it relies on existing well-known
libraries) and the cost is low (it doesn't change internals
significantly). Looking at the concurrency-libtask branch, the API
design of the channel piece, at least, seems reasonable: it's well-known
to Go programmers and provides a nice message bus facility.

I don't know how Philipp's proposal interacts with the other concurrency
proposals, though.

HTH
Ted

Re: Concurrency, again

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Wed, 26 Oct 2016 10:03:54 -0400
> 
> Please don't forget that writing code is a small (10-15%) part of the
> overall cost. Documentation, backwards portability, handling bugs
> (including bugs caused by poor documentation or poor API design!),
> technical debt, and ongoing maintenance are costs that must be
> considered by the Emacs maintainers for any proposed change. The scope,
> therefore, matters: the wider the scope of the change, the more risk and
> cost it may incur.

FWIW, those additional costs were never considered by this project,
AFAIR.  If a feature was deemed useful and its implementation clean
enough, it was invariably accepted, under the assumption that the
costs will be bearable.  We even accepted significant features with
minimal or no documentation at all.  And at least in one case, the
developer actually dropped off the grid when the time came for
seriously shaking out his feature.

Re: Concurrency, again

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Tue, 25 Oct 2016 16:48:47 +0000
> 
>  I'm open to this. I'd *like* to see exploration of alternative paradigms for
>  expressing concurrency, so merging in the concurrency branch shouldn't be seen
>  as closing the door, or setting anything in stone.
> 
> That might be the intention, but I expect the outcome will be that interest in alternative paradigms gets lost
> (unless such alternative paradigms would also be merged and be available in parallel). 

Volunteers to come up with such alternatives are always welcome, of
course.

Re: Concurrency, again

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1975 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Mo., 10. Okt. 2016 um 09:17 Uhr:

>
> > The reason I love Haskell for its STM concurrency (software transactional
> > memory) is that it makes certain classes of problems impossible to
> express. I
> > believe we would need a mechanism like that for Emacs Lisp, so no one
> ever has
> > to hunt down cyclic mutex locks, or reference counts, or why two
> operations
> > that should be atomic aren't. I'd rather have a single-threaded Emacs
> for a
> > quite a while longer before inviting these sorts problems into our lives.
>
> But we don't even know whether these problems are relevant to what Tom
> implemented in the concurrency branch.
>

We have lots of experience from other programming languages. Emacs isn't
that different (Elisp is mostly Python with a strange syntax), so we can
transfer experience to Emacs. Specifically, we don't have to reinvent
everything from scratch. There are a couple of high-level concurrency
models (STM, CSP) that are widely used in practice already.
Out of curiosity I've implemented Plan9/Go-style CSP based on libtask
(without OS threads) in Emacs with minimal changes, and most things seem to
work just fine.


>
> We've gone a long way since this issue was first brought up.  We've
> changed our sources in significant ways to support concurrency: all
> those BVAR and KVAR macros that are all around the C sources were
> introduced for that very purpose.  Likewise, the globals.h header
> file, and the fact that each variable exposed to Lisp is a member of
> some C struct -- all this was for supporting concurrency.  The code
> for this is written, debugged, and is only a little ways from being
> ready for prime time.


We still have tons of global mutable state, e.g. the buffer list.
Traditional OS-level multithreading is impossible with that amount of
global state.
On the other hand, non-parallel concurrency doesn't care about global state
(which can just be swapped out) at all.

[-- Attachment #2: Type: text/html, Size: 2834 bytes --]

Re: Concurrency, again

[Stefan Monnier]

> Out of curiosity I've implemented Plan9/Go-style CSP based on libtask
> (without OS threads) in Emacs with minimal changes, and most things seem to
> work just fine.

Great.  Have you then tried to make Gnus use it?


        Stefan

Re: Concurrency, again

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Di., 25. Okt. 2016 um
16:26 Uhr:

> > Out of curiosity I've implemented Plan9/Go-style CSP based on libtask
> > (without OS threads) in Emacs with minimal changes, and most things seem
> to
> > work just fine.
>
> Great.  Have you then tried to make Gnus use it?
>

No, I don't use Gnus myself, so I couldn't compare it to the current state
anyway.

[-- Attachment #2: Type: text/html, Size: 805 bytes --]

Re: Concurrency, again

[Ted Zlatanov]

On Tue, 25 Oct 2016 14:41:54 +0000 Philipp Stephani <p.stephani2@gmail.com> wrote: 

PS> Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Di., 25. Okt. 2016 um
PS> 16:26 Uhr:

>> > Out of curiosity I've implemented Plan9/Go-style CSP based on libtask
>> > (without OS threads) in Emacs with minimal changes, and most things seem
>> to
>> > work just fine.
>> 
>> Great.  Have you then tried to make Gnus use it?

PS> No, I don't use Gnus myself, so I couldn't compare it to the current state
PS> anyway.

Take a single buffer, clear it, then take a randomized big vector,
filter it by different keys to reduce to about 50% of the original size,
then sort it multiple times by different keys, inserting the elements
one by one into the buffer. For each insertion, do a formatting step
that calls the same 3-4 functions every time. That would be somewhat
similar to what Gnus does to build the article list.

This might be a good performance test for any proposed concurrent changes.

Ted

Re: Concurrency, again

[Stefan Monnier]

>>> > Out of curiosity I've implemented Plan9/Go-style CSP based on
>>> > libtask (without OS threads) in Emacs with minimal changes, and
>>> > most things seem to work just fine.
>>> Great.  Have you then tried to make Gnus use it?
PS> No, I don't use Gnus myself, so I couldn't compare it to the current state
PS> anyway.
[...]
> This might be a good performance test for any proposed concurrent changes.

I was thinking of another kind of performance test: how much time is
necessary to take existing code and make it use the concurrency support.


        Stefan

Re: Concurrency, again

[Ted Zlatanov]

On Wed, 26 Oct 2016 10:32:11 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> I was thinking of another kind of performance test: how much time is
SM> necessary to take existing code and make it use the concurrency support.

Include emacs-devel debates and it easily triples ;)

Ted

Re: Concurrency, again

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Tue, 25 Oct 2016 13:22:42 +0000
> Cc: lars@nocrew.org, toon@iotcl.com, emacs-devel@gnu.org
> 
> We still have tons of global mutable state, e.g. the buffer list. Traditional OS-level multithreading is impossible
> with that amount of global state.

But we are not doing OS-level multithreading on the concurrency branch.

> On the other hand, non-parallel concurrency doesn't care about global state (which can just be swapped out)
> at all.

Which is what we do.

Re: Concurrency, again

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > We have lots of experience from other programming languages. Emacs isn't
  > that different (Elisp is mostly Python with a strange syntax),

That is not true at all.
Lisp and Python are fundamentally different.

The basic operations of Lisp are read, eval, and print.
They don't exist in Python because its structure has no
room for any such thing.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[raman]

John Wiegley <jwiegley@gmail.com> writes:

1+ on this. 
>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>> There's already a concurrency branch in the Emacs repository. It needs some
>> work before it could be merged to master, so if there are people here who
>> wants this badly enough, I suggest that they continue work almost done by
>> Tom Tromey, who developed that branch, instead of starting anew.
>
> I think it needs more than just a little work. I spoke to Tom on the phone
> last year, and we both agreed it's not a foregone conclusion that that branch
> represents the right way of approaching concurrency for an application like
> Emacs.
>
> In general, users don't want programmatic concurrency. What they want is for
> Emacs not to freeze up after they've asked it to do something. And there are
> other ways of achieving this end -- depending on the nature of the problem --
> than making concurrency a first-class citizen, with the innumerable problems
> it brings. I fear we'd be debugging subtle interaction issues for the rest of
> our lives if we just merged Tom's branch in today. It implements a Java-style
> form of concurrency, dependent on mutexes and locking to achieve coherency,
> which is fiendishly difficult to get right, even if you confine its use to
> just a few small tasks. The hardest bug I ever debugged in my life was a
> concurrency bug; the second hardest wasn't even close (and thankfully, I
> didn't try to solve it at the same time).
>
> I'd much rather we re-examine the goals we want to achieve with concurrency,
> an then ask if there are other ways to get there besides, well, concurrency.
> For example, if we had a lightweight forking mechanism with transparent
> communication between sub-ordinate processes (think async.el, but in C and
> highly efficient), I think we could achieve what users want, without the
> downsides developers don't want. Even the popular web browsers are moving in
> this direction, since it gives a similar effect to threading, but without as
> many of the downsides; or take the popularity of the Actor model, used to
> reduce the coherency problem down to just mailboxes.
>
> The reason I love Haskell for its STM concurrency (software transactional
> memory) is that it makes certain classes of problems impossible to express. I
> believe we would need a mechanism like that for Emacs Lisp, so no one ever has
> to hunt down cyclic mutex locks, or reference counts, or why two operations
> that should be atomic aren't. I'd rather have a single-threaded Emacs for a
> quite a while longer before inviting these sorts problems into our lives.

-- 

Re: Emacs Lisp's future

[Søren Pilgård]

On Sun, Oct 9, 2016 at 2:03 PM, Toon Claes <toon@iotcl.com> wrote:
> On 7 Oct 2016, at 19:07, John Wiegley <jwiegley@gmail.com> wrote:
>
>  It would be nice to have real compilation
> of Emacs Lisp after the manner of Common Lisp, and maybe the concurrency
> support;
>
>
> Oh yes, please add concurrency support!
> Would it help if we would start a crowdfunding campaign for this? Because I
> know a few fellow emacs user who would definitely back it.
>
> Regards,
> Toon Claes

The thing about concurrency is that it is very hard to integrate
meaningfully and robustly into Emacs.
All Emacs lisp code has the basic assumption, that when it is running,
it is the only code running.
Introducing concurrency would either break that assumption or have to
make it explicit somehow, potentially breaking tons of existing code.

As an example, lets say you have a plugin that count occurrences of a
word, and another that corrects spelling mistakes.
As it is now, the counting plugin can assume no other code runs when
it counts and it can thus simply scan the buffer.
But if we let both run simultaneously the spelling correcter could
modify the buffer resulting in the wrong count as some were done
before and some after the modification.
Oh, but couldn't we then just let each buffer have its own "world"
where everything is sequential?
Well we would lose the ability for the spelling correcter to run while
we were working negating the benefit.
And further, what if the plugin counting occurrences gave the total
count for all buffers?
Then it would have to lock all buffers somehow, so no other code could
modify them simultaneously.

As you can see, introducing concurrency has wide implications for Emacs.
Finding a way to go about it that will still allow old "naive" code to
function correctly is what makes it so tough.
Hopefully we will someday have that, but it requires a lot of hard work.
Maybe we could start out with something like javascripts webworkers. A
function running in its own world without access to buffers or other
shared state, and then we could communicate by message passing.
But as the keen eye would observe that is very similar to simply
spawning an external process.
And it wouldn't benefit much for problems that rely on buffers or global state.

This problem will clearly not be solved just by switching the
underlying platform.

Re: Emacs Lisp's future

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

Søren Pilgård <fiskomaten@gmail.com> schrieb am So., 9. Okt. 2016 um
14:44 Uhr:

> On Sun, Oct 9, 2016 at 2:03 PM, Toon Claes <toon@iotcl.com> wrote:
>
>
> As you can see, introducing concurrency has wide implications for Emacs.
>
> Finding a way to go about it that will still allow old "naive" code to
>
> function correctly is what makes it so tough.
>
> Hopefully we will someday have that, but it requires a lot of hard work.
>
> Maybe we could start out with something like javascripts webworkers. A
>
> function running in its own world without access to buffers or other
>
> shared state, and then we could communicate by message passing.
>


Yes, this is the right approach. It is clear that new threads couldn't be
allowed to access existing global state (dynamic variables, buffers,
obarray, ...); instead they would need to start with an empty global state.


>
> But as the keen eye would observe that is very similar to simply
>
> spawning an external process.
>
> And it wouldn't benefit much for problems that rely on buffers or global
> state.
>
>
>
That is a good thing; code should not use global state. Threads should have
a properly synchronized method of communicating with each other (e.g.
CSP-style channels), but shouldn't be able to access each other's state.

[-- Attachment #2: Type: text/html, Size: 2075 bytes --]

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Søren Pilgård <fiskomaten@gmail.com>
> Date: Sun, 9 Oct 2016 14:43:36 +0200
> Cc: John Wiegley <jwiegley@gmail.com>, emacs-devel@gnu.org,
> 	Lars Brinkhoff <lars@nocrew.org>
> 
> The thing about concurrency is that it is very hard to integrate
> meaningfully and robustly into Emacs.

See the concurrency branch in the Git repo for one possible way of
doing that.

> As you can see, introducing concurrency has wide implications for Emacs.
> Finding a way to go about it that will still allow old "naive" code to
> function correctly is what makes it so tough.

Fortunately, these issues were already figured out, and the result
needs just some final touches and some serious testing on various
platforms.  Motivated individuals are welcome to work on that branch
and bring it up to speed.

Thanks.

Re: Guile emacs thread (again)

[Lluís]

Christopher Allan Webber writes:
[...]
> I found it very inspiring.  It seems those things are fairly close.

> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?
>  - Is anyone running it?  How's it going for you?

I just read the thread, and many different and entwined points have been raised,
which I think make the discussion harder to follow. Thus I would like to
summarize the issues I've read regarding Guile (feel free to reply with any
points you think I've missed or misinterpreted):

* Manpower / responsiveness

  There's not much to say here. Whatever layers Emacs decides to use (if any),
  there should be an appropriate response delay to handle issues.

* Direction / design

  There could be design decisions conflicting between Emacs' and Guile's
  desires. Such issues can only be resolved through mutual and good-hearted
  understanding.

* Multi-language support

  As someone said, using Guile does not necessarily mean exposing all these
  languages in Emacs (i.e., just use Guile's lower-level compiler/VM layers).

* Development model / organization

  It has been argued that Guile's development model does not give enough
  stability assurances. I'm sure that's something that can be easily fixed if
  Guile's developers slightly change their workflow to maintain a more stable
  branch that Emacs (and other projects) can use.

* It's a GNU project

  If that really counts for any purpose.


Thus I would say that the only critical point here is the first one
(manpower/responsiveness). To some extent, the second point could be a problem
too (direction/design), but targeting Emacs to the lower-level layers of Guile
could limit the impact of this factor.


Thanks,
  Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Guile emacs thread (again)

[Lally Singh]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

On Wed, Sep 17, 2014 at 3:29 PM, Lluís <xscript@gmx.net> wrote:

> [..]
> Thus I would say that the only critical point here is the first one
> (manpower/responsiveness). To some extent, the second point could be a
> problem
> too (direction/design), but targeting Emacs to the lower-level layers of
> Guile
> could limit the impact of this factor.
>

We should also be mindful of the effects of what such a large project as
emacs would have suddenly coming into a project with a substantially
smaller community.  The emacs'ers could end up taking over, even if nobody
intended for that to happen.

[-- Attachment #2: Type: text/html, Size: 986 bytes --]

Re: Guile emacs thread (again)

[Robin Templeton]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Hello!  It's been a while since the topic has come up on this list, but
> many of us are interested in it, and maybe some developers don't know,
> and I hadn't seen any conversations on the list despite recent progress.
>
> Anyway, it seems that BT Templeton's emacs on guile project this summer
> has gone fairly well, and it looks like almost everything runs.  See:
>   http://www.emacswiki.org/emacs/GuileEmacs
>
> I remember reading Andy Wingo's email about this a few years ago,
> "guile and emacs and elisp, oh my!":
>   https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00665.html
>
> I found it very inspiring.  It seems those things are fairly close.
>
> So this email is partly a:
>  - What now?  What's the chance of work towards guilemacs moving over to
>    an official emacs git branch, and that port happening, anytime soon?
>  - Is anyone running it?  How's it going for you?
>
> (I'm mid-compile of the guile wip branch right now...!)
>  - Chris

First, I'd like to clarify some things about Guile-Emacs for those just
tuning in. The goal of the project is to create a better implementation
of Elisp than Emacs provides -- providing full compatibility with
existing Elisp packages -- and to use that as Emacs's Lisp
implementation. That implementation is Guile-Elisp, a compiler targeting
the Guile VM, and Guile-Emacs is the variant of Emacs based on this
compiler.

What will this mean for Emacs? Better performance and new language
features. Guile provides a fast virtual machine and a sophisticated
compiler infrastructure, and the Guile VM outperforms the Elisp bytecode
interpreter on a variety of benchmarks. Guile natively provides many
features from Scheme and Common Lisp not present in Elisp, including a
full numeric tower, structure types, CLOS-based object orientation, a
foreign function interface, delimited continuations, a module system,
hygienic macros, multiple values, and threads.

It will not mean that Elisp programs will need to be rewritten in
Scheme, or even that Emacs will necessarily support extensions written
in Scheme (much less JavaScript!). Interaction with other languages is a
possibility, but the exact relationship between Elisp and other
languages will be up to the Emacs maintainers and community. Elisp is
here to stay; Guile-Emacs will not remove Elisp but will instead enable
and accelerate its evolution.

The fifth Guile-Emacs Summer of Code project has concluded successfully.
Thanks to Andy Wingo for supervising my work, and to Google for funding
the project. This summer is when everything really came together. The
project previously had two independent components developed in parallel:
Guile-Elisp and Guile-Emacs. Guile-Emacs used libguile for garbage
collection and object representation, but retained the original
interpreter and bytecode interpreter and did not, initially, use the
compiler at all. The two components have been successfully united. Emacs
now compiles all Elisp code using Guile-Elisp, and all programs are
executed on the Guile VM. The old interpreter and bytecode interpreter
have been excised entirely. Guile-Emacs loads, compiles and executes the
programs included in Emacs by default, plus Gnus, Org-Mode, and more.

Additionally, Guile-Emacs was rebased onto a recent development version
of Emacs. Guile-Elisp was ported to guile master, which will become
Guile 2.2 and includes a new and faster virtual machine and upgraded
compiler tower. Other additions include compiler macros a la Common Lisp
used for inlining calls to Guile primitives, a reasonably complete FFI,
and Scheme Interaction Mode. The build system has been updated so that
Guile-Emacs can build itself cleanly from a fresh git checkout, with no
assistance from an existing Emacs installation and no undocumented
command line jiggery-pokery. Now the build procedure is "./configure &&
make".

Furthermore, I made a number of modifications to low-level support code
for Guile compatibility. The TL;DR on this is that the language changes
introduced in Guile-Elisp shouldn't matter much for normal programs, but
do require adjustment to low-level code, including debuggers,
macroexpanders and code-walkers. I'll write up a summary of the
differences separately, but the intent is that Guile-Emacs should be no
more disruptive than the addition of lexical binding from an Elisp
programmer's perspective.

For more information on Guile-Emacs, please see the Wiki page at
<http://www.emacswiki.org/emacs/GuileEmacs>, and keep an eye on
<http://www.guile-emacs.org/> for future updates.

Happy hacking,
Robin
-- 
Inteligenta persono lernas la lingvon Esperanton rapide kaj facile.
Esperanto estas moderna, kultura lingvo por la mondo. Simpla, fleksebla,
belsona, Esperanto estas la praktika solvo de la problemo de universala
interkompreno. Lernu la interlingvon Esperanton!

Re: Guile emacs thread (again)

[Christopher Allan Webber]

Robin Templeton writes:

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>> Hello!  It's been a while since the topic has come up on this list, but
>> many of us are interested in it, and maybe some developers don't know,
>> and I hadn't seen any conversations on the list despite recent progress.
>>
>> Anyway, it seems that BT Templeton's emacs on guile project this summer
>> has gone fairly well, and it looks like almost everything runs.  See:
>>   http://www.emacswiki.org/emacs/GuileEmacs
>>
>> I remember reading Andy Wingo's email about this a few years ago,
>> "guile and emacs and elisp, oh my!":
>>   https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00665.html
>>
>> I found it very inspiring.  It seems those things are fairly close.
>>
>> So this email is partly a:
>>  - What now?  What's the chance of work towards guilemacs moving over to
>>    an official emacs git branch, and that port happening, anytime soon?
>>  - Is anyone running it?  How's it going for you?
>>
>> (I'm mid-compile of the guile wip branch right now...!)
>>  - Chris
>
> First, I'd like to clarify some things about Guile-Emacs for those just
> tuning in. The goal of the project is to create a better implementation
> of Elisp than Emacs provides -- providing full compatibility with
> existing Elisp packages -- and to use that as Emacs's Lisp
> implementation. That implementation is Guile-Elisp, a compiler targeting
> the Guile VM, and Guile-Emacs is the variant of Emacs based on this
> compiler.
>
> What will this mean for Emacs? Better performance and new language
> features. Guile provides a fast virtual machine and a sophisticated
> compiler infrastructure, and the Guile VM outperforms the Elisp bytecode
> interpreter on a variety of benchmarks. Guile natively provides many
> features from Scheme and Common Lisp not present in Elisp, including a
> full numeric tower, structure types, CLOS-based object orientation, a
> foreign function interface, delimited continuations, a module system,
> hygienic macros, multiple values, and threads.
>
> It will not mean that Elisp programs will need to be rewritten in
> Scheme, or even that Emacs will necessarily support extensions written
> in Scheme (much less JavaScript!). Interaction with other languages is a
> possibility, but the exact relationship between Elisp and other
> languages will be up to the Emacs maintainers and community. Elisp is
> here to stay; Guile-Emacs will not remove Elisp but will instead enable
> and accelerate its evolution.
>
> The fifth Guile-Emacs Summer of Code project has concluded successfully.
> Thanks to Andy Wingo for supervising my work, and to Google for funding
> the project. This summer is when everything really came together. The
> project previously had two independent components developed in parallel:
> Guile-Elisp and Guile-Emacs. Guile-Emacs used libguile for garbage
> collection and object representation, but retained the original
> interpreter and bytecode interpreter and did not, initially, use the
> compiler at all. The two components have been successfully united. Emacs
> now compiles all Elisp code using Guile-Elisp, and all programs are
> executed on the Guile VM. The old interpreter and bytecode interpreter
> have been excised entirely. Guile-Emacs loads, compiles and executes the
> programs included in Emacs by default, plus Gnus, Org-Mode, and more.
>
> Additionally, Guile-Emacs was rebased onto a recent development version
> of Emacs. Guile-Elisp was ported to guile master, which will become
> Guile 2.2 and includes a new and faster virtual machine and upgraded
> compiler tower. Other additions include compiler macros a la Common Lisp
> used for inlining calls to Guile primitives, a reasonably complete FFI,
> and Scheme Interaction Mode. The build system has been updated so that
> Guile-Emacs can build itself cleanly from a fresh git checkout, with no
> assistance from an existing Emacs installation and no undocumented
> command line jiggery-pokery. Now the build procedure is "./configure &&
> make".
>
> Furthermore, I made a number of modifications to low-level support code
> for Guile compatibility. The TL;DR on this is that the language changes
> introduced in Guile-Elisp shouldn't matter much for normal programs, but
> do require adjustment to low-level code, including debuggers,
> macroexpanders and code-walkers. I'll write up a summary of the
> differences separately, but the intent is that Guile-Emacs should be no
> more disruptive than the addition of lexical binding from an Elisp
> programmer's perspective.
>
> For more information on Guile-Emacs, please see the Wiki page at
> <http://www.emacswiki.org/emacs/GuileEmacs>, and keep an eye on
> <http://www.guile-emacs.org/> for future updates.
>
> Happy hacking,
> Robin

Hey Robin, thanks for this really great email.  I was excited for
guilemacs before, but I'm even more excited now.  :)

Re: Guile emacs thread (again)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    It will not mean that Elisp programs will need to be rewritten in
    Scheme, or even that Emacs will necessarily support extensions written
    in Scheme

Supporting Emacs extensions written in Scheme is part of the goal I
had in mind when I proposed this.  That's the tangible benefit that we
would get from Scheme support.  If all Scheme does is serve as a platform
for Emacs Lisp, it is no real advance.

It is ok if it is necessary to write these extensions in a slightly
unusual style to get them to interoperate properly with Emacs Lisp
programs.  For instance, maybe it would require using special
functions to test for Elisp nil.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Guile emacs thread (again)

[Eli Zaretskii]

> Date: Sat, 20 Sep 2014 09:20:16 -0400
> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Supporting Emacs extensions written in Scheme is part of the goal I
> had in mind when I proposed this.  That's the tangible benefit that we
> would get from Scheme support.  If all Scheme does is serve as a platform
> for Emacs Lisp, it is no real advance.

Then perhaps the goals of the Guile-Emacs project should be shifted,
such that the first goal is to have Emacs that can use Scheme
extensions, and push the goal of having Emacs Lisp based on Scheme
farther into the future.  Doing so would make sense to me, since this
is what other GNU projects do (Make, GDB), and it postpones the need
for solving a whole lot of non-trivial problems, some of which were
discussed here.  It would also take the sting out of the fears
expressed by some that Guile development team might not be up to the
task of supporting an infrastructure which Emacs cannot do without.
If Guile is just another extension mechanism, losing it will not be a
fatal blow.

Re: Guile emacs thread (again)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > Supporting Emacs extensions written in Scheme is part of the goal I
    > had in mind when I proposed this.  That's the tangible benefit that we
    > would get from Scheme support.  If all Scheme does is serve as a platform
    > for Emacs Lisp, it is no real advance.

    Then perhaps the goals of the Guile-Emacs project should be shifted,
    such that the first goal is to have Emacs that can use Scheme
    extensions, and push the goal of having Emacs Lisp based on Scheme
    farther into the future.

The extensions written in Scheme need to interoperate with all the
extensions written in Emacs Lisp.  Whether the roadmap for getting
there ought to be changed, I am not sure.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Guile emacs thread (again)

[Stefan Monnier]

> Then perhaps the goals of the Guile-Emacs project should be shifted,
> such that the first goal is to have Emacs that can use Scheme
> extensions,

On the contrary, this should be one of the last goals.
The main goal should be to maximize the amount of existing code that
Emacs can benefit from.

IOW for Emacs-specific libraries, this requires bug-compatible Elisp
support, and for the rest it requires an FFI (and if some of the rest
can be supported more directly by sharing the underlying VM, that's even
better, tho I'd simply consider it as a "particular good kind of FFI").

Being able to access existing Scheme libraries is great.  But being able
to write Emacs extensions in Scheme is definitely not a priority and
might even be harmful to Emacs by making splitting the community and
wasting time on improving interoperability between the languages.


        Stefan

Re: Guile emacs thread (again)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rms@gnu.org,  robin@terpri.org,  emacs-devel@gnu.org
> Date: Sun, 21 Sep 2014 12:16:07 -0400
> 
> > Then perhaps the goals of the Guile-Emacs project should be shifted,
> > such that the first goal is to have Emacs that can use Scheme
> > extensions,
> 
> On the contrary, this should be one of the last goals.
> The main goal should be to maximize the amount of existing code that
> Emacs can benefit from.

You mean, what Guile already has, like FFI?  You will have that by
using Scheme extensions.  How's that a problem?

> Being able to access existing Scheme libraries is great.  But being able
> to write Emacs extensions in Scheme is definitely not a priority and
> might even be harmful to Emacs by making splitting the community and
> wasting time on improving interoperability between the languages.

OTOH, doing what I suggest is much easier, and will allow both sides
to learn about the other one without the kind of pressure which comes
with replacing the existing Lisp interpreter with a new one.

Re: Guile emacs thread (again)

[Stefan Monnier]

> OTOH, doing what I suggest is much easier, and will allow both sides
> to learn about the other one without the kind of pressure which comes
> with replacing the existing Lisp interpreter with a new one.

That was the state of Guile-Emacs last year, pretty much (i.e. don't use
the Guile-VM byte-code engine, but keep using our own interpreters
instead, while sharing the heap with Guile, i.e. using Guile's GC).

That's OK as well.

The purpose of using the Guile-VM is that it should be able to give us
better performance.


        Stefan

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lally Singh]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

So if this a runtime system issue, what about elisp on LLVM?  Let the LLVM
project handle the backend and performance issues, and emacs can maintain
one language frontend.  There are plenty of people working on that, so
emacs can ride that for almost free.

I'm assuming that there are reasons why it doesn't work, as someone
(apparently) did the work some time ago:
https://github.com/boostpro/emacs-llvm-jit  Perhaps it just needs a little
TLC?

[-- Attachment #2: Type: text/html, Size: 577 bytes --]

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Tom]

Lally Singh <lally.singh <at> gmail.com> writes:

> 
> So if this a runtime system issue, what about elisp on LLVM?  Let the LLVM
project handle the backend and performance issues, and emacs can maintain
one language frontend.  There are plenty of people working on that, so emacs
can ride that for almost free.

In the long run this would be the most practical solution. Choosing
a well supported and widely used VM which gets  tons of developer 
attention, so the VM development would be taken care of, 
and Elisp could be implemented as a frontend.

This approach has the advantage of supporting the existing Elisp code
base and it would also make it possible to use other languages to extend
emacs, because widdely used VMs have frontend implementations for
many modern languages (Python, etc.)

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It is not acceptable to base a GNU package on LLVM.  It is a
non-copylefted competitor to an important GNU package.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lally Singh]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

Would DotGNU work?  Ignore C# and put in a elisp frontend.  The benefit
being that we can watch what the larger efforts are doing in those fronts,
and pick-and-choose the good ideas with little effort on our side.

DotGNU would get a boost of support from emacs, and it's a familiar enough
environment that a lot of programmers would be willing to go in and hack on
it.  They'd often already have transferrable experience on the proprietary
versions of DotGNU, but would be able to vent their innate need to hack on
dotgnu instead of the more proprietary versions.  Emacs has a huge user
footprint, and leveraging it to boost another project would help.
 Especially a project that rides rather close to a very popular proprietary
platform - that's a lot of hackers to harness.

On Wed, Sep 17, 2014 at 7:43 AM, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> It is not acceptable to base a GNU package on LLVM.  It is a
> non-copylefted competitor to an important GNU package.
>
> --
> Dr Richard Stallman
> President, Free Software Foundation
> 51 Franklin St
> Boston MA 02110
> USA
> www.fsf.org  www.gnu.org
> Skype: No way! That's nonfree (freedom-denying) software.
>   Use Ekiga or an ordinary phone call.
>
>

[-- Attachment #2: Type: text/html, Size: 1981 bytes --]

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Kristian Nygaard Jensen]

 > Of course, that's for the language side, but on the implementation side,
 > I don't really know what Common-Lisp implementation we could re-use
 > (both GNU implementations are dormant, so there's no manpower for us
 > tap into).  Still: there are many Common-Lisp implementations out there,
 > so there's probably one that could work for us.

Embeddable Common-Lisp (http://sourceforge.net/projects/ecls/) seems 
alive, it is lgpl, so there would be no license issue

-- 

Kristian Nygaard Jensen

Emacs Lisp's future (was: Guile emacs thread (again))

[Nic Ferrier]

Stefan Monnier wrote:

> First, of course we can keep on evolving Elisp on its own.  This has
> worked OK for the last 30 years, so it's not such a terrible choice.
> The main problems I see with that:
> - Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
>   noticed more often.
> - Lack of some features, most notably FFI and concurrency.
> - Lack of manpower.
>
> This last point is for me the strongest motivation to try and move to
> some other system, where we could use other people's work.

I don't see that this is going to happen though. Emacs is an unusual
system. Moving the extension language to another community is just going
to cause more arguing along the lines of "this is how X lang does it" vs
"but we're Emacs and don't want to do it like that".

My view is we should improve the contribution process to get more
manpower for elisp. We have been doing that as a community. As a
reminder we have:

- adopted packaging allowing many more people to contribute pure elisp
- accepted a move to the most commonly used support tools (git, etc...)
- started to talk about changing the documentation format to a more
  common format

I see a new spirit of openness and willingness to change in the Emacs
community and it's really great.

I would implore you, my fellow emacs hackers, not to make too hasty a
decision on platform. Guile-Emacs may be cool, but if we can increase
developer diversity in Emacs through git and so on (I for one will be
contributing to the core thanks to this) then we may get all the
advantages of the Guile VM without having to go to Guile.

I'm sure there is more that we could do to get more man and woman
power. I hope that we consider those things as well as techy projects
like switching to Guile's VM.


Nic