Emacs rewrite in a maintainable language

Emacs rewrite in a maintainable language

[Przemysław Wojnowski]

Hello everybody,

Recently I was going through Emacs C code and immediately one thing come to my
mind: "Did anyone consider rewrite it a more maintainable language?" :-)

Did anyone consider pros/cons of having base code in other language?
AFAIK maintenance cost of C code is pretty high, because the code is tangled,
has no tests and the language has its limits (e.g. no namespaces, no
abstractions, etc.). Maintaining that and adding new features will be harder
and harder.

Having Emacs base code in a more maintainable language (maybe some Scheme
version?) would reduce maintenance cost over the years. Also more maintainable
code makes it easier for others to joins and improve it.

Quick search gave this http://tromey.com/blog/?p=709 but the article talks only 
about performance, not maintainability, which is extremely important for long 
living projects like Emacs.

It's just a question, not a proposition. :-)

Cheers,
Przemysław

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Przemysław Wojnowski <esperanto@cumego.com> writes:

> Hello everybody,
>
> Recently I was going through Emacs C code and immediately one thing
> come to my mind: "Did anyone consider rewrite it a more maintainable
> language?" :-)

Emacs is mostly written in Emacs Lisp.  C is used only for the core.
Encapsulation, namespaces and modularity are not really an issue there
as it is a closed and confined working set.

They are an issue in Elisp.  So if you are worried about
maintainability, rewriting Elisp into a more maintainable language would
be by far more effective.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Marcin Borkowski]

On 2015-10-11, at 10:17, David Kastrup <dak@gnu.org> wrote:

> Przemysław Wojnowski <esperanto@cumego.com> writes:
>
>> Hello everybody,
>>
>> Recently I was going through Emacs C code and immediately one thing
>> come to my mind: "Did anyone consider rewrite it a more maintainable
>> language?" :-)
>
> Emacs is mostly written in Emacs Lisp.  C is used only for the core.
> Encapsulation, namespaces and modularity are not really an issue there
> as it is a closed and confined working set.
>
> They are an issue in Elisp.  So if you are worried about
> maintainability, rewriting Elisp into a more maintainable language would
> be by far more effective.

One thing that comes to my mind when I read this discussion is something
that was mentioned (AFAIR) on one of Emacs blogs a few years ago (or
so).

That is, it might be a good idea to move some parts of Emacs core (which
are not system-dependent or performance-critical, e.g., parts of
interactive behavior like `self-insert-command' might fit here) from
C to Elisp.  We would gain at least two (connected) important things
this way:

1. more of Emacs code would be easier to understand for more people.
And hence,

2. (even) more of Emacs would be customizable without C wizardry.

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Marcin Borkowski <mbork@mbork.pl> writes:

> That is, it might be a good idea to move some parts of Emacs core (which are
> not system-dependent or performance-critical, e.g., parts of interactive
> behavior like `self-insert-command' might fit here) from C to Elisp.

If you can identify functions whose efficiency and behavior are not affected
by moving them to Elisp, that would be a great idea.

John

Re: Emacs rewrite in a maintainable language

[Óscar Fuentes]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Marcin Borkowski <mbork@mbork.pl> writes:
>
>> That is, it might be a good idea to move some parts of Emacs core (which are
>> not system-dependent or performance-critical, e.g., parts of interactive
>> behavior like `self-insert-command' might fit here) from C to Elisp.
>
> If you can identify functions whose efficiency and behavior are not affected
> by moving them to Elisp, that would be a great idea.

One year or two ago, I found some obvious candidates and asked for
Stefan's opinion, who didn't show much interest. Can't locate the
message now.

Searching the C sources for small, loop-free functions marked as
interactive should not take much time.

Re: Emacs rewrite in a maintainable language

[Marcin Borkowski]

On 2015-10-12, at 00:14, John Wiegley <johnw@newartisans.com> wrote:

>>>>>> Marcin Borkowski <mbork@mbork.pl> writes:
>
>> That is, it might be a good idea to move some parts of Emacs core (which are
>> not system-dependent or performance-critical, e.g., parts of interactive
>> behavior like `self-insert-command' might fit here) from C to Elisp.
>
> If you can identify functions whose efficiency and behavior are not affected
> by moving them to Elisp, that would be a great idea.

Hi John,

I'm not sure whether you were ironic or not, but assuming you were not,
here's my answer: no, I cannot /identify/ them, but sometimes I can
/suspect/ them.  More seasoned developers should make the decision,
though.

Sometimes I look up some function or command, and I'm surprised it's
defined in C and not Elisp.  I could start making notes about those
situations.

As I said, `self-insert-command' seems a natural candidate.  It is
probably almost never called from Elisp code, and I would guess it's
similar for C code.  (One exception might be keyboard macros - if yes,
this might be a performance bottleneck.)

And btw: is there a way to search for functions according to the
filename they are defined in, possibly using a regex, so that one could
search e.g. for functions defined in C by specifying "\.c$"?
(Drew?  Icicles?)

> John

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Marcin Borkowski <mbork@mbork.pl> writes:

> I'm not sure whether you were ironic or not, but assuming you were not,
> here's my answer: no, I cannot /identify/ them, but sometimes I can
> /suspect/ them. More seasoned developers should make the decision, though.
>
> Sometimes I look up some function or command, and I'm surprised it's defined
> in C and not Elisp. I could start making notes about those situations.

No, not ironic, just encouraging you. :)

> As I said, `self-insert-command' seems a natural candidate. It is probably
> almost never called from Elisp code, and I would guess it's similar for C
> code. (One exception might be keyboard macros - if yes, this might be a
> performance bottleneck.)

I wonder, since this command is called so often, what the impact would be in
this case. We'd need hard numbers to show that it doesn't affect input latency
at all.

John

Re: Emacs rewrite in a maintainable language

[Óscar Fuentes]

Marcin Borkowski <mbork@mbork.pl> writes:

> I'm not sure whether you were ironic or not,

I think John is being totally serious here. Why he shouldn't?

> but assuming you were not,
> here's my answer: no, I cannot /identify/ them, but sometimes I can
> /suspect/ them.  More seasoned developers should make the decision,
> though.

See my other message. The last detail is that the function is not called
from C. Maybe we should check that it is not used in very hot spots in
the Elisp sources as well. If you are not sure, ask here.

[snip]

> And btw: is there a way to search for functions according to the
> filename they are defined in, possibly using a regex, so that one could
> search e.g. for functions defined in C by specifying "\.c$"?
> (Drew?  Icicles?)

grep -nH "DEFUN (" *.c

in the src/ directory will show you the functions defined in C which are
available in Elisp.

RE: Emacs rewrite in a maintainable language

[Drew Adams]

> And btw: is there a way to search for functions according to the
> filename they are defined in, possibly using a regex, so that one
> could search e.g. for functions defined in C by specifying "\.c$"?
> (Drew?  Icicles?)

If you create a TAGS file for stuff defined only in C files then
sure.  Vanilla `find-tag' can do that (as well as similar Icicles
commands).

(You could also use `icicle-find-file', matching both the file
name (e.g. .*\.c) and a file-content regexp that matches
Emacs-Lisp function definitions in the C source files.)

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Marcin Borkowski <mbork@mbork.pl>
> Date: Mon, 12 Oct 2015 00:02:43 +0200
> 
> it might be a good idea to move some parts of Emacs core (which are
> not system-dependent or performance-critical, e.g., parts of
> interactive behavior like `self-insert-command' might fit here) from
> C to Elisp.

We are doing this all the time.  Patches to that effect are welcome.

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> Date: Mon, 12 Oct 2015 05:43:02 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: esperanto@cumego.com, emacs-devel@gnu.org
> 
> > From: Marcin Borkowski <mbork@mbork.pl>
> > Date: Mon, 12 Oct 2015 00:02:43 +0200
> > 
> > it might be a good idea to move some parts of Emacs core (which are
> > not system-dependent or performance-critical, e.g., parts of
> > interactive behavior like `self-insert-command' might fit here) from
> > C to Elisp.
> 
> We are doing this all the time.  Patches to that effect are welcome.

Btw, most of the code for self-insert-command is in an internal
function, so rewriting that in Lisp is unlikely to help people
understand its internal hair.

More importantly, most of the complicated parts of the C code cannot
be rewritten in Lisp anyway, so only limited progress can be made this
way, which will not make it easier to understand the core.

Re: Emacs rewrite in a maintainable language

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

On 10/12/2015 08:35 AM, Eli Zaretskii wrote:
>> Date: Mon, 12 Oct 2015 05:43:02 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: esperanto@cumego.com, emacs-devel@gnu.org
>>
>>> From: Marcin Borkowski <mbork@mbork.pl>
>>> Date: Mon, 12 Oct 2015 00:02:43 +0200
>>>
>>> it might be a good idea to move some parts of Emacs core (which are
>>> not system-dependent or performance-critical, e.g., parts of
>>> interactive behavior like `self-insert-command' might fit here) from
>>> C to Elisp.
>>
>> We are doing this all the time.  Patches to that effect are welcome.
> 
> Btw, most of the code for self-insert-command is in an internal
> function, so rewriting that in Lisp is unlikely to help people
> understand its internal hair.
> 
> More importantly, most of the complicated parts of the C code cannot
> be rewritten in Lisp anyway, so only limited progress can be made this
> way, which will not make it easier to understand the core.

Exactly. The complexity in the core is not a result of its
implementation language. It's complex because it's complex. Suddenly
sprinkling "class" and vtables through the core isn't going to reduce
the complexity of the implemented logic.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs rewrite in a maintainable language

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That is, it might be a good idea to move some parts of Emacs core (which
  > are not system-dependent or performance-critical, e.g., parts of
  > interactive behavior like `self-insert-command' might fit here) from
  > C to Elisp.  We would gain at least two (connected) important things
  > this way:

In days long ago,
When clocks were so slow,
These basic keystrokes
Took too long for folks,
Unless they were writ
In C code to flit.
Today that's not so,
In Lisp they can go.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs rewrite in a maintainable language

[Alexis]

Przemysław Wojnowski <esperanto@cumego.com> writes:

> Recently I was going through Emacs C code and immediately one 
> thing come to my mind: "Did anyone consider rewrite it a more 
> maintainable language?" :-)
>
> Did anyone consider pros/cons of having base code in other 
> language?  AFAIK maintenance cost of C code is pretty high, 
> because the code is tangled, has no tests and the language has 
> its limits (e.g. no namespaces, no abstractions, 
> etc.). Maintaining that and adding new features will be harder 
> and harder.
>
> Having Emacs base code in a more maintainable language (maybe 
> some Scheme version?) would reduce maintenance cost over the 
> years. Also more maintainable code makes it easier for others to 
> joins and improve it.

In my experience, this suggestion comes up regularly; here's a 
brief comment i wrote on this topic about a year ago:

https://www.reddit.com/r/emacs/comments/2l1gp6/so_why_moving_the_code_base_of_emacs_to_another/clqp6p9


Alexis.

Re: Emacs rewrite in a maintainable language

[Przemysław Wojnowski]

Thank you very much!

Re: Emacs rewrite in a maintainable language

[Thomas Koch]

https://en.wikipedia.org/wiki/Yi_%28editor%29

Haskell is my personal favorite language when talking about maintainability.
A migration path from emacs to Yi would be wonderful, e.g. execute elisp in 
Yi?
You can also write emacs extensions in Haskell:
http://www.stackbuilders.com/news/having-fun-with-emacs-in-haskell

Thomas Koch

Re: Emacs rewrite in a maintainable language

[Dmitry Gutov]

On 10/11/2015 02:23 PM, Thomas Koch wrote:
> https://en.wikipedia.org/wiki/Yi_%28editor%29
>
> Haskell is my personal favorite language when talking about maintainability.
> A migration path from emacs to Yi would be wonderful, e.g. execute elisp in
> Yi?

I'm pretty sure the current core maintainers don't know Haskell, or 
Rust, or any other modern "hipster" language, to the extent a rewrite 
would require. Probably not everyone knows Common Lisp either.

So any volunteers to do the rewrite and then maintain the resulting 
codebase, should please step up.

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/11/2015 02:23 PM, Thomas Koch wrote:
>> https://en.wikipedia.org/wiki/Yi_%28editor%29
>>
>> Haskell is my personal favorite language when talking about maintainability.
>> A migration path from emacs to Yi would be wonderful, e.g. execute elisp in
>> Yi?
>
> I'm pretty sure the current core maintainers don't know Haskell, or
> Rust, or any other modern "hipster" language, to the extent a rewrite
> would require. Probably not everyone knows Common Lisp either.
>
> So any volunteers to do the rewrite and then maintain the resulting
> codebase, should please step up.

They did so long ago.  Hemlock is a Common Lisp Emacs recast if
I remember correctly.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Dmitry Gutov]

On 10/11/2015 04:36 PM, David Kastrup wrote:

> They did so long ago.  Hemlock is a Common Lisp Emacs recast if
> I remember correctly.

Can it run Emacs Lisp? Or does it have all core Elisp ported to CL, at 
least?

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/11/2015 04:36 PM, David Kastrup wrote:
>
>> They did so long ago.  Hemlock is a Common Lisp Emacs recast if
>> I remember correctly.
>
> Can it run Emacs Lisp? Or does it have all core Elisp ported to CL, at
> least?

I think it lost track of Emacs long, long ago.  It's heighday was before
XEmacs even existed as such IIRC.  Probably even before Lucid Emacs
forked.  And even then, I doubt that things like Emacs Calc ran.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Dmitry Gutov]

On 10/11/2015 04:55 PM, David Kastrup wrote:

> I think it lost track of Emacs long, long ago.  It's heighday was before
> XEmacs even existed as such IIRC.  Probably even before Lucid Emacs
> forked.  And even then, I doubt that things like Emacs Calc ran.

In that case, it's not really a (successful) rewrite. Not even back then.

Re: Emacs rewrite in a maintainable language

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1827 bytes --]

On 10/11/2015 01:11 AM, Przemysław Wojnowski wrote:
> Hello everybody,
> 
> Recently I was going through Emacs C code and immediately one thing come
> to my
> mind: "Did anyone consider rewrite it a more maintainable language?" :-)
> 
> Did anyone consider pros/cons of having base code in other language?
> AFAIK maintenance cost of C code is pretty high, because the code is
> tangled,
> has no tests and the language has its limits (e.g. no namespaces, no
> abstractions, etc.). Maintaining that and adding new features will be
> harder
> and harder.

As others have pointed out, most of Emacs is written in Emacs Lisp.

As for the rest: a common understanding among a certain crowd these days
is that C is a broken, legacy language that nobody really understands.
But the operating system kernel and window system you used to compose
your message are a giant C programs. Linux and Xorg are blasphemies
against hipster software engineering. Both generally work fine, as does
Emacs.

The Emacs C core isn't going anywhere. You're welcome to try to rewrite
it, but if C really were such an albatross, I'd expect at least one of
Hemlock, JEmacs, Ermacs, Ymacs, Climacs, or EdWin to have become a
viable contender by now. Yet here we are, talking about GNU Emacs.

We can improve the Emacs core by making it less "tangled" (e.g.,
abstracting away frame functions so that we can support more than one
window system) and by writing better tests for some of the hairier bits
of functionality (e.g., the recently-broken syntax engine). These
potential improvements have nothing to do with language choice.

I'd rather people focus on fixing long-standing problems and adding new
features than satisfying some aesthetic impulse by reimplementing the
same old functionality in a new language.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs rewrite in a maintainable language

[Fabrice Popineau]

Przemysław Wojnowski <esperanto <at> cumego.com> writes:

> 
> Hello everybody,
> 
> Recently I was going through Emacs C code and immediately one thing come 
to my
> mind: "Did anyone consider rewrite it a more maintainable language?" 

You mean in Common Lisp ? ;-)

Fabrice

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Przemysław Wojnowski <esperanto@cumego.com> writes:

> Recently I was going through Emacs C code and immediately one thing come to
> my mind: "Did anyone consider rewrite it a more maintainable language?" :-)

Hi Przemysław,

There is one reason I can think of not to rewrite Emacs in another language:
Because the language it's written in is working for us.

If C presented difficulties in terms of capability, efficiency, contribution,
debugging, etc., that would naturally drive us toward another language. But as
it stands, C is well understood, easy to resolve performance issues, there are
superb debugging and analysis tools available, and the runtime is exceedingly
mature.

And, as some have said, C code represents only a fraction of what Emacs is to
the user. Switching from Emacs Lisp to another high-level language might be a
more interesting discussion, which you could take up with the people working
on such projects, like Guile Emacs. If someone can prove to me that we gain a
lot and lose a little from such a switch, it's definitely worth considering!

John

Re: Emacs rewrite in a maintainable language

[Óscar Fuentes]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Przemysław Wojnowski <esperanto@cumego.com> writes:
>
>> Recently I was going through Emacs C code and immediately one thing come to
>> my mind: "Did anyone consider rewrite it a more maintainable language?" :-)
>
> Hi Przemysław,
>
> There is one reason I can think of not to rewrite Emacs in another language:
> Because the language it's written in is working for us.
>
> If C presented difficulties in terms of capability, efficiency, contribution,
> debugging, etc., that would naturally drive us toward another language. But as
> it stands, C is well understood, easy to resolve performance issues, there are
> superb debugging and analysis tools available, and the runtime is exceedingly
> mature.
>
> And, as some have said, C code represents only a fraction of what Emacs is to
> the user. Switching from Emacs Lisp to another high-level language might be a
> more interesting discussion, which you could take up with the people working
> on such projects, like Guile Emacs. If someone can prove to me that we gain a
> lot and lose a little from such a switch, it's definitely worth considering!

We have plenty of people writing Elisp code, either for the Emacs core,
packages distributed with the core or ELPA, or external packages.
However, the number of people able of doing significant work on the C
core is tiny. By your depiction of C, this is backwards: the popular
language has a few contributors, the application-specific language is
well populated.

The truth is that the C core is an obfuscated mess (ok ok, now someone
will come saying that the Emacs C code base is very well and all;
someone who surely has at least 10 years of experience with the C core.)

I wouldn't touch the C core with a ten foot pole, except for something
that is an obvious change, and then request a review by a veteran. I
know C, I have many years of C++ experience, but the Emacs C core is a
pseudo-language on itself, because C sucks at abstracting things. It
sucks too at helping the hacker to not shoot himself on the foot.

In large part because that reason, there are key are parts of the C core
that are only understood by one active hacker (being optimistic here).
This is a huge liability for Emacs. I don't know how this fact fits the
"C is working for us" stance.

Replacing C with a language that helps people to write readable, safe,
efficient and maintainable code without having to study the existing
code base for months looks quite appealing to me. That replacement could
be gradual (no "stopping the world while The Big Rewrite is underway.")
If that language is not so popular as C, that's no big problem as long
as it makes things less daunting for those willing to work on the Emacs
core. As already said, there is no dramatic shortage of people who learn
Elisp just for hacking Emacs.

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 11 Oct 2015 20:32:59 +0200
> 
> Replacing C with a language that helps people to write readable, safe,
> efficient and maintainable code without having to study the existing
> code base for months looks quite appealing to me. That replacement could
> be gradual (no "stopping the world while The Big Rewrite is underway.")
> If that language is not so popular as C, that's no big problem as long
> as it makes things less daunting for those willing to work on the Emacs
> core. As already said, there is no dramatic shortage of people who learn
> Elisp just for hacking Emacs.

It's all a pipe dream, no more, unless someone steps forward who
really intends and is able to do the job.  And that is even before we
start talking about which alternative language should that be,
something that in itself is not a trivial decision.

Also, please note that, due to a strange coincidence (or maybe
something else), doing so will require the same talents you rightfully
say are at premium.  What does that tell us about the probability of
finding such individuals?

Re: Emacs rewrite in a maintainable language

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> It's all a pipe dream, no more, unless someone steps forward who
> really intends and is able to do the job. And that is even before we
> start talking about which alternative language should that be,
> something that in itself is not a trivial decision.

I agree, but the key here is that others said "No, there is no need or
advantage on doing so." Your response is of a very different kind.

> Also, please note that, due to a strange coincidence (or maybe
> something else), doing so will require the same talents you rightfully
> say are at premium.  What does that tell us about the probability of
> finding such individuals?

Yes, coordinating all the volunteers for the task will be no problem.
However, it is not absolutely necessary to have a global understanding
of the module that it is being translated to the new language. Depending
on the target language, a shallow knowledge can be sufficient. This
makes the requirements for the translator job to be much less stringent
than those for the maintainer of the same module.

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 11 Oct 2015 21:43:08 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It's all a pipe dream, no more, unless someone steps forward who
> > really intends and is able to do the job. And that is even before we
> > start talking about which alternative language should that be,
> > something that in itself is not a trivial decision.
> 
> I agree, but the key here is that others said "No, there is no need or
> advantage on doing so." Your response is of a very different kind.

Not really.  I just stopped short of rationalizing why something that
is difficult is not really required ;-)

> However, it is not absolutely necessary to have a global understanding
> of the module that it is being translated to the new language. Depending
> on the target language, a shallow knowledge can be sufficient.

Only where the current code is self-contained, i.e. does not make any
serious use of system APIs and is not heavily used by other parts of
Emacs.

If these conditions are false, then a fairly complete understanding is
in fact necessary, otherwise we will have gobs of subtle bugs on our
hands.

Also, depending on the target language, simple 1:1 translation might
make no sense, because it could yield insufficient performance.  In
those cases, the translator will have to reverse-engineer the (largely
undocumented) requirements from the code, and then reimplement them
using more efficient primitives of the target language.  That also
requires a fairly good understanding of what the code does, and why.

And these issues are just the tip of the iceberg.

Re: Emacs rewrite in a maintainable language

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> However, it is not absolutely necessary to have a global understanding
>> of the module that it is being translated to the new language. Depending
>> on the target language, a shallow knowledge can be sufficient.
>
> Only where the current code is self-contained, i.e. does not make any
> serious use of system APIs and is not heavily used by other parts of
> Emacs.
>
> If these conditions are false, then a fairly complete understanding is
> in fact necessary, otherwise we will have gobs of subtle bugs on our
> hands.

If the target language is ABI-compatible with C and hence has no
problems calling system APIs, there is no problem with translating a
function to that language and make it available to the
still-untranslated functions. The calling functions would remain
untouched until they get their turn to be translated.

> Also, depending on the target language, simple 1:1 translation might
> make no sense, because it could yield insufficient performance.  In
> those cases, the translator will have to reverse-engineer the (largely
> undocumented) requirements from the code, and then reimplement them
> using more efficient primitives of the target language.  That also
> requires a fairly good understanding of what the code does, and why.

Same as above for the performance and 1:1 translation.

> And these issues are just the tip of the iceberg.

I have experience doing this type of work (from C++) but the target
language was custom-made for making the process as easy as possible
while being advanced enough to warrant the effort. It was (is) a huge
success.

It would be not so easy with one of the stablished languages (Scheme,
for instance, has all the problems you mentioned and some more.)

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 11 Oct 2015 22:13:53 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> However, it is not absolutely necessary to have a global understanding
> >> of the module that it is being translated to the new language. Depending
> >> on the target language, a shallow knowledge can be sufficient.
> >
> > Only where the current code is self-contained, i.e. does not make any
> > serious use of system APIs and is not heavily used by other parts of
> > Emacs.
> >
> > If these conditions are false, then a fairly complete understanding is
> > in fact necessary, otherwise we will have gobs of subtle bugs on our
> > hands.
> 
> If the target language is ABI-compatible with C and hence has no
> problems calling system APIs, there is no problem with translating a
> function to that language and make it available to the
> still-untranslated functions. The calling functions would remain
> untouched until they get their turn to be translated.

I don't know what you mean by "ABI compatible".  If you mean C++, then
it's a single language (and not the best choice, IMO, as it is not
much better for Emacs than C, given its decline and problems of
finding people who know it).  As soon as you start talking about Java
or Python (fully compatible with C, btw), the issues I mentioned start
piling: you cannot transform the code without understanding what it
does.

> > Also, depending on the target language, simple 1:1 translation might
> > make no sense, because it could yield insufficient performance.  In
> > those cases, the translator will have to reverse-engineer the (largely
> > undocumented) requirements from the code, and then reimplement them
> > using more efficient primitives of the target language.  That also
> > requires a fairly good understanding of what the code does, and why.
> 
> Same as above for the performance and 1:1 translation.

Same as above for counter-arguments.

> > And these issues are just the tip of the iceberg.
> 
> I have experience doing this type of work (from C++) but the target
> language was custom-made for making the process as easy as possible
> while being advanced enough to warrant the effort. It was (is) a huge
> success.

I also have experience (converting a large project from Fortran to C).

> It would be not so easy with one of the stablished languages (Scheme,
> for instance, has all the problems you mentioned and some more.)

I think any language which is "better" (i.e. is known by more
programmers) will have the same problems.

Re: Emacs rewrite in a maintainable language

[Paul Eggert]

Eli Zaretskii wrote:
> I think any language which is "better" (i.e. is known by more
> programmers) will have the same problems.

For October 2014 the TIOBE Programming Community index gives the following 
ratings for the ten most popular programming languages:

   19.543 Java
   16.190 C
    5.749 C++
    4.825 C#
    4.512 Python
    2.561 PHP
    2.462 VB.NET
    2.292 JavaScript
    2.247 Perl
    1.825 Ruby

Since 2002, the #1 language in this somewhat-volatile index has always been 
either C or Java. So, by this measure at least, the only plausible "better" 
language would be Java, and if we went back to October 2014 even Java would be 
off the table.

The TIOBE Index isn't a perfect measure, of course. And I am not advocating that 
we rewrite the Emacs core in Java. My point is merely that C is not an unpopular 
language. If we want something unpopular, we could try Cobol at 0.909, or 
Haskell at 0.290, or Clojure at 0.190....

My source:

http://www.tiobe.com/index.php/tiobe_index (retrieved 2015-10-11)

Re: Emacs rewrite in a maintainable language

[Steinar Bang]

>>>>> Paul Eggert <eggert@cs.ucla.edu>:

> The TIOBE Index isn't a perfect measure, of course. And I am not
> advocating that we rewrite the Emacs core in Java.

(FWIW it's already been done once, but the project doesn't seem to have
been touched for a decade... http://jemacs.sourceforge.net/ (unless it
has moved to a new home...? I found a 4 year old fork on github, but it hasn't
preserved the old jemacs history, so it is hard to see what it
adds... also the last commit was 4 years ago https://github.com/shadgregory/JEmacs ))

Re: Emacs rewrite in a maintainable language

[Marcin Borkowski]

On 2015-10-12, at 10:12, Steinar Bang <sb@dod.no> wrote:

>>>>>> Paul Eggert <eggert@cs.ucla.edu>:
>
>> The TIOBE Index isn't a perfect measure, of course. And I am not
>> advocating that we rewrite the Emacs core in Java.
>
> (FWIW it's already been done once, but the project doesn't seem to have
> been touched for a decade... http://jemacs.sourceforge.net/ (unless it
> has moved to a new home...? I found a 4 year old fork on github, but it hasn't
> preserved the old jemacs history, so it is hard to see what it
> adds... also the last commit was 4 years ago https://github.com/shadgregory/JEmacs ))

For the record: there was once a project to rewrite TeX in Java (it was
before the mankind learned that Java was not the solution to all the
humanity's problems).  It even got some funding (as in, real money).
The project is now considered a failure: while they /did/ produce
a working, 99.9% compatible TeX replacement, it was an order of
magnitude slower (or so), and the main goal - to have a TeX replacement
written in a more modular fashion, so that it would be easier to
e.g. experiment with various line/page-breaking algorithms and
extensions to TeX - was not achieved: nobody actually performed those
experiments.  (There aren't even many traces on the internet; while you
can read about this "New Typesetting System" in a few places, the actual
code is lost.  That is a pity, imho, since I hear that Java is much
faster nowadays than 10+ years ago, so it might work now.)

Another project, LuaTeX, took off and can be now considered very
successful.  This project, however, has different goals (instead of
replacing TeX with something different, they embedded Lua in TeX, and
then opend up a possibility to reimplement many parts of TeX in Lua.  In
effect, it seems that the goal of NTS - to be able to easily experiment
with TeX internals - was achieved, though in a completely different
way).  Also, it is run by a small group of dedicated individuals, and my
guess is that one of the reasons of its success is Hans Hagen and his
team.  One, they are exceptionally smart people.  Two, they /think/ and
/design/ before coding, not the other way round.  (Not to say that Emacs
devs don't do it, but the majority of the software world apparently got
this wrong.)  Last but not least, Hans' company is based around ConTeXt
and LuaTeX, so he has a /really/ strong motivation to have this system
working smoothly: he /actually relies on it for his income/.

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Marcin Borkowski <mbork@mbork.pl> writes:

> For the record: there was once a project to rewrite TeX in Java (it
> was before the mankind learned that Java was not the solution to all
> the humanity's problems).

NTS -- New Typesetting System.

> It even got some funding (as in, real money).  The project is now
> considered a failure: while they /did/ produce a working, 99.9%
> compatible TeX replacement, it was an order of magnitude slower (or
> so), and the main goal - to have a TeX replacement written in a more
> modular fashion, so that it would be easier to e.g. experiment with
> various line/page-breaking algorithms and extensions to TeX - was not
> achieved: nobody actually performed those experiments.  (There aren't
> even many traces on the internet; while you can read about this "New
> Typesetting System" in a few places, the actual code is lost.  That is
> a pity, imho, since I hear that Java is much faster nowadays than 10+
> years ago, so it might work now.)

Well, later there was exTeX <URL:http://www.extex.org>.  The front page
of the web site and the FAQ are still accessible, but most parts of the
site are dead these days.

> Another project, LuaTeX, took off and can be now considered very
> successful.  This project, however, has different goals (instead of
> replacing TeX with something different, they embedded Lua in TeX, and
> then opend up a possibility to reimplement many parts of TeX in Lua.
> In effect, it seems that the goal of NTS - to be able to easily
> experiment with TeX internals - was achieved, though in a completely
> different way).

It did not as much clone TeX as slowly turn it into a cyborg, putting
hooks into the existing works.

> Also, it is run by a small group of dedicated individuals, and my
> guess is that one of the reasons of its success is Hans Hagen and his
> team.  One, they are exceptionally smart people.  Two, they /think/
> and /design/ before coding, not the other way round.

Uh, are we talking about the same project here?  The whole point of
LuaTeX was _not_ to design anything but rather take the existing design
and code and "open it up" by adding hooks into its code and data
structures.  This most certainly _is_ a "code foremost, to heck with
design" project.

> (Not to say that Emacs devs don't do it, but the majority of the
> software world apparently got this wrong.)  Last but not least, Hans'
> company is based around ConTeXt and LuaTeX, so he has a /really/
> strong motivation to have this system working smoothly: he /actually
> relies on it for his income/.

He has a really strong motivation to have _something_ working _today_.
And he started with an existing complex format employing the unchanged
TeX engine, hooking into TeX where the limited facilities provided by
the base TeX engine were making the format rather awkward and
inefficient even though basically "everything needed" would have been
reasonably close under the hood in the underlying Pascal code.

I once translated a video game I wrote for Z80 processors at 4MHz
(obviously in assembly language) to C.  That was the opportunity to
rewrite everything in a modular manner, make the various "monsters" have
movement code in an object-oriented manner and so on.

In the end, this ended up more or less as a literal translation of the
logic of the assembly code, partly using even register assignments as
variables.  The game was coded very well and readable in spite of
6-character limits for identifiers (uppercase and digits only) and
almost all comments consisting in accumulative cycle counts for the sake
of the game's timing.  To "rewrite" in more modern idioms would have
required basically designing the code from scratch, and the existing
code was a very good "local minimum" with nothing in the line of
"incremental improvement" being in line of sight.

The same problem pervades in a larger scale the original code base of
TeX.  You either end up with redesigning it from scratch, or with just
having a clone not inherently different, in particular not any more
modular or maintainable.

LuaTeX does not even _try_ to touch the TeX "table of equivalence" and
its related dynamic scoping mechanisms.  From Lua, you just get to touch
the current group-local settings of TeX variables but do not get to
either read, simulate, or touch the TeX grouping itself.

There still isn't any way to embed Lua code in TeX or vice versa: any
Lua you want to execute from TeX is first _tokenized_ according to TeX's
input conversion rules and then _printed_ back into characters read into
Lua.  If Lua was a language picky about differences in whitespace (like
Python), this would be totally untenable.  The reverse is similar: Lua
can "print" into TeX's idea of the current input file, and TeX will at
some later time when Lua has ceded control pick that input up and
interpret it as part of whatever it was doing when it called Lua, of
course first turning the characters back into TeX tokens according to
whatever catcode regime it might be in.

That's not an integrated application as much as it is two different
systems conversing over byte-stream pipes with synchronous but
non-symmetric flow control.

They get results as opposed to the several failed projects preceding
them.  But the main reason is that they don't bother with a coherent
design rather than getting results.  In the course of their work, they
sometimes throw out some component of TeX altogether and replace its
Pascal code with something coded entirely in Lua, in order to make it
more accessible to their personal modifications.  So there _is_ some
"rewrite TeX in Lua" work involved, but it's not really part of a design
process but done ad-hoc on an as-needed basis.

So the overall message LuaTeX sends to people wanting to redesign
systems is rather bleak.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> I think any language which is "better" (i.e. is known by more
> programmers) will have the same problems.

I'll like to insist on the fact that Elisp (a paradigm of niche
language) has plenty of contributors, while just a few people work on
the part of Emacs which is written in that epitome of a succesful
language which is C.

Obviously, on this case, "popular" doesn't imply "better".

It would be interesting to know how many new hackers the C core got on,
say, the last 10 years, who made contributions beyond correcting the
occassional bug which is evident on a gdb backtrace.

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 12 Oct 2015 14:23:35 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think any language which is "better" (i.e. is known by more
> > programmers) will have the same problems.
> 
> I'll like to insist on the fact that Elisp (a paradigm of niche
> language) has plenty of contributors, while just a few people work on
> the part of Emacs which is written in that epitome of a succesful
> language which is C.
> 
> Obviously, on this case, "popular" doesn't imply "better".

I might even agree, but how does this contradict what I wrote and you
cited?

> It would be interesting to know how many new hackers the C core got on,
> say, the last 10 years, who made contributions beyond correcting the
> occassional bug which is evident on a gdb backtrace.

About 20.

Re: Emacs rewrite in a maintainable language

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I don't think we should rewrite Emacs into anything other than C.
It would not be an improvement.

Some parts of the Emacs C code could use rewriting -- for instance,
the parts that deal with management of X windows.  But there is no need
to change to a different language.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs rewrite in a maintainable language

[Rustom Mody]

On Tue, Oct 13, 2015 at 1:30 AM, Richard Stallman <rms@gnu.org> wrote:
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> I don't think we should rewrite Emacs into anything other than C.
> It would not be an improvement.
>
> Some parts of the Emacs C code could use rewriting -- for instance,
> the parts that deal with management of X windows.  But there is no need
> to change to a different language.
>

When the rewrite bug is going round this is required reading
http://www.joelonsoftware.com/articles/fog0000000069.html

On a personal note (its about 1989 so my memory is foggy...):

The FSF tape carrying gcc etc had just arrived and a colleague was struggling to
compile it on a local C compiler
I wrote to RIchard something to the effect:

  How could someone intelligent enough to write emacs (in lisp) write
gcc (in C)?


Note 1: Since my memory of these near 3-decade old things are so foggy, I'm sure
Richard does not remember
Note 2: Wisdom being directly proportional to age, I dont have that question now

Re: Emacs rewrite in a maintainable language

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> It's all a pipe dream, no more, unless someone steps forward who
> really intends and is able to do the job.  And that is even before we
> start talking about which alternative language should that be,
> something that in itself is not a trivial decision.

Of course could I also tell what my favorite language is and why we
should rewrite Emacs in it.  But Eli is right - this probably will just
not happen now.

IMHO it would be better to concentrate on problems where we can be
productive.  Like answering Richards question in

  "Intro to Emacs Lisp Programming and lexical variables"

where he asked whether the manual should be updated for lexical scoping.

If we don't even have the manpower to answer such a simple question
after one month, we maybe should defer rewriting Emacs.  Yes, dreaming
is fun, but also the annoying tasks need to be done.


Michael.

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Óscar Fuentes <ofv@wanadoo.es> writes:

> In large part because that reason, there are key are parts of the C core
> that are only understood by one active hacker (being optimistic here). This
> is a huge liability for Emacs. I don't know how this fact fits the "C is
> working for us" stance.

Of all the languages I know, I can't think of one that gives us everything we
need: stability, performance, portability, a large potential contributor base
(people who know C can learn the macro system Emacs uses), debugging tools,
etc.

The C core needs experts, not so much in C, but at writing display engines and
algorithms important to Emacs. I find it hard to believe that the "Lisp in C"
macros we use on the C side to allow seamless integration are the stumbling
point for those who want to contribute. Am I wrong in this? Has this turned
people away from working on Emacs core?

C works for us because it does what it says on the tin. Emacs runs on a huge
number of systems, is efficient, and the project continues to thrive after
many decades.

I'm against changing "because the grass is greener". Give it a decade or so,
and I'd happily join in rewriting the core in Haskell. But when I think about
the difficulty of running on alternative systems (the dependencies alone might
kill us), or non-expert contributors introducing space leaks that only
manifest for specialized use cases... I'll take C any day. C diffs are clear
and concise as to their actual semantic content. This is also a reason Linus
has given for keeping the Linux kernel in C.

I'm not saying I love C. I spent a career working in C, then finally fled for
greener pastures. But show me an alternative that comes even remotely close,
and I'll happily consider it.

John

Re: Emacs rewrite in a maintainable language

[David Kastrup]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> In large part because that reason, there are key are parts of the C core
>> that are only understood by one active hacker (being optimistic here). This
>> is a huge liability for Emacs. I don't know how this fact fits the "C is
>> working for us" stance.
>
> Of all the languages I know, I can't think of one that gives us
> everything we need: stability, performance, portability, a large
> potential contributor base (people who know C can learn the macro
> system Emacs uses), debugging tools, etc.

Lua is an impressive little and powerful language.  But you would not
want to build a Lisp-based system around it since Lisp's fundamental
data structure is the list, and Lua's fundamental data structure is the
table.  For writing something akin to Emacs from scratch, I'd consider
it a very nice choice.  And its program syntax fits on a single DIN-A5
page in the Reference Manual.

Now the thing people don't like about languages in the Lisp family is
the program syntax.  Which is sort of missing the point because Lisp
does not even have one.  It has a reader syntax (which is more of a
lexer than a parser) with a focus on lists, and you directly enter the
parse tree.

Which is akin to assembly language programming at the syntactical rather
than the code level (well, "car" does stand for "contents of address
register"), making Lisp a fun little system for goofing around with the
compiler.  Which is not really all that important in the context of an
editor.

But with the current code base in place, anything leaving Lisp behind is
going to leave Emacs behind and thus is going to be a different project.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Replacing C with a language that helps people to write readable, safe,
> efficient and maintainable code without having to study the existing
> code base for months looks quite appealing to me. That replacement could
> be gradual (no "stopping the world while The Big Rewrite is underway.")
> If that language is not so popular as C, that's no big problem as long
> as it makes things less daunting for those willing to work on the Emacs
> core. As already said, there is no dramatic shortage of people who learn
> Elisp just for hacking Emacs.

I think slowly moving the Emacs C core to C++ is a good idea.  The two
languages are supposed to be largely compile-compatible, i.e. g++ can
understand what was meant for gcc.

I've been trying to get into understanding Emacs C core, but it's just
so hard with all those macros around. I think even replacing `#define`
with `const` would be a huge improvement, worthwhile of the switch.  The
second step would be to replace some macros with type checked template
functions. Finally, write proper classes for buffer/window/frame/font
etc. Nothing too complex like virtual functions, operator overloading or
hardcore template meta-programming that puts people off C++, just basic
encapsulation to see which code uses what and the type of data being
operated on.

I don't have sufficient C/C++ experience to initiate this change.  But
if someone were to lead, I'd help as much as I could.

If other people and the new maintainer were to support the C -> C++
move, the change could be very gradual, with a lot of functions not
changing at all. In an ideal world, the change would be initiated by
e.g. deciding to use C++11 and swapping `cc` for `g++` in the Makefiles,
without changing the code at all. Of course it's never as simple as
that.

    Oleh

Re: Emacs rewrite in a maintainable language

[Óscar Fuentes]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> I think slowly moving the Emacs C core to C++ is a good idea.  The two
> languages are supposed to be largely compile-compatible, i.e. g++ can
> understand what was meant for gcc.

Many times, while looking at the Emacs C sources, I thought "this would
be much simpler to understand and modify with C++." Furthermore, I would
volunteer to work on it.

However, there are two factors against it:

 * If the rewrite happens, using a better language than C++ is
   desirable. C++ is better than C, but it sucks too. Developing our own
   subset of Elisp that can be compiled to native code and can
   inter-operate with C looks more attractive.

 * Current maintainers are not likely to welcome the idea, for several
   legitimate reasons ("don't know C++", "I dislike it", etc). And
   Stallman would forbide it alright ("C++ in my Emacs? No way!" :-)
   Making the pill easier to swallow by restricting ourselves to the
   simpler C++ features greatly diminishes its effectiveness too.

See the case of GCC. It was too obvious that switching to C++ would
improve the code, but it was resisted for many years.

[snip]

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Oleh Krehel <ohwoeowho@gmail.com> writes:
>
>> I think slowly moving the Emacs C core to C++ is a good idea.  The two
>> languages are supposed to be largely compile-compatible, i.e. g++ can
>> understand what was meant for gcc.
>
> Many times, while looking at the Emacs C sources, I thought "this would
> be much simpler to understand and modify with C++." Furthermore, I would
> volunteer to work on it.
>
> However, there are two factors against it:
>
>  * If the rewrite happens, using a better language than C++ is
>    desirable. C++ is better than C, but it sucks too. Developing our own
>    subset of Elisp that can be compiled to native code and can
>    inter-operate with C looks more attractive.

C++ doesn't suck. If C is good, than a subset of C++ that is C with
constants, templates and classes, but without polymorphism or operator
overloading is a better C - also good.

>  * Current maintainers are not likely to welcome the idea, for several
>    legitimate reasons ("don't know C++", "I dislike it", etc). And
>    Stallman would forbide it alright ("C++ in my Emacs? No way!" :-)

I don't know if this was ever brought up.

>    Making the pill easier to swallow by restricting ourselves to the
>    simpler C++ features greatly diminishes its effectiveness too.

I disagree. The features would be the current C features, only with
better type checking and encapsulation.

    Oleh

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Oleh Krehel <ohwoeowho@gmail.com> writes:
>>
>>> I think slowly moving the Emacs C core to C++ is a good idea.  The two
>>> languages are supposed to be largely compile-compatible, i.e. g++ can
>>> understand what was meant for gcc.
>>
>> Many times, while looking at the Emacs C sources, I thought "this would
>> be much simpler to understand and modify with C++." Furthermore, I would
>> volunteer to work on it.
>>
>> However, there are two factors against it:
>>
>>  * If the rewrite happens, using a better language than C++ is
>>    desirable. C++ is better than C, but it sucks too. Developing our own
>>    subset of Elisp that can be compiled to native code and can
>>    inter-operate with C looks more attractive.
>
> C++ doesn't suck. If C is good, than a subset of C++ that is C with
> constants, templates and classes, but without polymorphism or operator
> overloading is a better C - also good.

No, because such a subset is neither documented independently nor
enforced by the toolchain.  You'll not find any tutorials written for it
nor standard documents describing it.  It's ad hoc.  The set of C macros
used for Elisp interfacing is ad-hoc as well, but the C language
features used for implementing them are a small, well-known set, and
knowing that, the Emacs sources tell the rest of the story if you cannot
get them from anywhere else.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Óscar Fuentes <ofv@wanadoo.es> writes:

> Many times, while looking at the Emacs C sources, I thought "this would be
> much simpler to understand and modify with C++." Furthermore, I would
> volunteer to work on it.

Please, not C++. I don't believe the result would be any easier to maintain
overall. The difficulties would just be shifted. Further, our candidate
contributor pool would shrink.

One of my open source projects, Ledger, is in C++. Despite having many active
users, the number of pull requests I've received for the C++ code can be
counted on both hands. Meanwhile, pull requests for ledger-mode continue to
come in, and I was able to find a full maintainer for that part of the code.

The thought of an Elisp->C compiler that could produce well-performing code is
very interesting. I starting working on an LLVM JIT for Emacs bytecode at one
point -- got as far as hello world -- and it was not terribly difficult. The
main reason I stopped is that it didn't help performance enough. The slowness
imposed by constantly looking up symbol names at global scope, due to dynamic
binding, dominated the profiling results.

A subset of Elisp that doesn't have dynamic binding might well produce code
that could be of use to us, and wouldn't need any of the macros we currently
use to communicate between the two languages.

John

Re: Emacs rewrite in a maintainable language

[Óscar Fuentes]

"John Wiegley" <johnw@newartisans.com> writes:

[snip]

> A subset of Elisp that doesn't have dynamic binding might well produce
> code that could be of use to us,

That's what I had in mind all the time. We could aim to the subset of
Elisp that can be translated to a fast enough target code, while
remaining superior to C in convenience.

Or a more radical approach: a Lispy language, easy to learn by those who
already know Elisp, but which can be translated all the way to something
as performant as the current C code, while being capable of interacting
with it. This way, whenever someone needs fast code, he would use this
other dialect instead of writing C, or the slow Elisp (or avoiding the
task altogether.)

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Óscar Fuentes <ofv@wanadoo.es> writes:

>> A subset of Elisp that doesn't have dynamic binding might well produce code
>> that could be of use to us,

> That's what I had in mind all the time. We could aim to the subset of Elisp
> that can be translated to a fast enough target code, while remaining
> superior to C in convenience.

> Or a more radical approach: a Lispy language, easy to learn by those who
> already know Elisp, but which can be translated all the way to something as
> performant as the current C code, while being capable of interacting with
> it. This way, whenever someone needs fast code, he would use this other
> dialect instead of writing C, or the slow Elisp (or avoiding the task
> altogether.)

If it can leverage what the contributor already knows about Elisp, I'm all for
it. I wonder what RMS' and Eli's reaction would be.

If they are amenable, would you be willing to prototype it? If we could see a
language description for the subset/alternate, and a sample compilation into C
for some of the functions we have in Emacs core (that is, reimplementing them
as a proof of concept), that would help sell it.

John

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: "John Wiegley" <johnw@newartisans.com>
> Date: Mon, 12 Oct 2015 12:39:32 -0700
> 
> >>>>> Óscar Fuentes <ofv@wanadoo.es> writes:
> 
> > Or a more radical approach: a Lispy language, easy to learn by those who
> > already know Elisp, but which can be translated all the way to something as
> > performant as the current C code, while being capable of interacting with
> > it. This way, whenever someone needs fast code, he would use this other
> > dialect instead of writing C, or the slow Elisp (or avoiding the task
> > altogether.)
> 
> If it can leverage what the contributor already knows about Elisp, I'm all for
> it. I wonder what RMS' and Eli's reaction would be.

I just gave it.  But I don't think it matters, as long as there's no
volunteers that work on this in earnest.  We are still dreaming.  I
withhold my real reaction until I actually see some language like
that.

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> Date: Mon, 12 Oct 2015 22:46:35 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: "John Wiegley" <johnw@newartisans.com>
> > Date: Mon, 12 Oct 2015 12:39:32 -0700
> > 
> > >>>>> Óscar Fuentes <ofv@wanadoo.es> writes:
> > 
> > > Or a more radical approach: a Lispy language, easy to learn by those who
> > > already know Elisp, but which can be translated all the way to something as
> > > performant as the current C code, while being capable of interacting with
> > > it. This way, whenever someone needs fast code, he would use this other
> > > dialect instead of writing C, or the slow Elisp (or avoiding the task
> > > altogether.)
> > 
> > If it can leverage what the contributor already knows about Elisp, I'm all for
> > it. I wonder what RMS' and Eli's reaction would be.
> 
> I just gave it.

On second thought, I don't think I understand the idea at all.  What
does it mean "a Lispy language, easy to learn"?  Is it a Lisp dialect,
or is it C with a set of Lisp-like macros preprocessed into C?  What
exactly are the C aspects that we are trying to save the programmer
from?  And which part(s) of the core do we expect to be able to
rewrite in this "Lispy" language?

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> On second thought, I don't think I understand the idea at all. What does it
> mean "a Lispy language, easy to learn"? Is it a Lisp dialect, or is it C
> with a set of Lisp-like macros preprocessed into C? What exactly are the C
> aspects that we are trying to save the programmer from? And which part(s) of
> the core do we expect to be able to rewrite in this "Lispy" language?

Picture what we currently write in C, but a Lisp syntax, and all the macros we
currently use removed. So, the essence of our C, written like it was Lisp.

If that Lisp can get close enough to Emacs Lisp, so that knowing one means
knowing the other, we've just made it easier for anyone to write what we now
have to write in C.

I can imagine that complex things, like type declarations, would be done with
anti-quoted blocks, or by direct support for inclusion of header files.

John

Re: Emacs rewrite in a maintainable language

[Marcin Borkowski]

On 2015-10-12, at 22:11, John Wiegley <johnw@newartisans.com> wrote:

> If that Lisp can get close enough to Emacs Lisp, so that knowing one means
> knowing the other, we've just made it easier for anyone to write what we now
> have to write in C.

By no means am I an expert on this, but isn't "close enough, yet not
identical" a recipe for nasty bugs?

> I can imagine that complex things, like type declarations, would be done with
> anti-quoted blocks, or by direct support for inclusion of header files.
>
> John

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University

Re: Emacs rewrite in a maintainable language

[Óscar Fuentes]

John Wiegley <johnw@newartisans.com> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>> On second thought, I don't think I understand the idea at all. What does it
>> mean "a Lispy language, easy to learn"? Is it a Lisp dialect, or is it C
>> with a set of Lisp-like macros preprocessed into C? What exactly are the C
>> aspects that we are trying to save the programmer from? And which part(s) of
>> the core do we expect to be able to rewrite in this "Lispy" language?
>
> Picture what we currently write in C, but a Lisp syntax, and all the macros we
> currently use removed. So, the essence of our C, written like it was Lisp.
>
> If that Lisp can get close enough to Emacs Lisp, so that knowing one means
> knowing the other, we've just made it easier for anyone to write what we now
> have to write in C.

Yes.

> I can imagine that complex things, like type declarations, would be done with
> anti-quoted blocks, or by direct support for inclusion of header files.

The project I made for extending/replacing a C++ code base uses Clang
for parsing the headers, extracting the interesting bits and emitting
the information required by the Lispy language to access or call the C++
part. As depending on Clang is not allowed, another tool is required,
but there are some, or the info can be generated once and put into a
file and then maintained as the C source code changes, altough the
problem with this approach are the platform-dependent bits.

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: John Wiegley <johnw@newartisans.com>
> Date: Mon, 12 Oct 2015 13:11:30 -0700
> 
> > On second thought, I don't think I understand the idea at all. What does it
> > mean "a Lispy language, easy to learn"? Is it a Lisp dialect, or is it C
> > with a set of Lisp-like macros preprocessed into C? What exactly are the C
> > aspects that we are trying to save the programmer from? And which part(s) of
> > the core do we expect to be able to rewrite in this "Lispy" language?
> 
> Picture what we currently write in C, but a Lisp syntax, and all the macros we
> currently use removed. So, the essence of our C, written like it was Lisp.

I did picture that.  That's why I'm asking.

Can we please see a couple of examples, as a kind of concept
demonstration, just to feel the taste of this?  For example, how would
you write make_frame in this language?  (This would be an example of
creating a Lisp object which is backed up by a large C struct that has
members not visible from Lisp.)  Will we have some kind of 'defstruct'
in this "subset"?  How will we distinguish Lisp objects from C
objects?

Also, please try to answer the other questions I asked above.  I think
they are important for us to understand what exactly is being
proposed and what do we try to accomplish.

> If that Lisp can get close enough to Emacs Lisp, so that knowing one means
> knowing the other, we've just made it easier for anyone to write what we now
> have to write in C.

Sorry, I came to the conclusion that I don't really understand what
this means in practice.  So I'm asking for examples.

If this means we will have C code written in Lisp syntax, then are you
saying the syntax of C is the main difficulty for understanding the C
core code?  E.g., let's imagine that we rewrote the display engine
this "Lispy" language -- do we really believe it will be easier to
understand and maintain?

Or what about regex.c -- are we going to "lispify" that?

> I can imagine that complex things, like type declarations, would be done with
> anti-quoted blocks, or by direct support for inclusion of header files.

So we will have a mixture of C and Lisp, or C blocks within Lisp code?
How will that help readability and maintainability?

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> > On second thought, I don't think I understand the idea at all. What does it
>> > mean "a Lispy language, easy to learn"? Is it a Lisp dialect, or is it C
>> > with a set of Lisp-like macros preprocessed into C? What exactly are the C
>> > aspects that we are trying to save the programmer from? And which part(s) of
>> > the core do we expect to be able to rewrite in this "Lispy" language?

> Can we please see a couple of examples, as a kind of concept demonstration,
> just to feel the taste of this? For example, how would you write make_frame
> in this language? (This would be an example of creating a Lisp object which
> is backed up by a large C struct that has members not visible from Lisp.)
> Will we have some kind of 'defstruct' in this "subset"? How will we
> distinguish Lisp objects from C objects?
>
> Also, please try to answer the other questions I asked above. I think they
> are important for us to understand what exactly is being proposed and what
> do we try to accomplish.

I'd like to heartily second these requests from Eli, as I wonder the same
things.

> If this means we will have C code written in Lisp syntax, then are you
> saying the syntax of C is the main difficulty for understanding the C core
> code? E.g., let's imagine that we rewrote the display engine this "Lispy"
> language -- do we really believe it will be easier to understand and
> maintain?

Not the syntax of C itself, but the macros and tests that happen within that C
code. The Lispy language *should* be more transparent as to the intent of the
function. But as you say, we need to see real examples of this before anyone
should believe that.

> Or what about regex.c -- are we going to "lispify" that?

I don't know how much C we should get rid of, actually. At the very least,
we'd need C to bootstrap the Lispy compiler, for portability.

> So we will have a mixture of C and Lisp, or C blocks within Lisp code? How
> will that help readability and maintainability?

I think the idea was just to shrink the surface area of the C code, and get
closer and closer to "everything in Lisp". But as you say, we'd need to see
some experiments first.

The C we have works for us, so this idea is pretty low priority. As you wrote
in another thread, there are several, higher priority items that would win us
more contributors faster (documentation, etc). Even changing from C to a Lisp
would not be a silver bullet: the display code would still be a hard problem
to solve.

John

RE: Emacs rewrite in a maintainable language

[Drew Adams]

> The C we have works for us, so this idea is pretty low priority. As you
> wrote in another thread, there are several, higher priority items that
> would win us more contributors faster (documentation, etc). Even changing
> from C to a Lisp would not be a silver bullet: the display code would
> still be a hard problem to solve.

And yet the thread goes on.  Lower priority, little light.

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Drew Adams <drew.adams@oracle.com> writes:

>> The C we have works for us, so this idea is pretty low priority. As you
>> wrote in another thread, there are several, higher priority items that
>> would win us more contributors faster (documentation, etc). Even changing
>> from C to a Lisp would not be a silver bullet: the display code would still
>> be a hard problem to solve.

> And yet the thread goes on.  Lower priority, little light.

I'm OK with this discussion, since we have developers who are curious about
the feasibility and benefits such a change might bring. Yes, it's low
priority, but that doesn't make it uninteresting. :)

John

RE: Emacs rewrite in a maintainable language

[Drew Adams]

> > This way, whenever someone needs fast code, he would use this other
> > dialect instead of writing C, or the slow Elisp (or avoiding the
> > task altogether.)
> 
> If it can leverage what the contributor already knows about Elisp,
> I'm all for it. I wonder what RMS' and Eli's reaction would be.

I'm not for it.  I guess it depends on just how "whenever someone
needs fast code" is interpreted, in practice.

I do not wish to see Emacs core developers start opting for
something like this instead of using Lisp, with the excuse that
they want their given code to be "fast".

That would be an unwelcome blow to Emacs, the extendable,
customizable editor.  Emacs is in Lisp for the sake of its users,
above all.  The only difference today between implementation
language and user language is the C code.  I think of that
(relatively small) bit of code as essentially a necessary evil.

Emacs _is_ a Lisp environment.  This has been the case since
before the existence of Lisp machines.  It is no accident that
the development of Emacs (in various flavors) went hand in hand
with Lisp development.

> compilation into C for some of the functions we have in Emacs
> core

This is backwards from the direction we have been moving with
Stefan and Eli, which is toward moving core stuff from C to
Lisp when possible.

> (that is, reimplementing them as a proof of concept),

OTOH, if it's _only_ to test POC, then I suppose it's hard to
object.  But I would not want to see compiling core Lisp code
to the proposed language be taken seriously.  That would be a
step backward, IMO.

Leave such code in Lisp, please.  And move more core code to
Lisp, when that is feasible.  (As Eli has noted, most of the
C code cannot feasibly be moved to Lisp.)

> that would help sell it.

To whom are you trying to sell it?  What's the point?  Emacs
is too slow?  C is too hard to maintain?

To be clear, if it is a question of using such a language
_only_ for the equivalent of what C is _necessarily_ used for,
it's hard to object.  But if it is just to have "fast" core
code, where that code could be in Lisp instead, then I object,
as one user.

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Drew Adams <drew.adams@oracle.com> writes:

> I'm not for it. I guess it depends on just how "whenever someone needs fast
> code" is interpreted, in practice.
> 
> I do not wish to see Emacs core developers start opting for something like
> this instead of using Lisp, with the excuse that they want their given code
> to be "fast".

If the gap between C (or a hypothetical Lisp-like), and Emacs Lisp, could be
reduced to zero, I'd absolutely want everything to be in Emacs Lisp. That has
been my dream for Emacs since I can remember, as I'm prefer playing on the
Lisp side much more than the C side.

I once tried to make Emacs Lisp my "primary language". The speed just wasn't
there, not to mention a few other things (control over memory layout, direct
access to file handles, integration with C libraries, etc). There are several
things we'd need to turn Emacs Lisp into a general purpose, well-performing
language, sufficient to replace all our C code.

>> compilation into C for some of the functions we have in Emacs core

> This is backwards from the direction we have been moving with Stefan and
> Eli, which is toward moving core stuff from C to Lisp when possible.

I want the C->Lisp direction. I was talking about proving any new candidate
that might replace C, to show it provides a comparable implementation.

>> (that is, reimplementing them as a proof of concept),

> OTOH, if it's _only_ to test POC, then I suppose it's hard to object. But I
> would not want to see compiling core Lisp code to the proposed language be
> taken seriously. That would be a step backward, IMO.

Don't worry, not suggesting that. :)

> Leave such code in Lisp, please. And move more core code to Lisp, when that
> is feasible. (As Eli has noted, most of the C code cannot feasibly be moved
> to Lisp.)

100% agreed.

> To whom are you trying to sell it? What's the point? Emacs is too slow? C is
> too hard to maintain?

The objective of the proposal is to not lose speed -- not necessarily to be
any faster -- and to gain contributors by offering a language closer to (if
not actually) Emacs Lisp.

> To be clear, if it is a question of using such a language _only_ for the
> equivalent of what C is _necessarily_ used for, it's hard to object.

Yes, only for that. As much as is practical, code should be in Emacs Lisp. I
fully back that as a strategy for the future, since C is not the dire
necessity it once was, per Richard's cute poem.

John

Re: Emacs rewrite in a maintainable language

[immerrr again]

On Tue, Oct 13, 2015 at 7:18 AM, John Wiegley <johnw@newartisans.com> wrote:
>
> If the gap between C (or a hypothetical Lisp-like), and Emacs Lisp, could be
> reduced to zero, I'd absolutely want everything to be in Emacs Lisp. That has
> been my dream for Emacs since I can remember, as I'm prefer playing on the
> Lisp side much more than the C side.
>
> I once tried to make Emacs Lisp my "primary language". The speed just wasn't
> there, not to mention a few other things (control over memory layout, direct
> access to file handles, integration with C libraries, etc). There are several
> things we'd need to turn Emacs Lisp into a general purpose, well-performing
> language, sufficient to replace all our C code.
>

A wild idea would be to try rewriting the Emacs Lisp interpreter in
RPython [1]. It is a "framework for writing dynamic language
interpreters" and is used for implementing pypy [2] and pixie [3]
(which is a Lisp dialect). It accepts interpreter code in a subset of
Python, which is still quite high-level, that can be translated to C
using GCC as a backend. It comes with interpreter-related goodies,
like FFI, several implementations of GC, code flow analyzers and an
implementation of a tracing JIT. I would guess it would help with
performance a bit.

1. https://rpython.readthedocs.org/
2. http://pypy.org/
3. https://github.com/pixie-lang/pixie

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> Date: Tue, 13 Oct 2015 09:00:15 +0300
> From: immerrr again <immerrr@gmail.com>
> 
> A wild idea would be to try rewriting the Emacs Lisp interpreter in
> RPython [1]. It is a "framework for writing dynamic language
> interpreters" and is used for implementing pypy [2] and pixie [3]
> (which is a Lisp dialect). It accepts interpreter code in a subset of
> Python, which is still quite high-level, that can be translated to C
> using GCC as a backend. It comes with interpreter-related goodies,
> like FFI, several implementations of GC, code flow analyzers and an
> implementation of a tracing JIT. I would guess it would help with
> performance a bit.

The interpreter code proper is about 16K lines of C out of the grand
total of more than 320K, i.e. just 5%.  The rest is implementation of
primitives and Lisp objects + support code, and the display engine.

Re: Emacs rewrite in a maintainable language

[Óscar Fuentes]

"John Wiegley" <johnw@newartisans.com> writes:

> If it can leverage what the contributor already knows about Elisp, I'm all for
> it. I wonder what RMS' and Eli's reaction would be.
>
> If they are amenable, would you be willing to prototype it?

Maybe the next time I get the flu :-)

[snip]

Re: Emacs rewrite in a maintainable language

[David Kastrup]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>>> A subset of Elisp that doesn't have dynamic binding might well produce code
>>> that could be of use to us,
>
>> That's what I had in mind all the time. We could aim to the subset of Elisp
>> that can be translated to a fast enough target code, while remaining
>> superior to C in convenience.
>
>> Or a more radical approach: a Lispy language, easy to learn by those
>> who already know Elisp, but which can be translated all the way to
>> something as performant as the current C code, while being capable of
>> interacting with it. This way, whenever someone needs fast code, he
>> would use this other dialect instead of writing C, or the slow Elisp
>> (or avoiding the task altogether.)
>
> If it can leverage what the contributor already knows about Elisp, I'm
> all for it. I wonder what RMS' and Eli's reaction would be.

Shrug.  There is the GUILE branch.  In my opinion its largest obstacle
towards acceptance is the failure of GUILE strings to offer a comparable
feature set to Emacs multibyte strings: nothing outside of the Unicode
range, no byte transparent reproduction.

This needs an overhaul of GUILE's string handling (which delegates parts
of it to helper libraries, some of those being under control of GNU,
some not).

Another tricky thing is that '() and #f (boolean false) are different
entities which Lisp both represents as nil.  That will make things
somewhat more awkward when writing code for Elisp in Scheme.  But the
string thing is a performance killer (the branch currently converts back
and forth between "Emacs string" and "GUILE string" or possibly byte
vectors all the time if I understand correctly).

> If they are amenable, would you be willing to prototype it? If we
> could see a language description for the subset/alternate, and a
> sample compilation into C for some of the functions we have in Emacs
> core (that is, reimplementing them as a proof of concept), that would
> help sell it.

GUILE's byte compiler is supposed to do a better job than Elisp.  It's
still not native, but the layering of GUILE's compiler might make native
code generation more realistic than with Elisp.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 13 Oct 2015 12:01:50 +0200
> 
> > If it can leverage what the contributor already knows about Elisp, I'm
> > all for it. I wonder what RMS' and Eli's reaction would be.
> 
> There is the GUILE branch.

First, Guile's Scheme is not Emacs Lisp, there are significant
differences.

Second, Guile itself is written in C, so what exactly is gained here?

Third, AFAIR the Guile branch doesn't replace all of Emacs's C core.

Fourth, that branch is far from ready for prime time (as you know and
point out).

> GUILE's byte compiler is supposed to do a better job than Elisp.

But for now it has known problems with ELisp (some tests fail).  Also,
at least Guile's own byte code (the *.go files) are not
architecture-independent, so building a Guile Emacs will need a long
compilation on the target machine.  Not a catastrophe, but hardly a
nice thing.

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 13 Oct 2015 12:01:50 +0200
>> 
>> > If it can leverage what the contributor already knows about Elisp,
>> > I'm all for it. I wonder what RMS' and Eli's reaction would be.
>> 
>> There is the GUILE branch.
>
> First, Guile's Scheme is not Emacs Lisp, there are significant
> differences.

Uh, the topic was rewriting Emacs in a Lisp-like language performing
better than Elisp.  I was pointing out that treating this topic as if
something like that would be totally new was leaving out a significant
bit of history.

> Second, Guile itself is written in C, so what exactly is gained here?

As I wrote and you removed in order to pretend I didn't, the GUILE
compiler stack is posited a whole lot better for native code generation
than Elisp and does a better job even on its own byte code machine.

> Third, AFAIR the Guile branch doesn't replace all of Emacs's C core.

Most definitely not.

> Fourth, that branch is far from ready for prime time (as you know and
> point out).

So it makes more sense to discuss branches that have not even started?
Without even considering the lessons learnt so far on the GUILE branch?
Is this discussion intended to stay at the level of pipe dreams?

>> GUILE's byte compiler is supposed to do a better job than Elisp.
>
> But for now it has known problems with ELisp (some tests fail).  Also,
> at least Guile's own byte code (the *.go files) are not
> architecture-independent, so building a Guile Emacs will need a long
> compilation on the target machine.  Not a catastrophe, but hardly a
> nice thing.

Yes, certainly much worse than our fantasy Lisp-like language compiling
into machine code that has no problem interfacing with half of Emacs and
replacing the rest.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Chris Patti]

On Tue, Oct 13, 2015 at 11:20 AM, David Kastrup <dak@gnu.org> wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: David Kastrup <dak@gnu.org>
>>> Date: Tue, 13 Oct 2015 12:01:50 +0200
>>>
>>> > If it can leverage what the contributor already knows about Elisp,
>>> > I'm all for it. I wonder what RMS' and Eli's reaction would be.
>>>
>>> There is the GUILE branch.
>>
>> First, Guile's Scheme is not Emacs Lisp, there are significant
>> differences.
>
> Uh, the topic was rewriting Emacs in a Lisp-like language performing
> better than Elisp.  I was pointing out that treating this topic as if
> something like that would be totally new was leaving out a significant
> bit of history.

This would not be GNU emacs.  GNU emacs is what it is because it's
written in and runs elisp.  There is SO MUCH elisp out there that does
so much for so many people, this just doesn't make any sense.

Go create a new thing if you like and call it foomacs, but IMO emacs
performs just great on modern machines, so why fix what isn't broken?

Just my $.02 of course.

-Chris
-- 
Christopher Patti - Geek At Large | GTalk: cpatti@gmail.com | AIM:
chrisfeohpatti | P: (260) 54PATTI
"Technology challenges art, art inspires technology." - John Lasseter, Pixar

Re: Emacs rewrite in a maintainable language

[Jay Belanger]

Chris Patti <cpatti@gmail.com> writes:

> This would not be GNU emacs.  GNU emacs is what it is because it's
> written in and runs elisp.  There is SO MUCH elisp out there that does
> so much for so many people, this just doesn't make any sense.

GuileEmacs is designed to run elisp.

Guile-Emacs

[Ludovic Courtès]

Eli Zaretskii <eliz@gnu.org> skribis:

> First, Guile's Scheme is not Emacs Lisp, there are significant
> differences.

Guile’s compiler has an Emacs Lisp front-end.

> Second, Guile itself is written in C, so what exactly is gained here?

More than half is in Scheme (see <https://www.openhub.net/p/guile>.)

The C part is a subset of the run-time support library, libguile.  The
compiler itself, among other things, is written in Scheme.

> Third, AFAIR the Guile branch doesn't replace all of Emacs's C core.

Right.  Part of Emacs’ C core becomes redundant with libguile, though,
so it should be possible to remove that part.

> Fourth, that branch is far from ready for prime time (as you know and
> point out).

This is true.  That said, of all the proposals discussed here, it seems
to be by far the one that is the closest to being ready.  :-)

<http://emacswiki.org/emacs/GuileEmacs> provides a good summary.

>> GUILE's byte compiler is supposed to do a better job than Elisp.
>
> But for now it has known problems with ELisp (some tests fail).  Also,
> at least Guile's own byte code (the *.go files) are not
> architecture-independent, so building a Guile Emacs will need a long
> compilation on the target machine.  Not a catastrophe, but hardly a
> nice thing.

There’s a trade-off here: those files can be mmap’d directly, which
reduces resource usage.

It would be wonderful if Emacs hackers could look at Guile-Emacs, talk
to Robin Templeton, identify problems, and help out.  After all the
thought and sweat Robin has put into this project, I’m confident that
it’s not (no longer!) an unrealistic project.

More importantly, the benefits are way more tangible than those of a C++
rewrite of the core; see <http://emacswiki.org/emacs/GuileEmacs>.

Let’s just do it!  :-)

Ludo’.

Re: Emacs rewrite in a maintainable language

[Ludovic Courtès]

David Kastrup <dak@gnu.org> skribis:

> Shrug.  There is the GUILE branch.  In my opinion its largest obstacle
> towards acceptance is the failure of GUILE strings to offer a comparable
> feature set to Emacs multibyte strings: nothing outside of the Unicode
> range, no byte transparent reproduction.

OTOH, I imagine Emacs could build its own abstraction, if needed, on top
of Guile bytevectors.

> This needs an overhaul of GUILE's string handling (which delegates parts
> of it to helper libraries, some of those being under control of GNU,
> some not).

String handling is partly delegated to GNU libunistring.  There’s no
other library involved.

> GUILE's byte compiler is supposed to do a better job than Elisp.  It's
> still not native, but the layering of GUILE's compiler might make native
> code generation more realistic than with Elisp.

Agreed.

Ludo’.

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: ludo@gnu.org (Ludovic Courtès)
> Date: Fri, 16 Oct 2015 14:03:39 +0200
> 
> David Kastrup <dak@gnu.org> skribis:
> 
> > This needs an overhaul of GUILE's string handling (which delegates parts
> > of it to helper libraries, some of those being under control of GNU,
> > some not).
> 
> String handling is partly delegated to GNU libunistring.  There’s no
> other library involved.

Yes, but libunistring is not a good basis for Emacs's i18n features,
because it relies on the underlying C library for most of its
features.  That doesn't allow writing programs that behave exactly the
same on any platform, something that Emacs must support.  It also is
too locale-dependent, which is a Bad Thing on platforms which don't
have the glibc new-locale feature (and is IMO inelegant in use even on
glibc platforms).

Another minor point is that Guile still doesn't allow to control the
EOL format of text I/O independently of binary I/O mode, again
something that Emacs must (and does) have.

Re: Emacs rewrite in a maintainable language

[Wolfgang Jenkner]

On Fri, Oct 16 2015, Eli Zaretskii wrote:

> Yes, but libunistring is not a good basis for Emacs's i18n features,
> because it relies on the underlying C library for most of its
> features.  That doesn't allow writing programs that behave exactly the
> same on any platform, something that Emacs must support.  

IIUC, the behaviour of the recently added `string-collate-lessp' and
`string-collate-equalp' already depends on libc (and the locale data).

(Lest this be interpreted as criticism: IMHO, depending on POSIX
facilities is a good thing, even for emacs.)

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Cc: ludo@gnu.org (Ludovic Courtès),  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 16:55:37 +0200
> 
> On Fri, Oct 16 2015, Eli Zaretskii wrote:
> 
> > Yes, but libunistring is not a good basis for Emacs's i18n features,
> > because it relies on the underlying C library for most of its
> > features.  That doesn't allow writing programs that behave exactly the
> > same on any platform, something that Emacs must support.  
> 
> IIUC, the behaviour of the recently added `string-collate-lessp' and
> `string-collate-equalp' already depends on libc (and the locale data).

Yes, but these two functions are not called by anything else, they
exist so Lisp applications could use locale-specific collation data.

I wasn't talking about that, I was talking about stuff like decoding a
Latin-1 file into the internal UTF-8 based representation of
characters, or vice versa.  Emacs does that all the time.  What
libunistring does is call the libc's mbstowc, or some such.  That
requires libc to support all the encodings Emacs needs to support,
which I think doesn't happen even with glibc, let alone other system
libraries.

> (Lest this be interpreted as criticism: IMHO, depending on POSIX
> facilities is a good thing, even for emacs.)

Emacs should first and foremost present the same behavior and
capabilities on all platforms.  Different platforms implement Posix to
different degrees, so relying on Posix capabilities provided by the
platform has its limits, and interferes with the above goal of
presenting the same behavior.

Basing i18n on locale-specific behavior is especially wrong.

Re: Emacs rewrite in a maintainable language

[Ludovic Courtès]

Eli Zaretskii <eliz@gnu.org> skribis:

>> From: ludo@gnu.org (Ludovic Courtès)
>> Date: Fri, 16 Oct 2015 14:03:39 +0200
>> 
>> David Kastrup <dak@gnu.org> skribis:
>> 
>> > This needs an overhaul of GUILE's string handling (which delegates parts
>> > of it to helper libraries, some of those being under control of GNU,
>> > some not).
>> 
>> String handling is partly delegated to GNU libunistring.  There’s no
>> other library involved.
>
> Yes, but libunistring is not a good basis for Emacs's i18n features,
> because it relies on the underlying C library for most of its
> features.  That doesn't allow writing programs that behave exactly the
> same on any platform, something that Emacs must support.  It also is
> too locale-dependent, which is a Bad Thing on platforms which don't
> have the glibc new-locale feature (and is IMO inelegant in use even on
> glibc platforms).

Well yes, libguile relies on things like iconv, either via libunistring
or directly.  Locale-dependent code is limited to the (ice-9 i18n)
module, which is about locales.

I don’t think it’s the end of the world, though.  Emacs could pretty
much keep using its own text I/O routines if those of Guile don’t fit
the bill.

> Another minor point is that Guile still doesn't allow to control the
> EOL format of text I/O independently of binary I/O mode, again
> something that Emacs must (and does) have.

Guile’s I/O ports could be extended to handle that, but again, even if
they don’t, Emacs could do its own thing.

I doubt there are major blockers.  Surely things will need to be tweaked
here and there, but only hacking can tell.  :-)

Ludo’.

Re: Emacs rewrite in a maintainable language

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Eli Zaretskii <eliz@gnu.org> skribis:

> I don’t think it’s the end of the world, though.  Emacs could pretty
> much keep using its own text I/O routines if those of Guile don’t fit
> the bill.
>
>> Another minor point is that Guile still doesn't allow to control the
>> EOL format of text I/O independently of binary I/O mode, again
>> something that Emacs must (and does) have.
>
> Guile’s I/O ports could be extended to handle that, but again, even if
> they don’t, Emacs could do its own thing.
>
> I doubt there are major blockers.  Surely things will need to be tweaked
> here and there, but only hacking can tell.  :-)

Everything's easy once one decides somebody else is the right person to
be doing it.  At any rate, the way I see it, the previous work done on
GUILE-Emacs have been doing a lot with bringing GUILE's strong suits
into Emacs.  But the work won't be done without bringing Emacs' strong
suits into GUILE.  Emacs has decades of character handling internally
and externally under its belt and lots of experience gained the hard
way.  It will require quite a bit of dedicated work to bring GUILE up to
speed.  And since there aren't decades of time available for repeating
all the respective experience, it will have to take a large page out of
how Emacs does it.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> David Kastrup <dak@gnu.org> skribis:
>
>> Shrug.  There is the GUILE branch.  In my opinion its largest obstacle
>> towards acceptance is the failure of GUILE strings to offer a comparable
>> feature set to Emacs multibyte strings: nothing outside of the Unicode
>> range, no byte transparent reproduction.
>
> OTOH, I imagine Emacs could build its own abstraction, if needed, on top
> of Guile bytevectors.

Sure, but then Emacs strings cannot be manipulated using GUILE string
functions.  Same with Emacs buffers (which otherwise could be something
like random-access string ports).

>> This needs an overhaul of GUILE's string handling (which delegates
>> parts of it to helper libraries, some of those being under control of
>> GNU, some not).
>
> String handling is partly delegated to GNU libunistring.  There’s no
> other library involved.

What about libiconv?  It would appear that parts of it are in the GUILE
source tree (admittedly, under GNU control).

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Fri, 16 Oct 2015 16:29:00 +0200
> Cc: emacs-devel@gnu.org
> 
> >> This needs an overhaul of GUILE's string handling (which delegates
> >> parts of it to helper libraries, some of those being under control of
> >> GNU, some not).
> >
> > String handling is partly delegated to GNU libunistring.  There’s no
> > other library involved.
> 
> What about libiconv?  It would appear that parts of it are in the GUILE
> source tree (admittedly, under GNU control).

libunistring calls libiconv.

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Fri, 16 Oct 2015 16:29:00 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> >> This needs an overhaul of GUILE's string handling (which delegates
>> >> parts of it to helper libraries, some of those being under control of
>> >> GNU, some not).
>> >
>> > String handling is partly delegated to GNU libunistring.  There’s no
>> > other library involved.
>> 
>> What about libiconv?  It would appear that parts of it are in the GUILE
>> source tree (admittedly, under GNU control).
>
> libunistring calls libiconv.

Optionally.  And GUILE includes parts of libiconv and calls them itself.

I'm just pointing out that this is quite nontrivial to bring up to Emacs
standards in string processing because so many dependencies are
involved, most of them having a rather simplistic view of invalid or
overlong byte sequences and characters outside of the Unicode range.

They don't allow as many character codes as Emacs needs to represent its
various inputs.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: ludo@gnu.org,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 17:28:26 +0200
> 
> >> What about libiconv?  It would appear that parts of it are in the GUILE
> >> source tree (admittedly, under GNU control).
> >
> > libunistring calls libiconv.
> 
> Optionally.  And GUILE includes parts of libiconv and calls them itself.

Yes, that's true.

> I'm just pointing out that this is quite nontrivial to bring up to Emacs
> standards in string processing because so many dependencies are
> involved, most of them having a rather simplistic view of invalid or
> overlong byte sequences and characters outside of the Unicode range.
> 
> They don't allow as many character codes as Emacs needs to represent its
> various inputs.

I agree.

Re: Emacs rewrite in a maintainable language

[Ludovic Courtès]

Eli Zaretskii <eliz@gnu.org> skribis:

>> From: David Kastrup <dak@gnu.org>
>> Date: Fri, 16 Oct 2015 16:29:00 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> >> This needs an overhaul of GUILE's string handling (which delegates
>> >> parts of it to helper libraries, some of those being under control of
>> >> GNU, some not).
>> >
>> > String handling is partly delegated to GNU libunistring.  There’s no
>> > other library involved.
>> 
>> What about libiconv?  It would appear that parts of it are in the GUILE
>> source tree (admittedly, under GNU control).
>
> libunistring calls libiconv.

‘iconv’ is in libc on most systems though, the exceptions being Windows
and old proprietary Unices, AFAIK.

Ludo’.

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: ludo@gnu.org (Ludovic Courtès)
> Cc: David Kastrup <dak@gnu.org>,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 17:31:32 +0200
> 
> > libunistring calls libiconv.
> 
> ‘iconv’ is in libc on most systems though, the exceptions being Windows
> and old proprietary Unices, AFAIK.

Which leaves what, apart of GNU/Linux?

In any case, Emacs can never be satisfied with the current Guile
infrastructure for i18n.  There are too many shortcomings, some of
them were mentioned here.  Yes, Guile can be fixed to be better in
that area, but no one is working on that, AFAIK, and what's more
important, lead Guile developers don't even agree Guile should move in
that direction.  (This especially puzzles me: to have a good example
before you and not follow it?  Emacs learned what it has now the hard
way, have paid in blood, sweat and tears for that knowledge, and still
Guile developers think they "know better"?  Present parties excluded,
of course.)

Re: Emacs rewrite in a maintainable language

[Przemysław Wojnowski]

W dniu 16.10.2015 o 18:11, Eli Zaretskii pisze:
> In any case, Emacs can never be satisfied with the current Guile
> infrastructure for i18n.  There are too many shortcomings, some of
> them were mentioned here.  Yes, Guile can be fixed to be better in
> that area, but no one is working on that, AFAIK, and what's more
> important, lead Guile developers don't even agree Guile should move in
> that direction.  (This especially puzzles me: to have a good example
> before you and not follow it?  Emacs learned what it has now the hard
> way, have paid in blood, sweat and tears for that knowledge, and still
> Guile developers think they "know better"?  Present parties excluded,
> of course.)

Is i18n the only obstacle?
IOW if someone would improve Guile's i18n infrastructure to match that of
Emacs, would it be included into Emacs?

To be clear, I'm not asking to make a pressure on you, but only in context of
motivation. If someone would know that i18n is the only obstacle, maybe this
person(s) would find motivation to do the job. Without that it is discouraging.

This way or another, a clear decision would be helpful here. "Yes, we want
if...", "No, we don't want, because...". At least it would clarify everything
and close the topic.

Thanks,
Przemysław

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Przemysław Wojnowski <esperanto@cumego.com> writes:

> W dniu 16.10.2015 o 18:11, Eli Zaretskii pisze:
>> In any case, Emacs can never be satisfied with the current Guile
>> infrastructure for i18n.  There are too many shortcomings, some of
>> them were mentioned here.  Yes, Guile can be fixed to be better in
>> that area, but no one is working on that, AFAIK, and what's more
>> important, lead Guile developers don't even agree Guile should move in
>> that direction.  (This especially puzzles me: to have a good example
>> before you and not follow it?  Emacs learned what it has now the hard
>> way, have paid in blood, sweat and tears for that knowledge, and still
>> Guile developers think they "know better"?  Present parties excluded,
>> of course.)
>
> Is i18n the only obstacle?
> IOW if someone would improve Guile's i18n infrastructure to match that of
> Emacs, would it be included into Emacs?

Don't underestimate the effort, but I think that is the largest obstacle
currently.

> To be clear, I'm not asking to make a pressure on you, but only in
> context of motivation. If someone would know that i18n is the only
> obstacle,

It certainly isn't, but it's the elephant in the room.  Emacs is very
good at dealing with lots of textual stuff without getting confused and
in many cases being capable of perfect reproduction.  That's all rather
important for making GUILE a universal extension language for multiple
applications.  So it's not just for Emacs' sake.

> maybe this person(s) would find motivation to do the job. Without that
> it is discouraging.

Well, the other non-easy problem is that of nil, '(), and #f.  The rest
seems comparably straightforward.

So there would be still work remaining, and sort of a fight for getting
there.  But currently there is not even a fight: Guile-Emacs is not a
serious contender.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> Cc: dak@gnu.org, emacs-devel@gnu.org
> From: Przemysław Wojnowski <esperanto@cumego.com>
> Date: Fri, 16 Oct 2015 21:34:19 +0200
> 
> W dniu 16.10.2015 o 18:11, Eli Zaretskii pisze:
> > In any case, Emacs can never be satisfied with the current Guile
> > infrastructure for i18n.  There are too many shortcomings, some of
> > them were mentioned here.  Yes, Guile can be fixed to be better in
> > that area, but no one is working on that, AFAIK, and what's more
> > important, lead Guile developers don't even agree Guile should move in
> > that direction.  (This especially puzzles me: to have a good example
> > before you and not follow it?  Emacs learned what it has now the hard
> > way, have paid in blood, sweat and tears for that knowledge, and still
> > Guile developers think they "know better"?  Present parties excluded,
> > of course.)
> 
> Is i18n the only obstacle?

The only major one I know of.  There are quite a few minor ones, which
will also have to be handled.  In general, Guile is much less portable
to non-GNU/Linux systems than Emacs.

> IOW if someone would improve Guile's i18n infrastructure to match that of
> Emacs, would it be included into Emacs?

Including Guile in Emacs is a project by itself, regardless of
features missing from Guile itself.

> To be clear, I'm not asking to make a pressure on you, but only in context of
> motivation. If someone would know that i18n is the only obstacle, maybe this
> person(s) would find motivation to do the job. Without that it is discouraging.
> 
> This way or another, a clear decision would be helpful here. "Yes, we want
> if...", "No, we don't want, because...". At least it would clarify everything
> and close the topic.

Theoretically, there's no problem: Guile is _the_ GNU extension
language, so accepting it in Emacs is a no-brainer.

Practically, I see no reason to make any decisions until someone comes
up and publishes a repository where all the necessary work has been
done.  There's a person working on that, on and off, but that job is
far from complete.

Re: Emacs rewrite in a maintainable language

[Ludovic Courtès]

Eli Zaretskii <eliz@gnu.org> skribis:

> In any case, Emacs can never be satisfied with the current Guile
> infrastructure for i18n.  There are too many shortcomings, some of
> them were mentioned here.  Yes, Guile can be fixed to be better in
> that area, but no one is working on that, AFAIK, and what's more
> important, lead Guile developers don't even agree Guile should move in
> that direction.  (This especially puzzles me: to have a good example
> before you and not follow it?  Emacs learned what it has now the hard
> way, have paid in blood, sweat and tears for that knowledge, and still
> Guile developers think they "know better"?  Present parties excluded,
> of course.)

My point is: Emacs can keep doing its own thing in that area.

(And I would guess that neither C++, nor Lua, nor anything else would
provide an i18n infrastructure that would satisfy Emacs out-of-the-box.)

Ludo’.

Re: Emacs rewrite in a maintainable language

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Eli Zaretskii <eliz@gnu.org> skribis:
>
>> In any case, Emacs can never be satisfied with the current Guile
>> infrastructure for i18n.  There are too many shortcomings, some of
>> them were mentioned here.  Yes, Guile can be fixed to be better in
>> that area, but no one is working on that, AFAIK, and what's more
>> important, lead Guile developers don't even agree Guile should move in
>> that direction.  (This especially puzzles me: to have a good example
>> before you and not follow it?  Emacs learned what it has now the hard
>> way, have paid in blood, sweat and tears for that knowledge, and still
>> Guile developers think they "know better"?  Present parties excluded,
>> of course.)
>
> My point is: Emacs can keep doing its own thing in that area.
>
> (And I would guess that neither C++, nor Lua, nor anything else would
> provide an i18n infrastructure that would satisfy Emacs out-of-the-box.)

Either of those at least get out of Emacs' hair, as would have GUILE 1.
But GUILE 2 has a competing compulsory multibyte character concept.  "So
just switch everything to Latin-1 amounting to binary" you probably
say.  If it were that easy, there would be a GUILE 2 version of LilyPond
by now.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: ludo@gnu.org (Ludovic Courtès)
> Cc: emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 22:51:26 +0200
> 
> Eli Zaretskii <eliz@gnu.org> skribis:
> 
> > In any case, Emacs can never be satisfied with the current Guile
> > infrastructure for i18n.  There are too many shortcomings, some of
> > them were mentioned here.  Yes, Guile can be fixed to be better in
> > that area, but no one is working on that, AFAIK, and what's more
> > important, lead Guile developers don't even agree Guile should move in
> > that direction.  (This especially puzzles me: to have a good example
> > before you and not follow it?  Emacs learned what it has now the hard
> > way, have paid in blood, sweat and tears for that knowledge, and still
> > Guile developers think they "know better"?  Present parties excluded,
> > of course.)
> 
> My point is: Emacs can keep doing its own thing in that area.

Of course.  But that takes away a serious chunk of arguments in favor
of Guile-based Emacs, for 2 reasons: (a) there will have to be a
non-trivial translation layer between the two, and (b) a very large
part of Emacs's C core will have to be left intact, instead of
removing it because Guile already does that.

> (And I would guess that neither C++, nor Lua, nor anything else would
> provide an i18n infrastructure that would satisfy Emacs out-of-the-box.)

Of course.  But the Guile alternative is being brought up as being
much better than those others.  If we leave the strings and i18n
alone, a large part of that argument goes away.

Re: Emacs rewrite in a maintainable language

[Ludovic Courtès]

Eli Zaretskii <eliz@gnu.org> skribis:

>> From: ludo@gnu.org (Ludovic Courtès)
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 16 Oct 2015 22:51:26 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> skribis:
>> 
>> > In any case, Emacs can never be satisfied with the current Guile
>> > infrastructure for i18n.  There are too many shortcomings, some of
>> > them were mentioned here.  Yes, Guile can be fixed to be better in
>> > that area, but no one is working on that, AFAIK, and what's more
>> > important, lead Guile developers don't even agree Guile should move in
>> > that direction.  (This especially puzzles me: to have a good example
>> > before you and not follow it?  Emacs learned what it has now the hard
>> > way, have paid in blood, sweat and tears for that knowledge, and still
>> > Guile developers think they "know better"?  Present parties excluded,
>> > of course.)
>> 
>> My point is: Emacs can keep doing its own thing in that area.
>
> Of course.  But that takes away a serious chunk of arguments in favor
> of Guile-based Emacs, for 2 reasons: (a) there will have to be a
> non-trivial translation layer between the two, and (b) a very large
> part of Emacs's C core will have to be left intact, instead of
> removing it because Guile already does that.

AFAIK nobody claims that Guile is the right choice due to its i18n
support.

The main claims are: the compiler, VM (which would no longer be part of
Emacs), FFI, libraries, etc.

That chunk isn’t taken away.

>> (And I would guess that neither C++, nor Lua, nor anything else would
>> provide an i18n infrastructure that would satisfy Emacs out-of-the-box.)
>
> Of course.  But the Guile alternative is being brought up as being
> much better than those others.  If we leave the strings and i18n
> alone, a large part of that argument goes away.

Guile strings are fine, thank you.  I’ve used a bunch of
language/environments and honestly, I’m definitely not ashamed of what
Guile provides, contrary to what David and you seem to imply.

I fail to follow the reasoning anyway, but I don’t have much to add.

Thanks,
Ludo’.

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: ludo@gnu.org (Ludovic Courtès)
> Cc: emacs-devel@gnu.org
> Date: Sat, 17 Oct 2015 11:44:25 +0200
> 
> >> My point is: Emacs can keep doing its own thing in that area.
> >
> > Of course.  But that takes away a serious chunk of arguments in favor
> > of Guile-based Emacs, for 2 reasons: (a) there will have to be a
> > non-trivial translation layer between the two, and (b) a very large
> > part of Emacs's C core will have to be left intact, instead of
> > removing it because Guile already does that.
> 
> AFAIK nobody claims that Guile is the right choice due to its i18n
> support.
> 
> The main claims are: the compiler, VM (which would no longer be part of
> Emacs), FFI, libraries, etc.
> 
> That chunk isn’t taken away.

I agree.  But those features are additions to what Emacs already has.
They are advantages for the future.  This thread talks about a
"rewrite", i.e. about replacing the existing C code with something
that will be more maintainable and will attract new developers easier
than the current C core.  With that perspective, advantages for the
future are less relevant; what is more relevant is how much of the
current C core could be replaced by Guile, and how much will have to
be rewritten in Guile Scheme or left in C.

> Guile strings are fine, thank you.  I’ve used a bunch of
> language/environments and honestly, I’m definitely not ashamed of what
> Guile provides, contrary to what David and you seem to imply.

I didn't mean to imply that Guile development should be ashamed.  But
please consider this: the first version of Emacs i18n was released
with Emacs 20.1.  People who designed and implemented it were not
stupid, and the result was nothing to be ashamed of, either: it solved
a very large part of the problem it was intended to solve.  And yet
since then the v20.1 implementation on various levels was
significantly changed no less than 2 times, in response to real-life
experience and user feedback.  If we speak today about some
non-trivial conclusions drawn from that, we are speaking armed with
that experience.

Guile's i18n is today where Emacs was at v20.1, with the (important)
difference that its internal representation is based on Unicode and
UTF-8.  The lessons of Emacs development since then till today are yet
to be learned and incorporated into Guile.  I really think Guile
developers should seriously study the Emacs experience, if Guile is
supposed to support multi-lingual environments.

For example, IMO Guile relies too much on the underlying libc, in
particular on the locale-dependent services libc provides.  A
multi-lingual text-processing environment such as Emacs cannot depend
on locales, because each locale supports only one language.  Emacs
includes support for multiple languages in the same buffer/file that
simply cannot be easily implemented using locale-dependent features.
By contrast, Guile is quite good in supporting a single language, but
not as good when several different languages/scripts are to be mixed
in the same string.  E.g., consider a simple problem of counting
characters in such a string -- the libc functions like mbslen are not
guaranteed to support this outside of the current language/locale.

Re: Emacs rewrite in a maintainable language

[Ludovic Courtès]

Eli Zaretskii <eliz@gnu.org> skribis:

> Guile's i18n is today where Emacs was at v20.1, with the (important)
> difference that its internal representation is based on Unicode and
> UTF-8.

This is incorrect.

> The lessons of Emacs development since then till today are yet to be
> learned and incorporated into Guile.

Probably many of them, but maybe not all.  After all, Guile is not a
text editor, so the requirements may be different.

Ludo’.

Re: Emacs rewrite in a maintainable language

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Eli Zaretskii <eliz@gnu.org> skribis:
>
>> Guile's i18n is today where Emacs was at v20.1, with the (important)
>> difference that its internal representation is based on Unicode and
>> UTF-8.
>
> This is incorrect.
>
>> The lessons of Emacs development since then till today are yet to be
>> learned and incorporated into Guile.
>
> Probably many of them, but maybe not all.  After all, Guile is not a
> text editor, so the requirements may be different.

I quote from the GUILE project page:

    What is Guile? What can it do for you?

    Guile is the GNU Ubiquitous Intelligent Language for Extensions, the
    official extension language for the GNU operating system.

    Guile is a library designed to help programmers create flexible
    applications. Using Guile in an application allows the application's
    functionality to be extended by users or other programmers with
    plug-ins, modules, or scripts. Guile provides what might be
    described as "practical software freedom," making it possible for
    users to customize an application to meet their needs without
    digging into the application's internals.

I don't see a qualifier "unless you were planning to write a text
editor".  GUILE even has "Extensions" in its name.  As well as
"Ubiquitous".  So how do you reckon that it should not be used for
extending text editors?  And how do you plead at the same time that it
should be used for extending Emacs?

It's not just for the sake of GuileEmacs that GUILE needs to work on the
cost of passing even mere UTF-8 into GUILE and back out.  GUILE cannot
even pass its own strings through its own string ports without
converting back and forth.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

David Kastrup <dak@gnu.org> writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Eli Zaretskii <eliz@gnu.org> skribis:
>>
>>> Guile's i18n is today where Emacs was at v20.1, with the (important)
>>> difference that its internal representation is based on Unicode and
>>> UTF-8.
>>
>> This is incorrect.
>>
>>> The lessons of Emacs development since then till today are yet to be
>>> learned and incorporated into Guile.
>>
>> Probably many of them, but maybe not all.  After all, Guile is not a
>> text editor, so the requirements may be different.
>
> I quote from the GUILE project page:
>
>     What is Guile? What can it do for you?
>
>     Guile is the GNU Ubiquitous Intelligent Language for Extensions, the
>     official extension language for the GNU operating system.
>
>     Guile is a library designed to help programmers create flexible
>     applications. Using Guile in an application allows the application's
>     functionality to be extended by users or other programmers with
>     plug-ins, modules, or scripts. Guile provides what might be
>     described as "practical software freedom," making it possible for
>     users to customize an application to meet their needs without
>     digging into the application's internals.
>
> I don't see a qualifier "unless you were planning to write a text
> editor".  GUILE even has "Extensions" in its name.  As well as
> "Ubiquitous".  So how do you reckon that it should not be used for
> extending text editors?  And how do you plead at the same time that it
> should be used for extending Emacs?
>
> It's not just for the sake of GuileEmacs that GUILE needs to work on the
> cost of passing even mere UTF-8 into GUILE and back out.  GUILE cannot
> even pass its own strings through its own string ports without
> converting back and forth.

The state of the world of software would be even more horrible than it
is now if every program tried to accommodate for every possible invalid
input with something other than gracefully aborting.  (Many abort
ungracefully, sadly.)

Guile's string library is for working with strings (surprise!), not
arbitrary byte vectors.

For arbitrary byte vectors, Guile has bytevectors (surprise!!).

If one implemented a text editor from scratch in Guile and wanted to
allow editing files with no proper encoding, one would use an
abstraction on top of bytevectors.

I'm pretty sure that nearly every programming language with a sane
string library does more or less the same thing.  This is not a problem
of Guile; it's a fundamental problem of the requirement to allow working
with byte vectors that "ideally are but are not necessarily" strings.
Offering a data type encapsulating that idea that falls between byte
vectors and strings is a very special case and it should not be
surprising that a general-purpose programming language doesn't provide
such a data type by default.


For some fun that can result from functions with ill-defined domains,
see the shell-quasiquote thread.  Yay for code injection!

Taylan

Re: Emacs rewrite in a maintainable language

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> David Kastrup <dak@gnu.org> writes:
>
>> I quote from the GUILE project page:
>>
>>     What is Guile? What can it do for you?
>>
>>     Guile is the GNU Ubiquitous Intelligent Language for Extensions, the
>>     official extension language for the GNU operating system.
>>
>>     Guile is a library designed to help programmers create flexible
>>     applications. Using Guile in an application allows the application's
>>     functionality to be extended by users or other programmers with
>>     plug-ins, modules, or scripts. Guile provides what might be
>>     described as "practical software freedom," making it possible for
>>     users to customize an application to meet their needs without
>>     digging into the application's internals.
>>
>> I don't see a qualifier "unless you were planning to write a text
>> editor".  GUILE even has "Extensions" in its name.  As well as
>> "Ubiquitous".  So how do you reckon that it should not be used for
>> extending text editors?  And how do you plead at the same time that it
>> should be used for extending Emacs?
>>
>> It's not just for the sake of GuileEmacs that GUILE needs to work on the
>> cost of passing even mere UTF-8 into GUILE and back out.  GUILE cannot
>> even pass its own strings through its own string ports without
>> converting back and forth.
>
> The state of the world of software would be even more horrible than it
> is now if every program tried to accommodate for every possible
> invalid input with something other than gracefully aborting.  (Many
> abort ungracefully, sadly.)

It is not the job of an extension language to dictate application
behavior.

> Guile's string library is for working with strings (surprise!), not
> arbitrary byte vectors.

It is not the job of an extension language to dictate the concept of
"character".  Of course, not everything can be supported out of the box
and there are limits to what can be supported due to technical reasons.
However, the stance of the GUILE developers is to stop people using
GUILE from doing things they consider not their problem.

This stance, possibly partly due to only a single developer working in
isolation on the "unstable" branch and everybody else confined to
changes in the "stable" branch, is going to end a roadblock for projects
like GuileEmacs.

> For arbitrary byte vectors, Guile has bytevectors (surprise!!).
>
> If one implemented a text editor from scratch in Guile and wanted to
> allow editing files with no proper encoding, one would use an
> abstraction on top of bytevectors.

It is not the job of an extension language to dictate what constitutes
"proper encoding".  It is the job of the application.

> I'm pretty sure that nearly every programming language with a sane
> string library does more or less the same thing.

Even Emacs has unibyte strings.  But it does not refuse using string
operators on them.  GUILE bytevectors don't offer substrings and other
basic operations.

> This is not a problem of Guile; it's a fundamental problem of the
> requirement to allow working with byte vectors that "ideally are but
> are not necessarily" strings.  Offering a data type encapsulating that
> idea that falls between byte vectors and strings is a very special
> case and it should not be surprising that a general-purpose
> programming language doesn't provide such a data type by default.

GUILE is first and foremost an extension language.  That is how it
advertises itself on its web page.  That it supports the general-purpose
programming language Scheme is a boon and implementation choice.

If GUILE developers insist on GUILE being foremost a general-purpose
programming language rather than an extension language, they should
notify the GNU project and change their web page.  There is no point in
GNU promoting GUILE as an extension language if the GUILE developers are
no longer on board with that target.

> For some fun that can result from functions with ill-defined domains,
> see the shell-quasiquote thread.  Yay for code injection!

It's more like "yay for culture difference".  You prefer functions being
useful for fewer cases while touting that as an advantage.  That is a
pretty alien concept to established Emacs developers, Emacs being the
proverbial glue and shoe string across dozens of different platforms
right from the beginning of GNU where POSIX would have been luxury.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

David Kastrup <dak@gnu.org> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> I quote from the GUILE project page:
>>>
>>>     What is Guile? What can it do for you?
>>>
>>>     Guile is the GNU Ubiquitous Intelligent Language for Extensions, the
>>>     official extension language for the GNU operating system.
>>>
>>>     Guile is a library designed to help programmers create flexible
>>>     applications. Using Guile in an application allows the application's
>>>     functionality to be extended by users or other programmers with
>>>     plug-ins, modules, or scripts. Guile provides what might be
>>>     described as "practical software freedom," making it possible for
>>>     users to customize an application to meet their needs without
>>>     digging into the application's internals.
>>>
>>> I don't see a qualifier "unless you were planning to write a text
>>> editor".  GUILE even has "Extensions" in its name.  As well as
>>> "Ubiquitous".  So how do you reckon that it should not be used for
>>> extending text editors?  And how do you plead at the same time that it
>>> should be used for extending Emacs?
>>>
>>> It's not just for the sake of GuileEmacs that GUILE needs to work on the
>>> cost of passing even mere UTF-8 into GUILE and back out.  GUILE cannot
>>> even pass its own strings through its own string ports without
>>> converting back and forth.
>>
>> The state of the world of software would be even more horrible than it
>> is now if every program tried to accommodate for every possible
>> invalid input with something other than gracefully aborting.  (Many
>> abort ungracefully, sadly.)
>
> It is not the job of an extension language to dictate application
> behavior.

It is the job of programming tools to aid programmers in writing good
applications.

>> Guile's string library is for working with strings (surprise!), not
>> arbitrary byte vectors.
>
> It is not the job of an extension language to dictate the concept of
> "character".  Of course, not everything can be supported out of the box
> and there are limits to what can be supported due to technical reasons.
> However, the stance of the GUILE developers is to stop people using
> GUILE from doing things they consider not their problem.

What is a character and what is a string is understood very well, and
Guile provides clean abstractions for these, as any sane general purpose
language should.

While I can't speak for the Guile developers, I can assure you that
their stance is to provide the best possible abstractions a general
purpose language should provide.  During the transition from Guile 1.x
to 2.x, that sadly meant breaking some backwards compatibility, because
some abstractions in Guile 1 were not good enough.

> This stance, possibly partly due to only a single developer working in
> isolation on the "unstable" branch and everybody else confined to
> changes in the "stable" branch, is going to end a roadblock for projects
> like GuileEmacs.

You seem to be making wrong assumptions about Guile's development model.
I don't think the fact that the upcoming 2.2 has been mostly a one
person project has much to do with the incompatibilities between 1.x and
2.x.

GuileEmacs will most likely benefit significantly from further versions
in the 2.x series.  I seem to recall it hitting limitations in 2.0 that
were then put in prospect to be fixed in future versions.

>> For arbitrary byte vectors, Guile has bytevectors (surprise!!).
>>
>> If one implemented a text editor from scratch in Guile and wanted to
>> allow editing files with no proper encoding, one would use an
>> abstraction on top of bytevectors.
>
> It is not the job of an extension language to dictate what constitutes
> "proper encoding".  It is the job of the application.

Yes, it *is* the job of a string library to offer abstractions for
well-defined encodings of strings.

>> I'm pretty sure that nearly every programming language with a sane
>> string library does more or less the same thing.
>
> Even Emacs has unibyte strings.  But it does not refuse using string
> operators on them.  GUILE bytevectors don't offer substrings and other
> basic operations.

Bytevectors aren't strings.  I did not miss "substring" even while
implementing e.g. <https://github.com/TaylanUB/scheme-bytestructures>.
You can use bytevectors for a variety of things; if you want to use them
for a "maybe badly encoded string" data type, you'll need to build such
an abstraction.

>> This is not a problem of Guile; it's a fundamental problem of the
>> requirement to allow working with byte vectors that "ideally are but
>> are not necessarily" strings.  Offering a data type encapsulating that
>> idea that falls between byte vectors and strings is a very special
>> case and it should not be surprising that a general-purpose
>> programming language doesn't provide such a data type by default.
>
> GUILE is first and foremost an extension language.  That is how it
> advertises itself on its web page.  That it supports the general-purpose
> programming language Scheme is a boon and implementation choice.
>
> If GUILE developers insist on GUILE being foremost a general-purpose
> programming language rather than an extension language, they should
> notify the GNU project and change their web page.  There is no point in
> GNU promoting GUILE as an extension language if the GUILE developers are
> no longer on board with that target.

You seem to be making a strange distinction between a general-purpose
and an extension language and implying that an "extension language"
should fill exactly the use-cases of Emacs for some reason...

I would rather say that being an extension language means you can
integrate your own data types and subroutines easily with the language,
and the fact that GuileEmacs trivially just uses Elisp's existing string
data type and subroutines shows us that Guile fulfills that perfectly
well.

>> For some fun that can result from functions with ill-defined domains,
>> see the shell-quasiquote thread.  Yay for code injection!
>
> It's more like "yay for culture difference".  You prefer functions being
> useful for fewer cases while touting that as an advantage.

If with "being useful for fewer cases" you really mean "not being
vulnerable to code injection by clearly defining the domain of the
function" then yes, I very much tout that as an advantage.

> That is a pretty alien concept to established Emacs developers, Emacs
> being the proverbial glue and shoe string across dozens of different
> platforms right from the beginning of GNU where POSIX would have been
> luxury.

Maybe some established Emacs developers should get with the times then.


I'm not interested in continuing this discussion because it's just as
unproductive as the last time.  GuileEmacs has come *very* far, and most
arguments I hear against it seem based in apocalyptic FUD, and in
particular spitefulness due to past negative experiences with the Guile
1.x/2.x transition, which has really nothing to do with GuileEmacs, a
Guile 2.x project.  That is not to say it's going to be fairies and
pixies.  There's significant work to be done on it and I hope I will
find some time to contribute to it in the not too distant future.

Taylan

Re: Emacs rewrite in a maintainable language

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> David Kastrup <dak@gnu.org> writes:
>
>> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>>
>>> David Kastrup <dak@gnu.org> writes:
>>
>> It is not the job of an extension language to dictate application
>> behavior.
>
> It is the job of programming tools to aid programmers in writing good
> applications.

As long as you call Emacs a bad application because of its support of a
large number of encodings and situations beyond Unicode and utf-8,
support that was driven by user requirements and has been a long series
of improvements, you are not on a good road to sell GuileEmacs to either
Emacs or GUILE developers.

>> It is not the job of an extension language to dictate the concept of
>> "character".  Of course, not everything can be supported out of the
>> box and there are limits to what can be supported due to technical
>> reasons.  However, the stance of the GUILE developers is to stop
>> people using GUILE from doing things they consider not their problem.
>
> What is a character and what is a string is understood very well, and
> Guile provides clean abstractions for these, as any sane general
> purpose language should.

GUILE does not provide an "abstraction" but rather is Unicode-only to a
degree where (integer->char #xD800) produces an error, making it even
impossible to manipulate CESU-8 strings.

> While I can't speak for the Guile developers, I can assure you that
> their stance is to provide the best possible abstractions a general
> purpose language should provide.

Again: if GUILE is no longer considered an extension language, this
needs to be communicated to GNU in order to adapt its intended and
advertised role in the GNU project.

>> This stance, possibly partly due to only a single developer working
>> in isolation on the "unstable" branch and everybody else confined to
>> changes in the "stable" branch, is going to end a roadblock for
>> projects like GuileEmacs.
>
> You seem to be making wrong assumptions about Guile's development
> model.  I don't think the fact that the upcoming 2.2 has been mostly a
> one person project has much to do with the incompatibilities between
> 1.x and 2.x.

That's not what I said.  What I said is that work on the encoding
problems of GUILE (and having to do conversions for passing a
GUILE-internal string through a GUILE-internal string port is a problem)
has no place since the "unstable branch" contains just single-person
compiler-focused work while such changes in a "stable branch" would be
uncalled for.  This setup leads to a strong defensive stance preferring
the status quo since there is no place for most changes that are not
mere bugfixes or incremental improvements to go to.

>>> For arbitrary byte vectors, Guile has bytevectors (surprise!!).
>>>
>>> If one implemented a text editor from scratch in Guile and wanted to
>>> allow editing files with no proper encoding, one would use an
>>> abstraction on top of bytevectors.
>>
>> It is not the job of an extension language to dictate what
>> constitutes "proper encoding".  It is the job of the application.
>
> Yes, it *is* the job of a string library to offer abstractions for
> well-defined encodings of strings.

Shrug.  "abstraction" is not the same as "restriction".  Unicode is not
an abstraction.  It's a reference point for an implementation.  And as a
reference it was good and versatile enough that Emacs changed its
multibyte implementation in Emacs 23 to one based on UTF-8 and Unicode
internally.  And because Emacs multibyte encodings were rather
well-abstracted at the application layer, this change, while extensive
and pervasive, was surprisingly non-disruptive for applications.

>> GUILE is first and foremost an extension language.  That is how it
>> advertises itself on its web page.  That it supports the
>> general-purpose programming language Scheme is a boon and
>> implementation choice.
>>
>> If GUILE developers insist on GUILE being foremost a general-purpose
>> programming language rather than an extension language, they should
>> notify the GNU project and change their web page.  There is no point
>> in GNU promoting GUILE as an extension language if the GUILE
>> developers are no longer on board with that target.
>
> You seem to be making a strange distinction between a general-purpose
> and an extension language and implying that an "extension language"
> should fill exactly the use-cases of Emacs for some reason...

Not just Emacs.  Pretty much every application that requires more
specific error handling than rejecting an entire file or communication.
Like, say, LilyPond.

>> That is a pretty alien concept to established Emacs developers, Emacs
>> being the proverbial glue and shoe string across dozens of different
>> platforms right from the beginning of GNU where POSIX would have been
>> luxury.
>
> Maybe some established Emacs developers should get with the times
> then.

Shrug.  Files won't magically convert themselves into only containing
material encodable by proper UTF-8 and nothing else.  Dropping
everything achieved in character and string handling since Emacs 20 so
that GUILE must not adapt is not an option that is realistic.

> I'm not interested in continuing this discussion because it's just as
> unproductive as the last time.  GuileEmacs has come *very* far, and
> most arguments I hear against it seem based in apocalyptic FUD, and in
> particular spitefulness due to past negative experiences with the
> Guile 1.x/2.x transition, which has really nothing to do with
> GuileEmacs, a Guile 2.x project.  That is not to say it's going to be
> fairies and pixies.  There's significant work to be done on it and I
> hope I will find some time to contribute to it in the not too distant
> future.

You'll not be able to restrict all changes to Emacs (GuileEmacs being a
branch of Emacs development).  There are changes and improvements
necessary in GUILE itself to make it viable as a platform for running
Emacs on.  Denying that is doing a disfavor to those who have invested
considerable work into GuileEmacs by now since it makes it less than
more likely that GuileEmacs can sensibly become the main implementation
of Emacs.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: ludo@gnu.org (Ludovic Courtès),  Eli Zaretskii
>  <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 18 Oct 2015 14:54:59 +0200
> 
> Guile's string library is for working with strings (surprise!), not
> arbitrary byte vectors.
> 
> For arbitrary byte vectors, Guile has bytevectors (surprise!!).

One of the major lessons Emacs development learned since Emacs 20.1 is
that raw bytes happen as part of text (a.k.a. "strings"), and
therefore there's a need to support a mixture of these two in the same
buffer/string.  I think that's something Guile should support as well,
as that will make it a more powerful and flexible extension language,
able to deal with a wider range of real-life situations.

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Cc: ludo@gnu.org (Ludovic Courtès),  Eli Zaretskii
>>  <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Sun, 18 Oct 2015 14:54:59 +0200
>> 
>> Guile's string library is for working with strings (surprise!), not
>> arbitrary byte vectors.
>> 
>> For arbitrary byte vectors, Guile has bytevectors (surprise!!).
>
> One of the major lessons Emacs development learned since Emacs 20.1 is
> that raw bytes happen as part of text (a.k.a. "strings"), and
> therefore there's a need to support a mixture of these two in the same
> buffer/string.  I think that's something Guile should support as well,
> as that will make it a more powerful and flexible extension language,
> able to deal with a wider range of real-life situations.

Text as understood by a user of a text editor does not necessarily
correspond to strings as understood by a programmer.

But indeed, it's foreseeable that Guile would support such things in its
native string data type eventually, when explicitly requested.  ISTR
this was mentioned before, by Guile developers themselves.  Then Elisp
and Guile strings would not need to be different data types anymore.

Though that doesn't seem to be an absolute requirement for letting
libguile compile Elisp, since it can just treat Elisp strings as a
separate data type, as GuileEmacs currently does.

Taylan

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Sun, 18 Oct 2015 18:37:24 +0200
> Cc: ludo@gnu.org, dak@gnu.org, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> >> Cc: ludo@gnu.org (Ludovic Courtès),  Eli Zaretskii
> >>  <eliz@gnu.org>,  emacs-devel@gnu.org
> >> Date: Sun, 18 Oct 2015 14:54:59 +0200
> >> 
> >> Guile's string library is for working with strings (surprise!), not
> >> arbitrary byte vectors.
> >> 
> >> For arbitrary byte vectors, Guile has bytevectors (surprise!!).
> >
> > One of the major lessons Emacs development learned since Emacs 20.1 is
> > that raw bytes happen as part of text (a.k.a. "strings"), and
> > therefore there's a need to support a mixture of these two in the same
> > buffer/string.  I think that's something Guile should support as well,
> > as that will make it a more powerful and flexible extension language,
> > able to deal with a wider range of real-life situations.
> 
> Text as understood by a user of a text editor does not necessarily
> correspond to strings as understood by a programmer.

It doesn't matter how this data type is called, what's important is
that it should support both human-readable text and raw bytes in the
same contiguous stream of "characters".

> Though that doesn't seem to be an absolute requirement for letting
> libguile compile Elisp, since it can just treat Elisp strings as a
> separate data type, as GuileEmacs currently does.

That'd be an inconvenience, since it will require a translation layer
between Emacs strings and Guile strings, and that will hamper
performance.

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Date: Sun, 18 Oct 2015 18:37:24 +0200
>> Cc: ludo@gnu.org, dak@gnu.org, emacs-devel@gnu.org
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> >> Cc: ludo@gnu.org (Ludovic Courtès),  Eli Zaretskii
>> >>  <eliz@gnu.org>,  emacs-devel@gnu.org
>> >> Date: Sun, 18 Oct 2015 14:54:59 +0200
>> >> 
>> >> Guile's string library is for working with strings (surprise!), not
>> >> arbitrary byte vectors.
>> >> 
>> >> For arbitrary byte vectors, Guile has bytevectors (surprise!!).
>> >
>> > One of the major lessons Emacs development learned since Emacs 20.1 is
>> > that raw bytes happen as part of text (a.k.a. "strings"), and
>> > therefore there's a need to support a mixture of these two in the same
>> > buffer/string.  I think that's something Guile should support as well,
>> > as that will make it a more powerful and flexible extension language,
>> > able to deal with a wider range of real-life situations.
>> 
>> Text as understood by a user of a text editor does not necessarily
>> correspond to strings as understood by a programmer.
>
> It doesn't matter how this data type is called, what's important is
> that it should support both human-readable text and raw bytes in the
> same contiguous stream of "characters".
>
>> Though that doesn't seem to be an absolute requirement for letting
>> libguile compile Elisp, since it can just treat Elisp strings as a
>> separate data type, as GuileEmacs currently does.
>
> That'd be an inconvenience, since it will require a translation layer
> between Emacs strings and Guile strings, and that will hamper
> performance.

I'm not sure in what situations such a translation would be necessary.
I don't think GuileEmacs currently does any such translations.

See e.g. the "can I script Emacs in Scheme already" examples here:
http://www.emacswiki.org/emacs/GuileEmacs

If one passed Elisp strings to a Scheme API or Scheme strings to an
Elisp API there, it would raise an invalid type error.

This is akin to how one cannot interchange C integers with Elisp
integers and all that stuff which one needs to know while working on the
C code of Emacs, except 1. in this case it only applies to strings, and
2. it cleanly raises a type error instead of doing something funny.
(The other day I was hit by this issue while working on Guile's C code,
where I forgot that a scm_blahblah_p predicate will return SCM_FALSE
which is not 0 thus not false in C.  Like Qnil in Emacs.)

Taylan

Re: Emacs rewrite in a maintainable language

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> (The other day I was hit by this issue while working on Guile's C code,
> where I forgot that a scm_blahblah_p predicate will return SCM_FALSE
> which is not 0 thus not false in C.  Like Qnil in Emacs.)

From src/lisp.h:

/* True, since Qnil's representation is zero.  Every place in the code
   that assumes Qnil is zero should verify (NIL_IS_ZERO), to make it easy
   to find such assumptions later if we change Qnil to be nonzero.  */
enum { NIL_IS_ZERO = XLI_BUILTIN_LISPSYM (iQnil) == 0 };


-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: ludo@gnu.org,  dak@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 18 Oct 2015 19:06:22 +0200
> 
> >> Though that doesn't seem to be an absolute requirement for letting
> >> libguile compile Elisp, since it can just treat Elisp strings as a
> >> separate data type, as GuileEmacs currently does.
> >
> > That'd be an inconvenience, since it will require a translation layer
> > between Emacs strings and Guile strings, and that will hamper
> > performance.
> 
> I'm not sure in what situations such a translation would be necessary.

When we need to use Guile APIs from Emacs.  That's one of the
advantages of using Guile: that it implements a lot of functionality
which it would be good to have in Emacs.

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> When we need to use Guile APIs from Emacs. That's one of the advantages of
> using Guile: that it implements a lot of functionality which it would be
> good to have in Emacs.

Total agreement! I've been wanting a Guile-based Emacs since 1998, if for no
other reason than that I could have used call/cc in Eshell, instead of the
horrendous CPS hack that exists there now.

John

Re: Emacs rewrite in a maintainable language

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

On 10/18/2015 10:52 AM, John Wiegley wrote:
>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> When we need to use Guile APIs from Emacs. That's one of the advantages of
>> using Guile: that it implements a lot of functionality which it would be
>> good to have in Emacs.
> 
> Total agreement! I've been wanting a Guile-based Emacs since 1998, if for no
> other reason than that I could have used call/cc in Eshell, instead of the
> horrendous CPS hack that exists there now.

You have generators in elisp already. Wanting to use one language is,
IMHO, a poor choice for wanting to completely swap out a language. I am
opposed to Guilemacs, not only on technical grounds, but also because
elisp is essential to Emacs (and not just an optional extension system),
and I want its implementation to live alongside the rest of the Emacs
core code.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 10/18/2015 10:52 AM, John Wiegley wrote:
>>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>>> When we need to use Guile APIs from Emacs. That's one of the
>>> advantages of using Guile: that it implements a lot of functionality
>>> which it would be good to have in Emacs.
>> 
>> Total agreement! I've been wanting a Guile-based Emacs since 1998, if
>> for no other reason than that I could have used call/cc in Eshell,
>> instead of the horrendous CPS hack that exists there now.
>
> You have generators in elisp already.

Well, lexical-binding is a lot more recent than the start of the
GuileEmacs project, and part of the appeal of GuileEmacs was indeed to
have lexical bindings.  But that does not make GUILE worse but rather
Elisp better.

> Wanting to use one language is, IMHO, a poor choice for wanting to
> completely swap out a language. I am opposed to Guilemacs, not only on
> technical grounds, but also because elisp is essential to Emacs (and
> not just an optional extension system), and I want its implementation
> to live alongside the rest of the Emacs core code.

I'm not convinced that it's a bad idea to separate the Elisp
implementation more from the Emacs core code.  It provides a
well-documented interface between the two: hacking the C code in Emacs
remains a considerable inside job and is not documented on its own.

So I consider this a strength rather than a weakness of the GuileEmacs
proposition in the long term.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1944 bytes --]

On 10/18/2015 11:35 AM, David Kastrup wrote:
> Daniel Colascione <dancol@dancol.org> writes:
>> Wanting to use one language is, IMHO, a poor choice for wanting to
>> completely swap out a language. I am opposed to Guilemacs, not only on
>> technical grounds, but also because elisp is essential to Emacs (and
>> not just an optional extension system), and I want its implementation
>> to live alongside the rest of the Emacs core code.
>
> I'm not convinced that it's a bad idea to separate the Elisp
> implementation more from the Emacs core code.  It provides a
> well-documented interface between the two: hacking the C code in Emacs
> remains a considerable inside job and is not documented on its own.
>
> So I consider this a strength rather than a weakness of the GuileEmacs
> proposition in the long term.

I disagree. Integrating the interpreter and the editor makes integrated
changes easy. It also makes elisp releases synchronous with Emacs ones.
I don't think a strong library separate here gives us anything useful.

Consider my recent change to add finalizers to elisp. I saw a need for
the feature and just implemented it directly in Emacs. What would the
equivalent be in a guilemacs world? I'd have had to make the change
upstream in guile (where I suspect the process is much more involved),
wait for a stable release of guile, added Emacs support, and then still
not have been able to rely on the feature until Emacs dropped support
for the last version of Guile to lack the feature.

We'd have to go to all that trouble for what, exactly? A cleaner
internal API? I don't buy it. Guilemacs has other disadvantages:
currently, Emacs supports _only_ elisp as a first-class extension
language. Guilemacs would invite people to write Emacs extensions in
Scheme, JavaScript, and whatever else Guile ends up supporting, which
will create fragmentation. A unified elisp ecosystem is a strength.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 10/18/2015 11:35 AM, David Kastrup wrote:
>> Daniel Colascione <dancol@dancol.org> writes:
>>> Wanting to use one language is, IMHO, a poor choice for wanting to
>>> completely swap out a language. I am opposed to Guilemacs, not only on
>>> technical grounds, but also because elisp is essential to Emacs (and
>>> not just an optional extension system), and I want its implementation
>>> to live alongside the rest of the Emacs core code.
>>
>> I'm not convinced that it's a bad idea to separate the Elisp
>> implementation more from the Emacs core code.  It provides a
>> well-documented interface between the two: hacking the C code in Emacs
>> remains a considerable inside job and is not documented on its own.
>>
>> So I consider this a strength rather than a weakness of the GuileEmacs
>> proposition in the long term.
>
> I disagree.

Then we disagree.

> Integrating the interpreter and the editor makes integrated changes
> easy. It also makes elisp releases synchronous with Emacs ones.  I
> don't think a strong library separate here gives us anything useful.
>
> Consider my recent change to add finalizers to elisp. I saw a need for
> the feature and just implemented it directly in Emacs. What would the
> equivalent be in a guilemacs world?

You'd have used GUILE's guardians, and could have done this as an ELPA
package with a GUILE component.  No recompilation required for an end
user.

> We'd have to go to all that trouble for what, exactly? A cleaner
> internal API? I don't buy it. Guilemacs has other disadvantages:
> currently, Emacs supports _only_ elisp as a first-class extension
> language. Guilemacs would invite people to write Emacs extensions in
> Scheme, JavaScript, and whatever else Guile ends up supporting, which
> will create fragmentation. A unified elisp ecosystem is a strength.

I don't consider options a problem.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Paul Eggert]

Daniel Colascione wrote:
> Consider my recent change to add finalizers to elisp. I saw a need for
> the feature and just implemented it directly in Emacs. What would the
> equivalent be in a guilemacs world? I'd have had to make the change
> upstream in guile (where I suspect the process is much more involved),
> wait for a stable release of guile, added Emacs support, and then still
> not have been able to rely on the feature until Emacs dropped support
> for the last version of Guile to lack the feature.

This is a reasonable objection. Perhaps we could overcome it the same way we 
overcome a similar problem with Emacs and Gnulib. When you need to make a change 
involving Gnulib code in Emacs (say, to the file lib/time.in.h), there isn't a 
problem: you can just change lib/time.in.h in the Savannah git master for Emacs. 
At some point a gnome comes along, notices that the Emacs copy of lib/time.in.h 
disagrees with the Gnulib copy, and merges the two.

We could do the same for Guile. That is, the Emacs savannah git could contain a 
copy of the relevant Guile sources (presumably only a subset of the Guile source 
code would be needed, just as only a small subset of Gnulib is needed). There 
would be an established procedure to merge the two when they become out of sync; 
for Gnulib, this is partly automated by Emacs's admin/merge-gnulib script.

For Gnulib I'm the gnome. For this process to work well with Guile, we'd need a 
gnome or two for Guile.

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

Daniel Colascione <dancol@dancol.org> writes:

> On 10/18/2015 11:35 AM, David Kastrup wrote:
>> Daniel Colascione <dancol@dancol.org> writes:
>>> Wanting to use one language is, IMHO, a poor choice for wanting to
>>> completely swap out a language. I am opposed to Guilemacs, not only on
>>> technical grounds, but also because elisp is essential to Emacs (and
>>> not just an optional extension system), and I want its implementation
>>> to live alongside the rest of the Emacs core code.
>>
>> I'm not convinced that it's a bad idea to separate the Elisp
>> implementation more from the Emacs core code.  It provides a
>> well-documented interface between the two: hacking the C code in Emacs
>> remains a considerable inside job and is not documented on its own.
>>
>> So I consider this a strength rather than a weakness of the GuileEmacs
>> proposition in the long term.
>
> I disagree. Integrating the interpreter and the editor makes integrated
> changes easy. It also makes elisp releases synchronous with Emacs ones.
> I don't think a strong library separate here gives us anything useful.
>
> Consider my recent change to add finalizers to elisp. I saw a need for
> the feature and just implemented it directly in Emacs. What would the
> equivalent be in a guilemacs world? I'd have had to make the change
> upstream in guile (where I suspect the process is much more involved),
> wait for a stable release of guile, added Emacs support, and then still
> not have been able to rely on the feature until Emacs dropped support
> for the last version of Guile to lack the feature.
>
> We'd have to go to all that trouble for what, exactly? A cleaner
> internal API? I don't buy it. Guilemacs has other disadvantages:
> currently, Emacs supports _only_ elisp as a first-class extension
> language. Guilemacs would invite people to write Emacs extensions in
> Scheme, JavaScript, and whatever else Guile ends up supporting, which
> will create fragmentation. A unified elisp ecosystem is a strength.

Guile already supports finalizers (and much, much more) so you wouldn't
have needed to implement it from scratch for Elisp. :-)

Taylan

Re: Emacs rewrite in a maintainable language

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 4459 bytes --]

On 10/18/2015 12:35 PM, Taylan Ulrich Bayırlı/Kammer wrote:
> Daniel Colascione <dancol@dancol.org> writes:
> 
>> On 10/18/2015 11:35 AM, David Kastrup wrote:
>>> Daniel Colascione <dancol@dancol.org> writes:
>>>> Wanting to use one language is, IMHO, a poor choice for wanting to
>>>> completely swap out a language. I am opposed to Guilemacs, not only on
>>>> technical grounds, but also because elisp is essential to Emacs (and
>>>> not just an optional extension system), and I want its implementation
>>>> to live alongside the rest of the Emacs core code.
>>>
>>> I'm not convinced that it's a bad idea to separate the Elisp
>>> implementation more from the Emacs core code.  It provides a
>>> well-documented interface between the two: hacking the C code in Emacs
>>> remains a considerable inside job and is not documented on its own.
>>>
>>> So I consider this a strength rather than a weakness of the GuileEmacs
>>> proposition in the long term.
>>
>> I disagree. Integrating the interpreter and the editor makes integrated
>> changes easy. It also makes elisp releases synchronous with Emacs ones.
>> I don't think a strong library separate here gives us anything useful.
>>
>> Consider my recent change to add finalizers to elisp. I saw a need for
>> the feature and just implemented it directly in Emacs. What would the
>> equivalent be in a guilemacs world? I'd have had to make the change
>> upstream in guile (where I suspect the process is much more involved),
>> wait for a stable release of guile, added Emacs support, and then still
>> not have been able to rely on the feature until Emacs dropped support
>> for the last version of Guile to lack the feature.
>>
>> We'd have to go to all that trouble for what, exactly? A cleaner
>> internal API? I don't buy it. Guilemacs has other disadvantages:
>> currently, Emacs supports _only_ elisp as a first-class extension
>> language. Guilemacs would invite people to write Emacs extensions in
>> Scheme, JavaScript, and whatever else Guile ends up supporting, which
>> will create fragmentation. A unified elisp ecosystem is a strength.
> 
> Guile already supports finalizers (and much, much more) so you wouldn't
> have needed to implement it from scratch for Elisp. :-)

I'm sure we'll have to extend something sometime.

> record types (like an improved defstruct)

We have EIEIO. And defstruct is fine for most use cases. Over in the
modules discussion, we're talking about letting C modules define
first-class types in elisp. That'd be neat, and defstruct could use the
same machinery instead of a type symbol.

> CLOS-like OOP

EIEIO.

> an FFI

We're getting modules separately.

> composable continuations

We have them already in the form of generators. I don't want
unrestricted call/cc.

> a module system

I don't want Guile's.

> hygienic macros

Unhygienic macros are useful. I don't want Guile's opinionated macros.
Besides, this point is out-of-date if Guile supports elisp, since
existing elisp macros are unhygienic.

> multiple-value returns

Returning lists works fine, and everyone I know finds CL's multiple
return feature confusing. We can live without it easily, since most
other languages have managed to live without it.

Besides: we support multiple return values _today_ in the form of
locatives: we call them gv-ref and gv-deref.

> and threads

Guile by itself sure as hell won't give us threads. You'll have to solve
all the synchronization problems that adding threads will inflict on the
Emacs core. The existing threads branch goes a long way toward doing that.

I really don't want to shoehorn Guile into Emacs. It's the wrong runtime
for the wrong language being advocated for the wrong reasons.

I wouldn't mind taking _pieces_ of Guile --- in particular, maybe the
JIT and GC --- and replacing the existing Emacs implementations of these
runtime facilities, but I don't want Emacs linking against libguile.so,
I don't want users to have to install a separate package, and I don't
want our platform support to be limited by whatever Guile provides.
Basically, I don't want our future in Guile's hands.

More broadly, I don't think the Emacs lisp implementation is anywhere
close to being a priority. It's adequate for now, especially given that
we have lexical binding and real closures. The IDE-like features we're
missing are much more important.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

Daniel Colascione <dancol@dancol.org> writes:

> On 10/18/2015 12:35 PM, Taylan Ulrich Bayırlı/Kammer wrote:
>> Daniel Colascione <dancol@dancol.org> writes:
>> 
>>> On 10/18/2015 11:35 AM, David Kastrup wrote:
>>>> Daniel Colascione <dancol@dancol.org> writes:
>>>>> Wanting to use one language is, IMHO, a poor choice for wanting to
>>>>> completely swap out a language. I am opposed to Guilemacs, not only on
>>>>> technical grounds, but also because elisp is essential to Emacs (and
>>>>> not just an optional extension system), and I want its implementation
>>>>> to live alongside the rest of the Emacs core code.
>>>>
>>>> I'm not convinced that it's a bad idea to separate the Elisp
>>>> implementation more from the Emacs core code.  It provides a
>>>> well-documented interface between the two: hacking the C code in Emacs
>>>> remains a considerable inside job and is not documented on its own.
>>>>
>>>> So I consider this a strength rather than a weakness of the GuileEmacs
>>>> proposition in the long term.
>>>
>>> I disagree. Integrating the interpreter and the editor makes integrated
>>> changes easy. It also makes elisp releases synchronous with Emacs ones.
>>> I don't think a strong library separate here gives us anything useful.
>>>
>>> Consider my recent change to add finalizers to elisp. I saw a need for
>>> the feature and just implemented it directly in Emacs. What would the
>>> equivalent be in a guilemacs world? I'd have had to make the change
>>> upstream in guile (where I suspect the process is much more involved),
>>> wait for a stable release of guile, added Emacs support, and then still
>>> not have been able to rely on the feature until Emacs dropped support
>>> for the last version of Guile to lack the feature.
>>>
>>> We'd have to go to all that trouble for what, exactly? A cleaner
>>> internal API? I don't buy it. Guilemacs has other disadvantages:
>>> currently, Emacs supports _only_ elisp as a first-class extension
>>> language. Guilemacs would invite people to write Emacs extensions in
>>> Scheme, JavaScript, and whatever else Guile ends up supporting, which
>>> will create fragmentation. A unified elisp ecosystem is a strength.
>> 
>> Guile already supports finalizers (and much, much more) so you wouldn't
>> have needed to implement it from scratch for Elisp. :-)
>
> I'm sure we'll have to extend something sometime.

I think Guile is probably more fertile ground for improvement of
language semantics and implementation.

>> record types (like an improved defstruct)
>
> We have EIEIO. And defstruct is fine for most use cases. Over in the
> modules discussion, we're talking about letting C modules define
> first-class types in elisp. That'd be neat, and defstruct could use the
> same machinery instead of a type symbol.
>
>> CLOS-like OOP
>
> EIEIO.

I've heard bad things about both defstruct and EIEIO for different
reasons.  The fact that most Elisp code is shy of using even defstruct
should tell us something.

You say "this and that will fix that" but it's already fixed in Guile.

Guile also already supports defining new types in C.

>> an FFI
>
> We're getting modules separately.

I'm not sure if that's comparable to an FFI.

>> composable continuations
>
> We have them already in the form of generators. I don't want
> unrestricted call/cc.

Call/cc is not composable.  Composable continuations can implement
generators and more, without having the problems of call/cc.

>> a module system
>
> I don't want Guile's.

And I do ... but it would be better if we stick to technical points
instead of personal preference. :-)

>> hygienic macros
>
> Unhygienic macros are useful. I don't want Guile's opinionated macros.
> Besides, this point is out-of-date if Guile supports elisp, since
> existing elisp macros are unhygienic.

Guile supports hygienic and unhygienic macros.  Hygienic macros are also
better called "not broken by default" macros, if we're going to start
using slants. ;-)

>> multiple-value returns
>
> Returning lists works fine, and everyone I know finds CL's multiple
> return feature confusing. We can live without it easily, since most
> other languages have managed to live without it.

MV returns can be very useful (and natural) for some things.
Programmers should have the choice to use them.

> Besides: we support multiple return values _today_ in the form of
> locatives: we call them gv-ref and gv-deref.

I don't think that's the same thing.

>> and threads
>
> Guile by itself sure as hell won't give us threads. You'll have to solve
> all the synchronization problems that adding threads will inflict on the
> Emacs core. The existing threads branch goes a long way toward doing that.
>
> I really don't want to shoehorn Guile into Emacs. It's the wrong runtime
> for the wrong language being advocated for the wrong reasons.
>
> I wouldn't mind taking _pieces_ of Guile --- in particular, maybe the
> JIT and GC --- and replacing the existing Emacs implementations of these
> runtime facilities, but I don't want Emacs linking against libguile.so,
> I don't want users to have to install a separate package, and I don't
> want our platform support to be limited by whatever Guile provides.
> Basically, I don't want our future in Guile's hands.
>
> More broadly, I don't think the Emacs lisp implementation is anywhere
> close to being a priority. It's adequate for now, especially given that
> we have lexical binding and real closures. The IDE-like features we're
> missing are much more important.

You want a lot of things and don't want a lot of things but only some of
them seem to be backed by concrete reasons instead of personal opinion.


I'll now leave this branch of the discussion too because it's probably
not going to head in a good direction.  I would much rather do something
constructive than try counter people's panicky reactions to Guile,
whatever reason they have.

Taylan

Re: Emacs rewrite in a maintainable language

[Stephen J. Turnbull]

Taylan Ulrich Bayırlı 	/Kammer writes:
 > I've heard bad things about both defstruct and EIEIO for different
 > reasons.  The fact that most Elisp code is shy of using even defstruct
 > should tell us something.

It does.  It tells us that RMS doesn't like abstract data types.
AFAICT there's little inherent problem with defstruct from cl-macs (or
cl-lib, I forget which), it's just a matter of style preference
(originally rooted in the claim that cl.el was just syntactic sugar so
it was a waste of pure space on small machines to require it).

 > >> an FFI
 > >
 > > We're getting modules separately.
 > 
 > I'm not sure if that's comparable to an FFI.

Does Guile's FFI refuse to load code if it doesn't call the I-swear-
I'm-GPLed function?  That's another requirement for an FFI/module
system in Emacs, at least for the present.

 > I'll now leave this branch of the discussion too because it's
 > probably not going to head in a good direction.  I would much
 > rather do something constructive than try counter people's panicky
 > reactions to Guile, whatever reason they have.

I'm not panicking.  GuileEmacs has zero attraction for me *personally*
because on the one hand its advocates admit it still needs work.  On
the other none of its claimed advantages excite *me* one bit.

If we can really rewrite all of Emacs with the exception of device
drivers in Guile, then I'd definitely be excited (I think that's what
Tom Tromey is talking about).  But I don't think that effort is likely
to succeed in providing an efficient Emacs at all soon, although it
might provide an efficient Emacs Lisp.  So we would end up with three
implementation languages required to understand important (generic)
components of Emacs.  Specifically I suppose that redisplay is likely
to remain in C for a long time.  I don't think that's a win,
especially with the e^i mental twist required when moving between
Scheme code and Lisp code.

It doesn't bother me if somebody else wants to do the work, though.

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> Date: Mon, 19 Oct 2015 19:50:53 +0900
> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Daniel Colascione <dancol@dancol.org>, David Kastrup <dak@gnu.org>,
> 	emacs-devel@gnu.org
> 
> Taylan Ulrich Bayırlı 	/Kammer writes:
>  > I've heard bad things about both defstruct and EIEIO for different
>  > reasons.  The fact that most Elisp code is shy of using even defstruct
>  > should tell us something.
> 
> It does.  It tells us that RMS doesn't like abstract data types.

Reality check:

  fgrep -Rw defstruct lisp --include="*.el" | wc
      172     815   12631

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 19 Oct 2015 19:50:53 +0900
>> From: "Stephen J. Turnbull" <stephen@xemacs.org>
>> Cc: Daniel Colascione <dancol@dancol.org>, David Kastrup <dak@gnu.org>,
>> 	emacs-devel@gnu.org
>> 
>> Taylan Ulrich Bayırlı 	/Kammer writes:
>>  > I've heard bad things about both defstruct and EIEIO for different
>>  > reasons.  The fact that most Elisp code is shy of using even defstruct
>>  > should tell us something.
>> 
>> It does.  It tells us that RMS doesn't like abstract data types.
>
> Reality check:
>
>   fgrep -Rw defstruct lisp --include="*.el" | wc
>       172     815   12631

That's 172 lines with occurences of defstruct (which usually occurs once
per line, but since those lines have also a file name and other words,
the second number appears to signify more than there is).

Here is the output of the above fgrep command until the actual use of a
native defstruct outside of a comment (I think we can savely assume that
RMS would not really use cl-defstruct as he is not known to be a fan of
cl):

lisp/url/url-queue.el:(cl-defstruct url-queue
lisp/url/url-parse.el:(cl-defstruct (url
lisp/url/url-future.el:;; Make a url-future (basically a defstruct):
lisp/url/url-future.el:(cl-defstruct url-future callback errorback status value)
lisp/url/url-cookie.el:(cl-defstruct (url-cookie
lisp/textmodes/rst.el:;;        `defstruct'.
lisp/textmodes/rst.el:;; FIXME: Return value should be a `defstruct'.
lisp/textmodes/rst.el:;; FIXME: Return value should be a `defstruct'.
lisp/textmodes/sgml-mode.el:(cl-defstruct (sgml-tag
lisp/mh-e/mh-acros.el:(defmacro mh-defstruct (name-spec &rest fields)
lisp/mh-e/mh-acros.el:  "Replacement for `defstruct' from the \"cl\" package.
lisp/mh-e/mh-acros.el:The `defstruct' in the \"cl\" library produces compiler warnings,
lisp/mh-e/mh-acros.el:describes the various structure fields. Lookup `defstruct' for
lisp/mh-e/mh-loaddefs.el:(autoload 'mh-defstruct "mh-acros" "\
lisp/mh-e/mh-loaddefs.el:Replacement for `defstruct' from the \"cl\" package.
lisp/mh-e/mh-loaddefs.el:The `defstruct' in the \"cl\" library produces compiler warnings,
lisp/mh-e/mh-loaddefs.el:describes the various structure fields. Lookup `defstruct' for
lisp/mh-e/mh-thread.el:(mh-defstruct (mh-thread-message (:conc-name mh-message-)
lisp/mh-e/mh-thread.el:(mh-defstruct (mh-thread-container (:conc-name mh-container-)
lisp/mh-e/mh-mime.el:(mh-defstruct (mh-buffer-data (:conc-name mh-mime-)
lisp/profiler.el:(cl-defstruct (profiler-profile (:type vector)
lisp/profiler.el:(cl-defstruct (profiler-calltree (:constructor profiler-make-calltree))
lisp/progmodes/ebrowse.el:(cl-defstruct (ebrowse-hs (:type vector) :named)
lisp/progmodes/ebrowse.el:(cl-defstruct (ebrowse-ts (:type vector) :named)
lisp/progmodes/ebrowse.el:(cl-defstruct (ebrowse-bs (:type vector) :named)
lisp/progmodes/ebrowse.el:(cl-defstruct (ebrowse-cs (:include ebrowse-bs) (:type vector) :named)
lisp/progmodes/ebrowse.el:(cl-defstruct (ebrowse-ms (:include ebrowse-bs) (:type vector) :named)
lisp/progmodes/ebrowse.el:(cl-defstruct (ebrowse-position (:type vector) :named)
lisp/progmodes/flymake.el:(cl-defstruct (flymake-ler
lisp/progmodes/compile.el:;; (cl-defstruct (compilation--loc
lisp/progmodes/compile.el:;; FIXME: We don't use a defstruct because of compilation-assq which looks up
lisp/progmodes/compile.el:(cl-defstruct (compilation--message
lisp/progmodes/js.el:(cl-defstruct (js--pitem (:type list))
lisp/progmodes/js.el:(cl-defstruct js--js-handle
lisp/progmodes/elisp-mode.el:                   ;; This doc string is defined in cl-macs.el cl-defstruct
lisp/progmodes/elisp-mode.el:              ;; cl-defstruct, so return the location of the cl-defstruct.
lisp/progmodes/elisp-mode.el:                                      'cl-defstruct
lisp/progmodes/elisp-mode.el:(cl-defstruct (xref-elisp-location
lisp/progmodes/gdb-mi.el:(cl-defstruct gdb-handler
lisp/progmodes/gdb-mi.el:(cl-defstruct gdb-table
lisp/tar-mode.el:(cl-defstruct (tar-header
lisp/org/ox.el:(defstruct (org-export-backend (:constructor org-export-create-backend)


-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Date: Mon, 19 Oct 2015 19:50:53 +0900
>>> From: "Stephen J. Turnbull" <stephen@xemacs.org>
>>> Cc: Daniel Colascione <dancol@dancol.org>, David Kastrup <dak@gnu.org>,
>>> 	emacs-devel@gnu.org
>>> 
>>> Taylan Ulrich Bayırlı 	/Kammer writes:
>>>  > I've heard bad things about both defstruct and EIEIO for different
>>>  > reasons.  The fact that most Elisp code is shy of using even defstruct
>>>  > should tell us something.
>>> 
>>> It does.  It tells us that RMS doesn't like abstract data types.
>>
>> Reality check:
>>
>>   fgrep -Rw defstruct lisp --include="*.el" | wc
>>       172     815   12631
>
> That's 172 lines with occurences of defstruct (which usually occurs once
> per line, but since those lines have also a file name and other words,
> the second number appears to signify more than there is).
>
> Here is the output of the above fgrep command until the actual use of a
> native defstruct outside of a comment (I think we can savely assume that
> RMS would not really use cl-defstruct as he is not known to be a fan of
> cl):
>
[...]

There are 37 active occurences of defstruct (I am not counting the code
for indenting defstruct in lisp-mode, for example), about half in
nnmaildir (older than 2004 when a gnus merge was done by Miles Bader)
and half in soap-client (in 2011) by Paul Jarc and Alexandru Harsanyi,
respectively.

Reality check indeed.  cl-defstruct occurs more often in that list.
I think it is safe to say that Richard never used defstruct.  Not that
I'd consider this tantamount to "doesn't like abstract data types".

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Taylan Ulrich Bayırlı/Kammer writes:
>  > I've heard bad things about both defstruct and EIEIO for different
>  > reasons.  The fact that most Elisp code is shy of using even defstruct
>  > should tell us something.
>
> It does.  It tells us that RMS doesn't like abstract data types.
> AFAICT there's little inherent problem with defstruct from cl-macs (or
> cl-lib, I forget which), it's just a matter of style preference
> (originally rooted in the claim that cl.el was just syntactic sugar so
> it was a waste of pure space on small machines to require it).
>
>  > >> an FFI
>  > >
>  > > We're getting modules separately.
>  > 
>  > I'm not sure if that's comparable to an FFI.
>
> Does Guile's FFI refuse to load code if it doesn't call the I-swear-
> I'm-GPLed function?  That's another requirement for an FFI/module
> system in Emacs, at least for the present.

My personal opinion is that It's unlikely to survive into GuileEmacs for
mostly technical reasons.  The best one could do is make it stick when a
module calls Emacs-specific functionality.  But if it uses the general
GUILE stuff (namely could be called from a standard GUILE interpreter),
Emacs cannot really make a licensing claim that sticks.

> I'm not panicking.  GuileEmacs has zero attraction for me *personally*
> because on the one hand its advocates admit it still needs work.  On
> the other none of its claimed advantages excite *me* one bit.

I like arbitrary precision integers and rationals and the whole numeric
stack.  I also like the availability of applications and libraries that
have not been written specifically for Emacs: replacing much of the
backend behind Emacs Calc with native Jacal would be an interesting
feat, and there are a number of other pathways opening up.

> If we can really rewrite all of Emacs with the exception of device
> drivers in Guile, then I'd definitely be excited (I think that's what
> Tom Tromey is talking about).  But I don't think that effort is likely
> to succeed in providing an efficient Emacs at all soon, although it
> might provide an efficient Emacs Lisp.  So we would end up with three
> implementation languages required to understand important (generic)
> components of Emacs.  Specifically I suppose that redisplay is likely
> to remain in C for a long time.  I don't think that's a win,
> especially with the e^i mental twist required when moving between
> Scheme code and Lisp code.

Well, it would move the worlds of SCSH and Scwm closer to Emacs.  Eshell
never took off, partly because of being severely underdocumented, partly
(related) because of being a one-person project.  Run Scsh inside of
Emacs and things may become interesting.

Elisp is not used for creating scripts to any serious degree, the
existence of rep notwithstanding (show of hands: which Emacs developer
ever worked with rep or even knows it?).

GUILE can (and will) be used for scripting.  So for GNU integration and
desktop cohesion, this may be a strategical step.  It also muddies the
application boundaries when Scheme applications gain enough glue to
integrate nicely into Emacs and that's something that makes the GPL
stickier again.

So I do see longterm goals and strategies that could be opened by a good
integration of GUILE as a core part of Emacs.

The most relevant obstacle _will_ be to overcome the "don't mess with my
project" stance in both Emacs and GUILE from people that have come to
like the respective culture and environment of the isolated projects.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Stephen J. Turnbull]

David Kastrup writes:

 > My personal opinion is that It's unlikely to survive into GuileEmacs for
 > mostly technical reasons.  The best one could do is make it stick when a
 > module calls Emacs-specific functionality.  But if it uses the general
 > GUILE stuff (namely could be called from a standard GUILE interpreter),
 > Emacs cannot really make a licensing claim that sticks.

[David pointed this out elsewhere, but I want to emphasize: GUILE
distributed as a part of Emacs is GPL, not LGPL, and the protection of
the GPL extends to all code in the Emacs process space.  GuileEmacs
can be distributed only because the LGPL allows promotion to GPL.]

If you're just referring to the fact that any code that doesn't work
with buffers or call Emacs redisplay etc can be licensed as you like,
sure.  But that's always true anyway.  If the module is written in
GUILE Scheme or via GUILE FFI with the intention that its input or
output be Emacs-specific, however, it becomes a derivative of Emacs,
and the GPL applies just as it would with "native" Emacs modules.
Proving that in court might be sticky, but I don't think it would stop
RMS from insisting on a "I'm GPL" API in GuileEmacs distributed by
GNU, and making downstream make any decision about allowing broader
modules (and perhaps suffer an FSF legal action if RMS believes that
appropriate).

What might give RMS pause would be the prospect of forking GUILE to
add such an API.

 > > I'm not panicking.  GuileEmacs has zero attraction for me *personally*
 > > because on the one hand its advocates admit it still needs work.  On
 > > the other none of its claimed advantages excite *me* one bit.
 > 
 > I like arbitrary precision integers and rationals and the whole numeric
 > stack.

I have those. ;-)  (Joke because XEmacs doesn't have a true numeric
tower.  SXEmacs may have made progress in that direction, though.)

 > I also like the availability of applications and libraries that
 > have not been written specifically for Emacs: replacing much of the
 > backend behind Emacs Calc with native Jacal would be an interesting
 > feat, and there are a number of other pathways opening up.

Indeed interesting, although these days I'd be more interested in
numpy and numba.  But I agree with your word "feat"; I'd be surprised
when it happened; I suspect Jay Belanger isn't so interested in that.
Maybe he'll speak up.

 > Elisp is not used for creating scripts to any serious degree, the
 > existence of rep notwithstanding (show of hands: which Emacs developer
 > ever worked with rep or even knows it?).

Isn't that the engine in the sawfish WM?  <raises limb="arm"/>

 > GUILE can (and will) be used for scripting.  So for GNU integration
 > and desktop cohesion, this may be a strategical step.

Agreed that's very interesting and *will* happen (but *my* desktop is
Mac OS X, hate GNOME 3, hate hate hate hate ;-) -- of course it would
be very cool for GNOME users, and there are many happy GNOME users).

 > So I do see longterm goals and strategies that could be opened by a good
 > integration of GUILE as a core part of Emacs.

Sure.  I just don't see them *for me* and *right now*.

 > The most relevant obstacle _will_ be to overcome the "don't mess
 > with my project" stance in both Emacs and GUILE from people that
 > have come to like the respective culture and environment of the
 > isolated projects.

Most likely. :-(  But if it really looks like a great idea, RMS will
come and threaten to knock heads, and that will be all it takes for
all to turn sweetness and light (you and me excepted, of course ;-).

Re: Emacs rewrite in a maintainable language

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > My personal opinion is that It's unlikely to survive into GuileEmacs for
>  > mostly technical reasons.  The best one could do is make it stick when a
>  > module calls Emacs-specific functionality.  But if it uses the general
>  > GUILE stuff (namely could be called from a standard GUILE interpreter),
>  > Emacs cannot really make a licensing claim that sticks.
>
> [David pointed this out elsewhere, but I want to emphasize: GUILE
> distributed as a part of Emacs is GPL, not LGPL, and the protection of
> the GPL extends to all code in the Emacs process space.  GuileEmacs
> can be distributed only because the LGPL allows promotion to GPL.]

"process space" is not a legal concept.  It might be a programmer's
rationale of the "dynamic and static linking make no difference with
regard to whether a single work is being formed" guideline (that has not
actually been tested in court I think, with the closest likely being
deep linking of http references so far).

Nobody claims that ld.so is a derivative of all the software it ends up
sharing process space with.  Of course, like with GCC stubs and a few
other things, the licensing tends to be chosen in a manner where this
question does not need to get resolved one way or the other.

>  > I also like the availability of applications and libraries that
>  > have not been written specifically for Emacs: replacing much of the
>  > backend behind Emacs Calc with native Jacal would be an interesting
>  > feat, and there are a number of other pathways opening up.
>
> Indeed interesting, although these days I'd be more interested in
> numpy and numba.  But I agree with your word "feat"; I'd be surprised
> when it happened; I suspect Jay Belanger isn't so interested in that.
> Maybe he'll speak up.

Well, realistically speaking calc is strictly in maintenance mode even
though I applaud the addition of some music-related functions recently.
I'm not even sure that I wasn't at one time its maintainer.  At least I
remember prompting David Gillespie monthly for trying to dig up and send
over his personal tree of Calc 2.0 (which would have significanly sped
up a lot of operations, partly due to better simplifications, partly
because of using Emacs 19 floating point numbers for getting much faster
initial approximations for the arbitrary-precision math stuff than the
24-bit integer stuff from Emacs 18 could).

This kind of surgery hasn't happened and Calc 2.0 will likely die along
with Dave's backup disks.  Swapping in something like gmp would likely
require lots fewer surgical cuts, and even swapping in Jacal while
keeping most of the frontend would likely be easier for someone with the
basic skill set "Emacs programmer and maintainer" rather than "numerical
algorithm hacker".

>  > So I do see longterm goals and strategies that could be opened by a
>  > good integration of GUILE as a core part of Emacs.
>
> Sure.  I just don't see them *for me* and *right now*.

Oh, *right now* never was much of a GNU priority.  It has always been
for the long haul.

>  > The most relevant obstacle _will_ be to overcome the "don't mess
>  > with my project" stance in both Emacs and GUILE from people that
>  > have come to like the respective culture and environment of the
>  > isolated projects.
>
> Most likely. :-( But if it really looks like a great idea, RMS will
> come and threaten to knock heads, and that will be all it takes for
> all to turn sweetness and light (you and me excepted, of course ;-).

Shrug.  The whetstone opposes the knife.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:

 > > [David pointed this out elsewhere, but I want to emphasize: GUILE
 > > distributed as a part of Emacs is GPL, not LGPL, and the protection of
 > > the GPL extends to all code in the Emacs process space.  GuileEmacs
 > > can be distributed only because the LGPL allows promotion to GPL.]
 > 
 > "process space" is not a legal concept.

No, but it is a rule of thumb used by a lot of lawyers, and in the
U.S. at least, judges.  It comes close for many purposes.

 > Nobody claims that ld.so is a derivative of all the software it
 > ends up sharing process space with.

I didn't write that /lib/ld-linux.so.2 is a derivative.  The object in
memory is a derivative (but that doesn't matter for the GPL or LGPL
unless you dump the memory to a file).  What it does is show that
somebody intended to create a derivative of the various files loaded
into memory, and that is relevant to judging whether certain files
scattered around the Internet should be considered to be a Work
covered by the GPL.

It's hard to prove that this was the intention *of the author*.  The
point of the "I-am-GPL" API is precisely that it forces an author to
make her intention public.  The question is whether Richard will side
with the omelette (Emacs's GPL) or with the eggs (the Guile-specific
modules that don't call the "I-am-GPL" API).  My bet is on the
omelette, but we'll see when it actually comes down to that.  (Richard
is usually firm about not making up his mind until there's actually a
problem to be solved.)

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Taylan Ulrich Bayırlı 	/Kammer writes:
>  > I've heard bad things about both defstruct and EIEIO for different
>  > reasons.  The fact that most Elisp code is shy of using even defstruct
>  > should tell us something.
>
> It does.  It tells us that RMS doesn't like abstract data types.
> AFAICT there's little inherent problem with defstruct from cl-macs (or
> cl-lib, I forget which), it's just a matter of style preference
> (originally rooted in the claim that cl.el was just syntactic sugar so
> it was a waste of pure space on small machines to require it).

OK, I didn't know of any past ordeals regarding defstruct.

>  > >> an FFI
>  > >
>  > > We're getting modules separately.
>  > 
>  > I'm not sure if that's comparable to an FFI.
>
> Does Guile's FFI refuse to load code if it doesn't call the I-swear-
> I'm-GPLed function?  That's another requirement for an FFI/module
> system in Emacs, at least for the present.

Would that really be a blocker if the feature just appears naturally as
part of the Guile integration?

(The answer to your question is no.  Guile is LGPL anyway.)

>  > I'll now leave this branch of the discussion too because it's
>  > probably not going to head in a good direction.  I would much
>  > rather do something constructive than try counter people's panicky
>  > reactions to Guile, whatever reason they have.
>
> I'm not panicking.  GuileEmacs has zero attraction for me *personally*
> because on the one hand its advocates admit it still needs work.  On
> the other none of its claimed advantages excite *me* one bit.

I didn't have you in mind when I said that. :-)

> If we can really rewrite all of Emacs with the exception of device
> drivers in Guile, then I'd definitely be excited (I think that's what
> Tom Tromey is talking about).  But I don't think that effort is likely
> to succeed in providing an efficient Emacs at all soon, although it
> might provide an efficient Emacs Lisp.  So we would end up with three
> implementation languages required to understand important (generic)
> components of Emacs.  Specifically I suppose that redisplay is likely
> to remain in C for a long time.  I don't think that's a win,
> especially with the e^i mental twist required when moving between
> Scheme code and Lisp code.

I don't think there's any clear plans to rewrite arbitrary or all parts
of Emacs's C code in Guile-Scheme.  The way I see it, the merge entails
broadly 1. letting Guile do all the things it can do better like
compiling and running Lisp (regardless of whether it's Guile's C or
Scheme code that does so), 2. having the Emacs-specific data types and
subroutines in Emacs's C core become libguile data types and libguile
procedures, without much change to the body of C code implementing them.

AFAIUI, that's precisely what GuileEmacs currently does, and the basic
strategy seems to work very well.  (It really works as a drop-in
replacement, just with bugs.)  For instance I can call the Elisp subr
`buffer-substring' not only from Elisp code that gets compiled by Guile,
I can also import the module `(elisp-functions)' in Scheme (or any other
language running on Guile) and have the subr normally found in
(symbol-function 'buffer-substring) appear within the Scheme variable
`buffer-substring' instead, with Scheme type 'procedure' and all the
related privileges of that type.

This offers a very nice integration on one hand (only Elisp strings are
a separate type from Guile strings), and requires no substantial change
to the C code behind many features on the other hand.

Actually rewriting parts of that C code in Scheme might be a future
endeavor, but I don't think it's a priority.  It might be a slow and
gradual process for the future, for those parts of the C code for which
it makes sense to rewrite in a higher level language.

> It doesn't bother me if somebody else wants to do the work, though.

Great. :-)


I do understand the worry about the additional language that might need
to be learned by some Emacs maintainers by the way.  That's one of the
actually sensible criticisms I've heard against GuileEmacs.  There's
probably no real solution to that, but I think it will be the correct
trade-off.  Scheme is more similar to than different from Elisp, and
Guile is gaining a huge amount of importance via Guix, and has also
otherwise become a really great general-purpose programming language.
It will most likely play a very important role in the future of GNU, so
I would say GNU hackers should look forward to learning it.  With Guix
you have Lisp in your initramfs, Lisp running your init system and
service management, Lisp driving your package manager, your package
recipes are Lisp expressions evaluating to Lisp objects, and so on and
so forth.  That Lisp is normally Guile Scheme, but if Emacs runs
seamlessly as part of Guile...

This is not a pipe dream.  Guix is working on the 0.9 release with very
steady progress, and GuileEmacs just needs some love.

Taylan

Re: Emacs rewrite in a maintainable language

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>
>> Taylan Ulrich Bayırlı 	/Kammer writes:
>>  > I've heard bad things about both defstruct and EIEIO for different
>>  > reasons.  The fact that most Elisp code is shy of using even defstruct
>>  > should tell us something.
>>
>> It does.  It tells us that RMS doesn't like abstract data types.
>> AFAICT there's little inherent problem with defstruct from cl-macs (or
>> cl-lib, I forget which), it's just a matter of style preference
>> (originally rooted in the claim that cl.el was just syntactic sugar so
>> it was a waste of pure space on small machines to require it).
>
> OK, I didn't know of any past ordeals regarding defstruct.
>
>>  > >> an FFI
>>  > >
>>  > > We're getting modules separately.
>>  > 
>>  > I'm not sure if that's comparable to an FFI.
>>
>> Does Guile's FFI refuse to load code if it doesn't call the I-swear-
>> I'm-GPLed function?  That's another requirement for an FFI/module
>> system in Emacs, at least for the present.
>
> Would that really be a blocker if the feature just appears naturally as
> part of the Guile integration?
>
> (The answer to your question is no.  Guile is LGPL anyway.)

A Guile linked into Emacs would of course inherit the GPL of Emacs.  I
don't see its standard dll-loader magically making any loaded module
become part of "software as a whole" with regard to licensing unless it
specifically called Emacs functions.  But as long as the module would
also work with a standalone GUILE interpreter, I don't see the GPL
gaining traction on it.

But of course a tight coupling between Emacs and GUILE _will_ make it
more attractive to write modules with components tieing into editing.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Tom Tromey]

Stephen> If we can really rewrite all of Emacs with the exception of device
Stephen> drivers in Guile, then I'd definitely be excited (I think that's what
Stephen> Tom Tromey is talking about).

No, I'm interested in writing Emacs in Emacs Lisp and evolving Emacs
Lisp in place; and against the Guile/Emacs thing, for basically the same
reasons as Daniel.

Tom

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Tom Tromey <tom@tromey.com> writes:

> Stephen> If we can really rewrite all of Emacs with the exception of device
> Stephen> drivers in Guile, then I'd definitely be excited (I think that's what
> Stephen> Tom Tromey is talking about).
>
> No, I'm interested in writing Emacs in Emacs Lisp and evolving Emacs
> Lisp in place; and against the Guile/Emacs thing, for basically the
> same reasons as Daniel.

Well, I may have limited comprehension, but the reasons I remember from
Daniel were mostly in the "Do not want" and "we could do something
similar if we worked hard enough" category.

Basically, fear of merging and the unknown.  Since the whole point was
to _stop_ working on compiler/optimizer/language details that are more
encompassingly solved in a system with a less heckish history as Emacs
Lisp, I am not convinced.

The main problem I see is that we have two communities who are quite
unenthusiastic about letting the respective other project shape vital
aspects of their own project.

Which is a problem.  I think that both projects would move considerably
out of their specialist corner if this worked out, gaining wider appeal
and acceptance as "serious" software useful for more than special
purposes.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Tom Tromey]

>> No, I'm interested in writing Emacs in Emacs Lisp and evolving Emacs
>> Lisp in place; and against the Guile/Emacs thing, for basically the
>> same reasons as Daniel.

David> Basically, fear of merging and the unknown.

I wouldn't say that describes my views very well.
Maybe you could re-read what Daniel wrote.

Tom

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 10/18/2015 12:35 PM, Taylan Ulrich Bayırlı/Kammer wrote:
>
>> Guile already supports finalizers (and much, much more) so you
>> wouldn't have needed to implement it from scratch for Elisp. :-)
>
> I'm sure we'll have to extend something sometime.

Sure.  But general programming facilities are not likely to be it.  in
spite of Taylan Ulrich's insistence, I am pretty sure that most of the
things in need of cross-community work are in the encoding area.  I'm
also pretty sure that the basic buffer mechanism (basically a string
with an insertion point) would warrant a more native reflection in GUILE
data structures.

>> record types (like an improved defstruct)
>
> We have EIEIO. And defstruct is fine for most use cases. Over in the
> modules discussion, we're talking about letting C modules define
> first-class types in elisp. That'd be neat, and defstruct could use the
> same machinery instead of a type symbol.
>
>> CLOS-like OOP
>
> EIEIO.
>
>> an FFI
>
> We're getting modules separately.
>
>> composable continuations
>
> We have them already in the form of generators. I don't want
> unrestricted call/cc.
>
>> a module system
>
> I don't want Guile's.

The underlying principle of GUILE's module system is to separate symbols
as globally interned strings from the associated variables containing
values.  Lisp symbols have 4 elements: print name, value cell, function
cell, property list.  GUILE symbols have the print name, everything else
is established separately.  That allows for designing arbitrary module
systems on top, so it would be feasible to give Elisp its own system.

>> hygienic macros
>
> Unhygienic macros are useful.

Nobody will take them from you.

> I don't want Guile's opinionated macros.  Besides, this point is
> out-of-date if Guile supports elisp, since existing elisp macros are
> unhygienic.
>
>> multiple-value returns
>
> Returning lists works fine, and everyone I know finds CL's multiple
> return feature confusing. We can live without it easily, since most
> other languages have managed to live without it.
>
> Besides: we support multiple return values _today_ in the form of
> locatives: we call them gv-ref and gv-deref.

Well, I'd want to have
<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17474> adopted in
order to make the C API more consistent.  I'd also want a single static
value for the empty string "" (regardless of encoding and r/w state
which does not make a difference when there are 0 characters).  And so
on.  Basically, getting any internal changes accepted or even discussed
as a GUILE outsider is a rather upstream battle.  Of course it doesn't
particularly help if you are banned from the developer list...

I digress.  Apart from the above-mentioned lack of streamlining the
no-return-value concepts of the C and Lisp APIs, the multiple-values
concept of Scheme is reasonably clear.  The programming constructs
around it are awkward, but there are by now destructuring forms of "let"
that are actually being reasonably nice to use when programming.

>> and threads
>
> Guile by itself sure as hell won't give us threads. You'll have to
> solve all the synchronization problems that adding threads will
> inflict on the Emacs core. The existing threads branch goes a long way
> toward doing that.

Sure, but using a thread-supporting underlying programming language
means that the Emacs code dealing with multithreading will only have to
concern itself with the Emacs-specific synchronization issues rather
than mixing this up with the interpreter-specific problems.

> I really don't want to shoehorn Guile into Emacs. It's the wrong
> runtime for the wrong language being advocated for the wrong reasons.
>
> I wouldn't mind taking _pieces_ of Guile --- in particular, maybe the
> JIT and GC

GUILE 2 has no JIT compiler and uses the Boehm GC (and still has not
fixed/worked-around/reported the finalizer problem in
<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19883>) so that's a
peculiar choice of things to take.

> --- and replacing the existing Emacs implementations of these runtime
> facilities, but I don't want Emacs linking against libguile.so, I
> don't want users to have to install a separate package, and I don't
> want our platform support to be limited by whatever Guile provides.
> Basically, I don't want our future in Guile's hands.

Well, in a nutshell you don't want the GUILE and Emacs communities to
join forces.  But that's what the strategic goal is.  Your main argument
is "we could make do without GUILE".  But that's not actually the
question.  Of course we could.

But basically if GUILE is not wanted as an extension language for Emacs,
GUILE's intended role as the extension language for GNU is a failure.

I think that if the Emacs and GUILE communities can get over their
respective "don't you mess with my project" stance and fears that GNU
overall would have something to gain.  This will require changes in
Emacs developer focus and attitudes, and it will require changes in
GUILE developer focus and attitudes, and currently there is a sizable
number of developers in both camps who would prefer the other camp to
drop off the surface of the Earth in order not to complicate matters.

I'm not convinced that this is the best way forward.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Przemysław Wojnowski]

W dniu 19.10.2015 o 10:46, David Kastrup pisze:
> I think that if the Emacs and GUILE communities can get over their
> respective "don't you mess with my project" stance and fears that GNU
> overall would have something to gain.  This will require changes in
> Emacs developer focus and attitudes, and it will require changes in
> GUILE developer focus and attitudes[...]
IMHO this is very important, in long run, to every community that wants to
survive.
It's like in the game of go: big groups (usually) win. If you've got many
groups, some of them are dead ant the rest is weak. ;-)

Re: Emacs rewrite in a maintainable language

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Guilemacs has other disadvantages:
  > currently, Emacs supports _only_ elisp as a first-class extension
  > language. Guilemacs would invite people to write Emacs extensions in
  > Scheme, JavaScript, and whatever else Guile ends up supporting, which
  > will create fragmentation. A unified elisp ecosystem is a strength.

Supporting various languages such that they can call each other easily
is a very nice feature, in my view.  In fact, that's why I think
incorporating Guile into Emacs is desirable.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs rewrite in a maintainable language

[Alexis]

Richard Stallman <rms@gnu.org> writes:

> Supporting various languages such that they can call each other 
> easily is a very nice feature, in my view.

Well, to a certain extent, this functionality is already somewhat 
provided by Org:

http://orgmode.org/manual/Evaluating-code-blocks.html#Evaluating-code-blocks

and the list of supported languages is quite impressive:

http://orgmode.org/manual/Languages.html#Languages


Alexis.

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

Daniel Colascione <dancol@dancol.org> writes:

> On 10/18/2015 10:52 AM, John Wiegley wrote:
>>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>>> When we need to use Guile APIs from Emacs. That's one of the advantages of
>>> using Guile: that it implements a lot of functionality which it would be
>>> good to have in Emacs.
>> 
>> Total agreement! I've been wanting a Guile-based Emacs since 1998, if for no
>> other reason than that I could have used call/cc in Eshell, instead of the
>> horrendous CPS hack that exists there now.
>
> You have generators in elisp already. Wanting to use one language is,
> IMHO, a poor choice for wanting to completely swap out a language. I am
> opposed to Guilemacs, not only on technical grounds, but also because
> elisp is essential to Emacs (and not just an optional extension system),
> and I want its implementation to live alongside the rest of the Emacs
> core code.

GuileEmacs would improve Elisp.  See first two points in Overview here:
http://www.emacswiki.org/emacs/GuileEmacs

Taylan

Re: Emacs rewrite in a maintainable language

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]

John Wiegley <johnw@newartisans.com> writes:

> Total agreement! I've been wanting a Guile-based Emacs since 1998, if for no
> other reason than that I could have used call/cc in Eshell, instead of the
> horrendous CPS hack that exists there now.

Out of cusiosity, where is that used?

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Nicolas Petton <nicolas@petton.fr> writes:

> John Wiegley <johnw@newartisans.com> writes:
>> Total agreement! I've been wanting a Guile-based Emacs since 1998, if for
>> no other reason than that I could have used call/cc in Eshell, instead of
>> the horrendous CPS hack that exists there now.

> Out of cusiosity, where is that used?

See `eshell-do-eval'.

John

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> One of the major lessons Emacs development learned since Emacs 20.1 is that
> raw bytes happen as part of text (a.k.a. "strings"), and therefore there's a
> need to support a mixture of these two in the same buffer/string. I think
> that's something Guile should support as well, as that will make it a more
> powerful and flexible extension language, able to deal with a wider range of
> real-life situations.

I'd like to second Eli's recommendation. In real life, encoding and decoding
of bytes to and from characters (codepoints) is never a simple problem. We do
need good flexibility here.

John

Re: Emacs rewrite in a maintainable language

[David Kastrup]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>> One of the major lessons Emacs development learned since Emacs 20.1
>> is that raw bytes happen as part of text (a.k.a. "strings"), and
>> therefore there's a need to support a mixture of these two in the
>> same buffer/string. I think that's something Guile should support as
>> well, as that will make it a more powerful and flexible extension
>> language, able to deal with a wider range of real-life situations.
>
> I'd like to second Eli's recommendation. In real life, encoding and
> decoding of bytes to and from characters (codepoints) is never a
> simple problem. We do need good flexibility here.

Personally I have no problem with an implementation insisting on certain
properties for its internal encoding.  But that implies that "internal
encoding" and "external UTF-8" may diverge when "external UTF-8" does
not exclusively contain valid UTF-8.

Maintaining that distinction for GUILE should not be hard as currently
its internal encoding is either Latin-1 or UCS-32 so it is not like it
currently _has_ an internal UTF-8 for strings even though it has a
number of functions taking UTF-8 input.

However, if "internal encoding" is not the same as "valid UTF-8"
throughout, it means that code called with it has to be able to deal
with the representations for invalid UTF-8.

Currently Emacs uses code points above the Unicode range for
representing non-Unicode characters from different encodings, and it
uses the 2-byte overlong byte sequences for 0-127 to represent raw bytes
128-255.  That's not cast into stone but pretty efficient (I think
Python uses 3-byte surrogate sequences for raw bytes, somewhat worse)
and straightforward as it keeps the basic UTF-8 coding scheme invariants
intact.

Of course, all of this can be done simpler using an UCS-32
representation, but the basic tradeoffs leading to Emacs using a
variable-size multibyte representation are still valid in my opinion.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Stephen J. Turnbull]

David Kastrup writes:

 > Personally I have no problem with an implementation insisting on
 > certain properties for its internal encoding.  But that implies
 > that "internal encoding" and "external UTF-8" may diverge when
 > "external UTF-8" does not exclusively contain valid UTF-8.

Then the external data shouldn't be called "UTF-8" in discussions like
this one.  The problem of data that is not valid for the presumed
encoding is not limited to UTF-8, Unicode, or even to text.  It just
happens that we have good solutions (not limited to ritual suicide)
for the text stream case.

Also, we should remember that Unicode is a wire protocol.  It's very
useful to adapt the formats defined by Unicode for constructing and
parsing internal and external data -- that can be very efficient.  But
we also need to have a strict-conformance option for I/O that is
declared to be Unicode, and that probably be the default.

 > However, if "internal encoding" is not the same as "valid UTF-8"
 > throughout, it means that code called with it has to be able to
 > deal with the representations for invalid UTF-8.

Emacs certainly can deal, since it has a 'binary' encoding and can
represent that internally.  But that's awfully inconvenient.
Something like Emacs's current implementation, Markus Kuhn's UTF-8b,
or Python's PEP 383 is really required for Emacs implementations.
(Does anybody remember that awful mail format of Win2k beta's version
of Outlook Express, where the HTML tags were encoded in ASCII and the
element content in little-endian UTF-16?)

 > [Emacs's internal text representation is] not cast into stone but
 > pretty efficient (I think Python uses 3-byte surrogate sequences
 > for raw bytes, somewhat worse)

No.  Python uses a wide-char representation.  In Python 2, it's 2
bytes on most non-glibc platforms, and 4 bytes on glibc.  In Python 3
with PEP 393 support, valid ISO-8859-1 text (even if decoded from
another external encoding) is represented in one byte, valid BMP text
(optionally with support for invalid "rawbytes", internally encoded as
lone trailing surrogates) in two bytes, and text containing characters
from the astral planes in four bytes (again with optional support for
invalid rawbytes).

 > and straightforward as it keeps the basic UTF-8 coding scheme
 > invariants intact.
 > 
 > Of course, all of this can be done simpler using an UCS-32
 > representation, but the basic tradeoffs leading to Emacs using a
 > variable-size multibyte representation are still valid in my
 > opinion.

Seems reasonable to me.  So far Python with PEP 393 has been pretty
successful, but since emoticons live in the astral planes, I suspect
it may not be the best representation for the web and phones -- one
smiley in ASCII text will quadruple the needed string storage.  I
don't see a good reason to change Emacs's representation at this
point.

Re: Emacs rewrite in a maintainable language

[Gian Uberto Lauri]

John Wiegley writes:
 > >>>>> Eli Zaretskii <eliz@gnu.org> writes:
 > 
 > > One of the major lessons Emacs development learned since Emacs
 > > 20.1 is that raw bytes happen as part of text (a.k.a. "strings"),
 > > and therefore there's a need to support a mixture of these two in
 > > the same buffer/string. I think that's something Guile should
 > > support as well, as that will make it a more powerful and
 > > flexible extension language, able to deal with a wider range of
 > > real-life situations.
 > 
 > I'd like to second Eli's recommendation. In real life, encoding and
 > decoding of bytes to and from characters (codepoints) is never a
 > simple problem. We do need good flexibility here.

My very humble opinion is that it Guile should acquire such a string
type it could gain a big edge over more used programming languages.

-- 
 /\           ___                                    Ubuntu: ancient
/___/\_|_|\_|__|___Gian Uberto Lauri_____               African word
  //--\| | \|  |   Integralista GNUslamico            meaning "I can
\/                 coltivatore diretto di software       not install
     già sistemista a tempo (altrui) perso...                Debian"

Warning: gnome-config-daemon considered more dangerous than GOTO

Re: Emacs rewrite in a maintainable language

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Guile strings are fine, thank you.  I’ve used a bunch of
> language/environments and honestly, I’m definitely not ashamed of what
> Guile provides, contrary to what David and you seem to imply.

It's quite irrelevant whether somebody is proud or ashamed rightfully or
not of what is there.  It does not fit the bill for Emacs and can't be
coaxed into fitting the bill without substantial changes.  Its character
range is hard-limited at the end of Unicode.  It has no representation
for non-utf8 code bytes.  It has no additional character codes available
for those (let alone the "overlong" 2-byte sequences that Emacs employs
for representing raw bytes 128-255 transparently) It cannot faithfully
represent a string where only parts are proper utf-8 and the rest is to
be reproduced exactly even when doing string operations on the utf-8
parts.  Its input- and output encodings are not independent from the
locale.

The underlying string handling libraries are not even _prepared_ to deal
with such requirements.  Nor are they prepared for passing such strings
transparently through and working with them.  In addition, Guile does
not actually have utf-8 strings but represents strings as either 8-bit
or 32-bit fixed-width entities.  Which makes indexing efficient but
incurs conversions even if you are just working with utf-8 like Emacs
does.

In spite of only working with either UCS-8 or UCS-32 in its strings,
many of the primitives of GUILE have a preference for utf-8: you cannot
even pass UCS-32 into GUILE or get it out easily in spite of it being
its internal format.

But when GUILE is used as an extension language rather than as a sole
implementation language, you'll need to pass strings in and out of it
constantly.  And the only format where strings will get passed without
conversion is Latin-1.  But when telling GUILE that everything is
Latin-1, you have only a limited amount of reasonably working string
operations at your disposal and you won't get to see character codes
larger than 255.

All this most definitely is nothing that could not be fixed.  But doing
that is comparatively harder when all attempts to do so or to bring the
problems to attention are met with defiance.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

David Kastrup <dak@gnu.org> writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Guile strings are fine, thank you.  I’ve used a bunch of
>> language/environments and honestly, I’m definitely not ashamed of what
>> Guile provides, contrary to what David and you seem to imply.
>
> It's quite irrelevant whether somebody is proud or ashamed rightfully or
> not of what is there.  It does not fit the bill for Emacs and can't be
> coaxed into fitting the bill without substantial changes.  Its character
> range is hard-limited at the end of Unicode.  It has no representation
> for non-utf8 code bytes.  It has no additional character codes available
> for those (let alone the "overlong" 2-byte sequences that Emacs employs
> for representing raw bytes 128-255 transparently) It cannot faithfully
> represent a string where only parts are proper utf-8 and the rest is to
> be reproduced exactly even when doing string operations on the utf-8
> parts.  Its input- and output encodings are not independent from the
> locale.
>
> The underlying string handling libraries are not even _prepared_ to deal
> with such requirements.  Nor are they prepared for passing such strings
> transparently through and working with them.  In addition, Guile does
> not actually have utf-8 strings but represents strings as either 8-bit
> or 32-bit fixed-width entities.  Which makes indexing efficient but
> incurs conversions even if you are just working with utf-8 like Emacs
> does.
>
> In spite of only working with either UCS-8 or UCS-32 in its strings,
> many of the primitives of GUILE have a preference for utf-8: you cannot
> even pass UCS-32 into GUILE or get it out easily in spite of it being
> its internal format.
>
> But when GUILE is used as an extension language rather than as a sole
> implementation language, you'll need to pass strings in and out of it
> constantly.  And the only format where strings will get passed without
> conversion is Latin-1.  But when telling GUILE that everything is
> Latin-1, you have only a limited amount of reasonably working string
> operations at your disposal and you won't get to see character codes
> larger than 255.
>
> All this most definitely is nothing that could not be fixed.  But doing
> that is comparatively harder when all attempts to do so or to bring the
> problems to attention are met with defiance.

I haven't followed the whole discussion but I'd like to remind that
GuileEmacs already runs currently, and from what I know none of the
problems it faces are related to string handling.  It just lets Elisp
strings be a separate data type.

I guess the discussion was about rewriting bigger parts of Emacs in
Guile-Scheme, which might be one possible way to go forward but not
necessary.

Most of the benefits in e.g. http://www.emacswiki.org/emacs/GuileEmacs
apply without having to solve the string problem.  Specifically, from
the overview section, the first two points and half of the third should
apply I think.  The third point only halfway because some Guile APIs
might expect to receive Guile strings, so using them from Elisp code
would be problematic, but Guile APIs which don't take strings should be
fine.  Anyway, the first two points are already pretty attractive,
especially the second.

Whatever language Emacs is written in, the native string library of the
language will be insufficient, so meh.

Taylan

Re: Emacs rewrite in a maintainable language

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> David Kastrup <dak@gnu.org> writes:
>
> Most of the benefits in e.g. http://www.emacswiki.org/emacs/GuileEmacs
> apply without having to solve the string problem.  Specifically, from
> the overview section, the first two points and half of the third
> should apply I think.  The third point only halfway because some Guile
> APIs might expect to receive Guile strings, so using them from Elisp
> code would be problematic, but Guile APIs which don't take strings
> should be fine.  Anyway, the first two points are already pretty
> attractive, especially the second.
>
> Whatever language Emacs is written in, the native string library of
> the language will be insufficient, so meh.

Shrug.  C strings are "insufficient" but they don't get in the way.  The
problem with GUILE 2 strings is not that they are insufficient, the
problem is that they are incompatible while not exposing the mechanisms
that would allow working around that incompatibility.

The way around that is to scrap _any_ string handling in GUILE and
instead map Emacs strings to byte vectors (and the related ports).
Which was actually what was recommended to me to do for LilyPond,
leading to me firing off a sequence of bug reports (I think the last one
was closed recently but the respective fix is not yet in any released
version of GUILE): this code has apparently not seen much exercise.  So
even letting GUILE just step aside with regard to encoding/decoding is
something that's not yet to be called reliable and well-tested.

But that at least improves over time since it cannot be called anything
but a bug.  The coding implementation issues, in contrast, are not
considered something where GUILE is in need of change or redesign and/or
where any work would be scheduled or advertised as necessary.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

David Kastrup <dak@gnu.org> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> Most of the benefits in e.g. http://www.emacswiki.org/emacs/GuileEmacs
>> apply without having to solve the string problem.  Specifically, from
>> the overview section, the first two points and half of the third
>> should apply I think.  The third point only halfway because some Guile
>> APIs might expect to receive Guile strings, so using them from Elisp
>> code would be problematic, but Guile APIs which don't take strings
>> should be fine.  Anyway, the first two points are already pretty
>> attractive, especially the second.
>>
>> Whatever language Emacs is written in, the native string library of
>> the language will be insufficient, so meh.
>
> Shrug.  C strings are "insufficient" but they don't get in the way.  The
> problem with GUILE 2 strings is not that they are insufficient, the
> problem is that they are incompatible while not exposing the mechanisms
> that would allow working around that incompatibility.
>
> The way around that is to scrap _any_ string handling in GUILE and
> instead map Emacs strings to byte vectors (and the related ports).
> Which was actually what was recommended to me to do for LilyPond,
> leading to me firing off a sequence of bug reports (I think the last one
> was closed recently but the respective fix is not yet in any released
> version of GUILE): this code has apparently not seen much exercise.  So
> even letting GUILE just step aside with regard to encoding/decoding is
> something that's not yet to be called reliable and well-tested.
>
> But that at least improves over time since it cannot be called anything
> but a bug.  The coding implementation issues, in contrast, are not
> considered something where GUILE is in need of change or redesign and/or
> where any work would be scheduled or advertised as necessary.

Maybe one could say GuileEmacs does indeed scrap any string handling in
Guile, since it simply uses the existing Elisp string data type.

I guess the problem for LilyPond was that it used to use the Guile 1
string data type which now doesn't exist anymore?  Thankfully no such
problem applies to GuileEmacs which can just stick to Elisp strings.


(I think I need to take back the "none of the current problems are
related to strings" though, since IIRC Elisp strings can't currently be
serialized into Guile's byte-code to be written onto the disk.  Not sure
if that's relevant to the topic.)

Taylan

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: ludo@gnu.org (Ludovic Courtès),  Eli Zaretskii
>  <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sat, 17 Oct 2015 18:25:21 +0200
> 
> I guess the discussion was about rewriting bigger parts of Emacs in
> Guile-Scheme

Yes, this is the issue; see the Subject.

> Most of the benefits in e.g. http://www.emacswiki.org/emacs/GuileEmacs
> apply without having to solve the string problem.

But this discussion is about replacing most or all of the Emacs C core
with Guile.  If we want to avoid rewriting, say, coding.c completely
from scratch, we need Guile to support the same operations as the ones
exposed by coding.c to Lisp applications.  Currently, Guile doesn't.

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Cc: ludo@gnu.org (Ludovic Courtès),  Eli Zaretskii
>>  <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Sat, 17 Oct 2015 18:25:21 +0200
>> 
>> I guess the discussion was about rewriting bigger parts of Emacs in
>> Guile-Scheme
>
> Yes, this is the issue; see the Subject.
>
>> Most of the benefits in e.g. http://www.emacswiki.org/emacs/GuileEmacs
>> apply without having to solve the string problem.
>
> But this discussion is about replacing most or all of the Emacs C core
> with Guile.  If we want to avoid rewriting, say, coding.c completely
> from scratch, we need Guile to support the same operations as the ones
> exposed by coding.c to Lisp applications.  Currently, Guile doesn't.

I really don't think rewriting *all* of Emacs's C code in another
language is a realistic goal at all.  So unrealistic that I couldn't
believe this gigantic thread is still discussing that. :-)

Rewriting select parts of it in Elisp and/or Scheme seems much more
realistic.  AFAIUI, GuileEmacs already does that with Scheme to some
degree, since the Guile compiler is written in Scheme.

Taylan

Re: Emacs rewrite in a maintainable language

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> I really don't think rewriting *all* of Emacs's C code in another
> language is a realistic goal at all.  So unrealistic that I couldn't
> believe this gigantic thread is still discussing that. :-)
>
> Rewriting select parts of it in Elisp and/or Scheme seems much more
> realistic.  AFAIUI, GuileEmacs already does that with Scheme to some
> degree, since the Guile compiler is written in Scheme.

The Elisp byte compiler is written in Elisp if I'm not mistaken.  So
that's not really in itself a difference.

The GUILE compiler is layered into separate stages, however, so each
particular stage is more amenable to understanding, changing and
adapting to other circumstances.

Even if pretty much all of it essentially appears to be a one-man job
without communicating with anybody else.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: dak@gnu.org,  ludo@gnu.org,  emacs-devel@gnu.org
> Date: Sat, 17 Oct 2015 19:03:18 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But this discussion is about replacing most or all of the Emacs C core
> > with Guile.  If we want to avoid rewriting, say, coding.c completely
> > from scratch, we need Guile to support the same operations as the ones
> > exposed by coding.c to Lisp applications.  Currently, Guile doesn't.
> 
> I really don't think rewriting *all* of Emacs's C code in another
> language is a realistic goal at all.

But some do, and they started this sub-thread.

> Rewriting select parts of it in Elisp and/or Scheme seems much more
> realistic.

But then it's prudent to ask do the gains justify the effort.

Re: Emacs rewrite in a maintainable language

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > > Rewriting select parts of it in Elisp and/or Scheme seems much more
 > > realistic.
 > 
 > But then it's prudent to ask do the gains justify the effort.

Rewriting "select" parts in Elisp clearly do, at least they have up to
now.  People regularly rewrite C features in Elisp when they run into
the inflexiblity of C (I suppose my estimate is biased since it's
based on [S]XEmacs practice, not Emacs, but [S]XEmacs practice is
somewhat relevant here, and I imagine Emacs practice is not so different).

Rewriting "select" parts in Scheme makes very little sense IMO (unless
you're already running GuileEmacs).  The cost of moving large parts of
core to Scheme not only includes the rewrites themselves (which I
suspect mostly have *not* yet been done in GuileEmacs), but the not-
to-be-underestimated costs of integrating a large branch, as well.
And of course it has a strong (partial) effect of *increasing*
maintenance cost, because the maintainers now have to be familiar with
3 different languages.

Re: Emacs rewrite in a maintainable language

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> I haven't followed the whole discussion but I'd like to remind that
> GuileEmacs already runs currently, and from what I know none of the
> problems it faces are related to string handling.  It just lets Elisp
> strings be a separate data type.
>
> I guess the discussion was about rewriting bigger parts of Emacs in
> Guile-Scheme, which might be one possible way to go forward but not
> necessary.
>
> Most of the benefits in e.g. http://www.emacswiki.org/emacs/GuileEmacs
> apply without having to solve the string problem.  Specifically, from
> the overview section, the first two points and half of the third should
> apply I think.

The first two points concern efficiency.  Emacs is _all_ about working
with strings, and the last report of GuileEmacs stated something like
GuileEmacs being about an order of magnitude slower specifically because
of having to special-case Emacs strings and buffers.

Emacs is first and foremost an editor.  Substantial increases of
efficiency in most general programming areas at the cost of abysmal
performance in text processing are not going to make the race.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

David Kastrup <dak@gnu.org> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> I haven't followed the whole discussion but I'd like to remind that
>> GuileEmacs already runs currently, and from what I know none of the
>> problems it faces are related to string handling.  It just lets Elisp
>> strings be a separate data type.
>>
>> I guess the discussion was about rewriting bigger parts of Emacs in
>> Guile-Scheme, which might be one possible way to go forward but not
>> necessary.
>>
>> Most of the benefits in e.g. http://www.emacswiki.org/emacs/GuileEmacs
>> apply without having to solve the string problem.  Specifically, from
>> the overview section, the first two points and half of the third should
>> apply I think.
>
> The first two points concern efficiency.  Emacs is _all_ about working
> with strings, and the last report of GuileEmacs stated something like
> GuileEmacs being about an order of magnitude slower specifically because
> of having to special-case Emacs strings and buffers.
>
> Emacs is first and foremost an editor.  Substantial increases of
> efficiency in most general programming areas at the cost of abysmal
> performance in text processing are not going to make the race.

I think you're misremembering.

It might be that you have strings in mind because not being able to
serialize Elisp strings means having to compile everything at each
startup, making startup slow.  And maybe you have buffers in mind
because buffer-local variables and other weird binding semantics of
Elisp need special handling.

Neither is related to the run-time performance of string handling, of
course.

Hmm, what would be a good benchmark of Elisp string handling that won't
be affected by the performance of dynamic binding of variables or other
such things?  I'd like to just test it instead of hypothesize.

Taylan

Re: Emacs rewrite in a maintainable language

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> David Kastrup <dak@gnu.org> writes:
>
>> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>>
>>> I haven't followed the whole discussion but I'd like to remind that
>>> GuileEmacs already runs currently, and from what I know none of the
>>> problems it faces are related to string handling.  It just lets Elisp
>>> strings be a separate data type.
>>>
>>> I guess the discussion was about rewriting bigger parts of Emacs in
>>> Guile-Scheme, which might be one possible way to go forward but not
>>> necessary.
>>>
>>> Most of the benefits in e.g. http://www.emacswiki.org/emacs/GuileEmacs
>>> apply without having to solve the string problem.  Specifically, from
>>> the overview section, the first two points and half of the third should
>>> apply I think.
>>
>> The first two points concern efficiency.  Emacs is _all_ about working
>> with strings, and the last report of GuileEmacs stated something like
>> GuileEmacs being about an order of magnitude slower specifically because
>> of having to special-case Emacs strings and buffers.
>>
>> Emacs is first and foremost an editor.  Substantial increases of
>> efficiency in most general programming areas at the cost of abysmal
>> performance in text processing are not going to make the race.
>
> I think you're misremembering.

So performance is comparable?

> It might be that you have strings in mind because not being able to
> serialize Elisp strings means having to compile everything at each
> startup, making startup slow.  And maybe you have buffers in mind
> because buffer-local variables and other weird binding semantics of
> Elisp need special handling.
>
> Neither is related to the run-time performance of string handling, of
> course.
>
> Hmm, what would be a good benchmark of Elisp string handling that
> won't be affected by the performance of dynamic binding of variables
> or other such things?  I'd like to just test it instead of
> hypothesize.

Most relevant candidates are likely file load/save, string display and
stuff like regexp searches through buffers.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

David Kastrup <dak@gnu.org> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>>>
>>>> I haven't followed the whole discussion but I'd like to remind that
>>>> GuileEmacs already runs currently, and from what I know none of the
>>>> problems it faces are related to string handling.  It just lets Elisp
>>>> strings be a separate data type.
>>>>
>>>> I guess the discussion was about rewriting bigger parts of Emacs in
>>>> Guile-Scheme, which might be one possible way to go forward but not
>>>> necessary.
>>>>
>>>> Most of the benefits in e.g. http://www.emacswiki.org/emacs/GuileEmacs
>>>> apply without having to solve the string problem.  Specifically, from
>>>> the overview section, the first two points and half of the third should
>>>> apply I think.
>>>
>>> The first two points concern efficiency.  Emacs is _all_ about working
>>> with strings, and the last report of GuileEmacs stated something like
>>> GuileEmacs being about an order of magnitude slower specifically because
>>> of having to special-case Emacs strings and buffers.
>>>
>>> Emacs is first and foremost an editor.  Substantial increases of
>>> efficiency in most general programming areas at the cost of abysmal
>>> performance in text processing are not going to make the race.
>>
>> I think you're misremembering.
>
> So performance is comparable?

Regarding Elisp string munging, probably, since the same C code as
before applies.

>> It might be that you have strings in mind because not being able to
>> serialize Elisp strings means having to compile everything at each
>> startup, making startup slow.  And maybe you have buffers in mind
>> because buffer-local variables and other weird binding semantics of
>> Elisp need special handling.
>>
>> Neither is related to the run-time performance of string handling, of
>> course.
>>
>> Hmm, what would be a good benchmark of Elisp string handling that
>> won't be affected by the performance of dynamic binding of variables
>> or other such things?  I'd like to just test it instead of
>> hypothesize.
>
> Most relevant candidates are likely file load/save, string display and
> stuff like regexp searches through buffers.

Opening a 30 MB IRC log file takes the same amount of felt time as on
upstream Emacs.  Saving it too.

(while (re-search-forward "taylan" nil t) (ignore)) in the same file is
pretty much instant on both.

I think there's simply no speed difference in string munging on
GuileEmacs so far.  It really just uses the existing C "subrs".

Taylan

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: Ludovic Courtès <ludo@gnu.org>,  Eli Zaretskii
>  <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sat, 17 Oct 2015 20:34:15 +0200
> 
> > Most relevant candidates are likely file load/save, string display and
> > stuff like regexp searches through buffers.
> 
> Opening a 30 MB IRC log file takes the same amount of felt time as on
> upstream Emacs.  Saving it too.

Can you time both and show the exact timings?

Also, the file should not be plain ASCII (and probably not UTF-8,
either), so that decoding the text is included in the comparison.

> (while (re-search-forward "taylan" nil t) (ignore)) in the same file is
> pretty much instant on both.

Once again, please show exact timings.  "Pretty much instant" could be
0.01 sec vs 0.02 sec, which is still twice slower.

Thanks.

Re: Emacs rewrite in a maintainable language

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Cc: Ludovic Courtès <ludo@gnu.org>,  Eli Zaretskii
>>  <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Sat, 17 Oct 2015 20:34:15 +0200
>> 
>> > Most relevant candidates are likely file load/save, string display and
>> > stuff like regexp searches through buffers.
>> 
>> Opening a 30 MB IRC log file takes the same amount of felt time as on
>> upstream Emacs.  Saving it too.
>
> Can you time both and show the exact timings?
>
> Also, the file should not be plain ASCII (and probably not UTF-8,
> either), so that decoding the text is included in the comparison.
>
>> (while (re-search-forward "taylan" nil t) (ignore)) in the same file is
>> pretty much instant on both.
>
> Once again, please show exact timings.  "Pretty much instant" could be
> 0.01 sec vs 0.02 sec, which is still twice slower.

Timing things mechanically is difficult when GuileEmacs takes so much
longer to boot.

The slow-downs in GuileEmacs are generally of the nature that cause
slowness of orders of magnitude (i.e. very obvious bugs) and I'm already
pretty sure that GuileEmacs simply uses existing Elisp string/buffer
functionality, so I was merely trying to prove that.

I now created a 300 MiB file with data from /dev/urandom and they still
both take the same amount of time to open it, which is about 3 seconds,
and scrolling through the file, using Isearch, etc. all works fine.  I
hope that's convincing enough.

There are currently some grave performance issues in GuileEmacs, but
they aren't related to string/buffer handling, because it simply uses
the existing C code there.

Taylan

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Taylan Ulrich "Bayırlı/Kammer" <taylanbayirli@gmail.com> writes:

> Timing things mechanically is difficult when GuileEmacs takes so much longer
> to boot.

If you write the timing code in Lisp itself, then the timing should only apply
to manipulations within the timed form.  If the same code is then run on
GuileEmacs and Emacs 10 times each for a large sample, with highest and lowest
thrown out and the remaining averaged, it should be a pretty fair comparison.

John

Re: Emacs rewrite in a maintainable language

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It is very important for GUILE to provide the facilities
needed to implement something like Emacs on it.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs rewrite in a maintainable language

[Tom Tromey]

John> If they are amenable, would you be willing to prototype it? If we
John> could see a language description for the subset/alternate, and a
John> sample compilation into C for some of the functions we have in
John> Emacs core (that is, reimplementing them as a proof of concept),
John> that would help sell it.

You don't really need to do any subsetting.  With a decent compiler you
can just write elisp.

This is pretty much what I am trying to do here:

https://github.com/tromey/el-compilador

My test case is a chunk of fns.c rewritten into ordinary Emacs Lisp.
See appended.  The C output is gross, but of course it doesn't matter.
(And the C output is incorrect, which does matter.  But it's a WIP.)

I chose chunks of fns.c to translate because these are close to "pure
lisp" - no difficult native code to handle.

One of the next steps is wiring it up to FFI so that native calls can be
implemented in the compiler.  See

https://github.com/tromey/emacs-ffi

This is even rougher; but the basic idea is to reuse FFI function
declarations in the C back end of the compiler to emit direct native
calls.

Tom


Input:

(defun nthcdr (num list)
  (cl-check-type num integer)
  (let ((i 0))
    (while (and (< i num) list)
      (setq list (cdr list))
      (setq i (1+ i)))
    list))

Output:

#include <config.h>
#include <lisp.h>

int plugin_is_GPL_compatible;

static Lisp_Object Qinteger;
static Lisp_Object Qnum;
static Lisp_Object Qwrong_type_argument;

static Lisp_Object Knthcdr;

DEFUN ("nthcdr", Fnthcdr, Snthcdr, 2, 2,
    0,
    doc: /* nothing?? */)
  (Lisp_Object num, Lisp_Object list)
{
  Lisp_Object G1_5;
  Lisp_Object i_13;
  Lisp_Object G4_18;
  Lisp_Object i_17;
  Lisp_Object num_14;
  Lisp_Object list_15;
  Lisp_Object list_36;
  Lisp_Object i_37;
  Lisp_Object G3_12;

BB_0:
  G1_5 = Fintegerp (num);
  if (!NILP (G1_5)) goto BB_1; else goto BB_2;
BB_1:
  i_13 = make_number (0);
  goto BB_5;
BB_5:
  G4_18 = F< (i_17, num_14);
  if (!NILP (G4_18)) goto BB_8; else goto BB_6;
BB_8:
  if (!NILP (list_15)) goto BB_10; else goto BB_6;
BB_10:
  list_36 = Fcdr (list_15);
  i_37 = F1+ (i_17);
  goto BB_5;
BB_6:
  return list_15;
BB_2:
  G3_12 = Flist (Qinteger, num, Qnum);
  Ffuncall (3, ((Lisp_Object[]) { signal, Qwrong_type_argument, G3_12 }));
}


void
init (void)
{
  Qinteger = intern_c_string ("integer");
  staticpro (&Qinteger);
  Qnum = intern_c_string ("num");
  staticpro (&Qnum);
  Qwrong_type_argument = intern_c_string ("wrong-type-argument");
  staticpro (&Qwrong_type_argument);

  defsubr (&Snthcdr);
  XSETSUBR (Knthcdr, &Snthcdr);
}

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Tom Tromey <tom@tromey.com> writes:

> You don't really need to do any subsetting. With a decent compiler you can
> just write elisp.
>
> This is pretty much what I am trying to do here:
>
> https://github.com/tromey/el-compilador

Ok, you've got the PL side of me rather excited now. :) Are you planning
optimization passes during the transform from elisp to C?

I noticed your files lack a copyright or license. Is this something you would
be willing to assign to the FSF once the bugs are worked out? Even if we don't
use it to rewrite our current C files, it would be an interesting addition to
the build process, if we do decide to cross-compile some Emacs Lisp code over
to C.

For it to be worth the complexity, however, I'd like to see some performance
numbers, or examples of features it gains us that we can't have easily
otherwise (like transparent FFI). If the difference between compiled-to-C and
our current byte-compiler is only 10%, that would be good to know. If the win
is huge, I wonder how long until I can have a compiled Gnus. :)

Other than that, I like it!

John

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: "John Wiegley" <johnw@newartisans.com>
> Date: Sun, 18 Oct 2015 10:40:58 -0700
> 
> I noticed your files lack a copyright or license. Is this something you would
> be willing to assign to the FSF once the bugs are worked out?

Tom has an assignment on file for Emacs (and has contributed quite a
lot of code in the past).

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "John Wiegley" <johnw@newartisans.com>
>> Date: Sun, 18 Oct 2015 10:40:58 -0700
>> 
>> I noticed your files lack a copyright or license. Is this something
>> you would be willing to assign to the FSF once the bugs are worked
>> out?
>
> Tom has an assignment on file for Emacs (and has contributed quite a
> lot of code in the past).

That will make it rather trivial for Tom to contribute this code to
Emacs.  But it doesn't mean that we can just can consider it
"contributed" and grab it.  We still need his explicit ok for that.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Tom Tromey]

John> Ok, you've got the PL side of me rather excited now. :) Are you planning
John> optimization passes during the transform from elisp to C?

There are already a bunch of optimization passes.

For translation to C specifically, I think the main thing of interest is
type inference.  That will enable unboxing optimizations.  Type
inference is working; but I haven't written the unboxing stuff yet.

The reason I think this is particularly interesting is that the C
compiler doesn't know much about the Lisp type system, so this is an
area where the higher-level compiler helps out.

John> I noticed your files lack a copyright or license. Is this
John> something you would be willing to assign to the FSF once the bugs
John> are worked out?

Sure, though Richard's message makes me question whether this is even
wanted.

John> For it to be worth the complexity, however, I'd like to see some
John> performance numbers, or examples of features it gains us that we
John> can't have easily otherwise (like transparent FFI).

The usual benefit of the translation approach is that you can automate
drudgery.  GCPRO is no longer needed, but there are still other things,
like unboxing everything, and remembering to write "NILP" in an "if"; or
even things that Emacs doesn't currently try to do, like "advice
hygiene" -- the idea that the core should only make direct calls to a
small set of functions and dispatch the rest via Ffuncall, which would,
e.g., let people experiment with advising "read" and other things.

I don't have performance numbers, but of course it will be faster, in
the same way that Emacs C code is faster than the corresponding lisp.  I
know the bytecode/LLVM experiment was not; but I think that is because
the Emacs bytecode is a particularly bad format.

Tom

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Tom Tromey <tom@tromey.com> writes:

> I don't have performance numbers, but of course it will be faster, in the
> same way that Emacs C code is faster than the corresponding lisp. I know the
> bytecode/LLVM experiment was not; but I think that is because the Emacs
> bytecode is a particularly bad format.

When it comes to performance, I don't believe in judging by reason. But I'm
eager to see your numbers!

If there are issues that make you think this work is unwanted, please bring
them to me (maybe on IRC), and we can discuss them. I wouldn't want to see
something of potentially great value get dropped.

John

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 12 Oct 2015 21:21:46 +0200
> 
> a Lispy language, easy to learn by those who already know Elisp, but
> which can be translated all the way to something as performant as
> the current C code, while being capable of interacting with it.

I don't think this is possible.  Some slowdown will be inevitable.

Also, if indeed the intent is to rewrite most of Emacs in this "lispy"
language, you'd need some non-Lisp (or at least non-subset) features,
like complex data structures whose members are directly accessible.
Otherwise, the translation will not be straightforward, and you get
bugs (and most probably more slowdown).

Re: Emacs rewrite in a maintainable language

[Przemysław Wojnowski]

W dniu 2015-10-12 21:43, Eli Zaretskii napisał(a):
> I don't think this is possible.  Some slowdown will be inevitable.
IMHO the performance is overestimated.

Look at Java based IDEs - Idea, Eclipse, Netbeans, etc.
All of them have been written in _slow_ Java (at least according
to C standard) and run on _slow_ JVM. But all of them are able to
attract more users that Emacs, which has core in blazing fast C.

The truth is that nobody cares about performance as long as it is
_good enough_. Of course it is important, but, as many other
things, to some level, beyond which further progress in that
direction doesn't matter much and other features become important.
Moreover, the value of performance will change over time (probably
decrease) as can be seen through last decades, because computers
have more resources.

I started the topic with the keyword "maintainable", because IMHO
it's very important factor for long living and growing projects.
In such projects (proprietary and open source) people come and
go with the knowledge and skills they had. So, from project
existence standpoint, it's important to make it easy for the new
developers to acquire the knowledge from the code/docs/whatsleft.

Languages will come and go (sometimes stay longer, though), but
if the code is easy to reason about, test, modify, etc. then it
will be easier to move (maybe even translate) to another language,
find bugs, and find new people who will be able to work with it.

IMHO C lacks basic abstractions that reduce conceptual weight of
any subsystem/library/concept/etc. Interfaces and namespaces do.
Static typing would also help.

Re: Emacs rewrite in a maintainable language

[Gian Uberto Lauri]

Przemysław Wojnowski writes:
 > W dniu 2015-10-12 21:43, Eli Zaretskii napisał(a):
 > > I don't think this is possible.  Some slowdown will be inevitable.
 > IMHO the performance is overestimated.
 > 
 > Look at Java based IDEs - Idea, Eclipse, Netbeans, etc.
 > All of them have been written in _slow_ Java (at least according
 > to C standard) and run on _slow_ JVM. But all of them are able to
 > attract more users that Emacs, which has core in blazing fast C.

Please consider that Eclipse and NetBeans do have big companies
spending money to yell to the world that their tool is a wonderful
tool.

Furthermore if you *need* (read: that's how you earn your wage) to
code in Java, the support you find in Emacs for that language is
limited, JDEE does not even support generics, therefore you need to
use one of such tools.

 > I started the topic with the keyword "maintainable", because IMHO
 > it's very important factor for long living and growing projects.

It is not the language that determines the success, longevity and
maintainability, but how the project is managed. 

 > IMHO C lacks basic abstractions that reduce conceptual weight of
 > any subsystem/library/concept/etc. Interfaces and namespaces do.
 > Static typing would also help.

<humorous but="not this much">
C, IMHO, is a good tool to see who "deserves" to live from coding and
who should choose another career to benefit more the world :>.
</humorous>

-- 
 /\           ___                                    Ubuntu: ancient
/___/\_|_|\_|__|___Gian Uberto Lauri_____               African word
  //--\| | \|  |   Integralista GNUslamico            meaning "I can
\/                 coltivatore diretto di software       not install
     già sistemista a tempo (altrui) perso...                Debian"

Warning: gnome-config-daemon considered more dangerous than GOTO

Re: Emacs rewrite in a maintainable language

[Tassilo Horn]

Przemysław Wojnowski <esperanto@cumego.com> writes:

>> I don't think this is possible.  Some slowdown will be inevitable.
>
> IMHO the performance is overestimated.
>
> Look at Java based IDEs - Idea, Eclipse, Netbeans, etc.  All of them
> have been written in _slow_ Java (at least according to C standard)
> and run on _slow_ JVM.

Yes, they are all a bit logy in general but there's one thing which they
(and any other editors I know of, say Vim, JEdit, gedit) do orders of
magnitude faster than Emacs: syntax highlighting.  Of course, Emacs'
font-lock is better and much more flexible because it's implemented in
Lisp but it's painfully slow anyway.

Bye,
Tassilo

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Tue, 13 Oct 2015 12:19:47 +0200
> 
> Yes, they are all a bit logy in general but there's one thing which they
> (and any other editors I know of, say Vim, JEdit, gedit) do orders of
> magnitude faster than Emacs: syntax highlighting.  Of course, Emacs'
> font-lock is better and much more flexible because it's implemented in
> Lisp but it's painfully slow anyway.

In which major modes is font-lock "painfully slow", in your
experience?  I only see with in C-derived languages.

Re: Emacs rewrite in a maintainable language

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> Yes, they are all a bit logy in general but there's one thing which
>> they (and any other editors I know of, say Vim, JEdit, gedit) do
>> orders of magnitude faster than Emacs: syntax highlighting.  Of
>> course, Emacs' font-lock is better and much more flexible because
>> it's implemented in Lisp but it's painfully slow anyway.
>
> In which major modes is font-lock "painfully slow", in your
> experience?  I only see with in C-derived languages.

Yes, C-derived modes are definitely the worst in that respect but I also
experience this problem with AUCTeX font-latex.el.  There the
fontification rules aren't that complex but there are a simply a lot of
rules.

I sometimes also have slowdowns in elisp-mode and clojure-mode but there
I think it's not the fault of font-lock but rather some minor-mode
hooking into `post-command-hook'.  I likely suspect could be
`aggressive-indent-mode' but I haven't profiled that yet because it
happens only sporadically.

So what I intended to say in the "performance matters or not" debate is
that it probably doesn’t matter too much except for evaluating Elisp.

Bye,
Tassilo

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> Date: Tue, 13 Oct 2015 10:27:16 +0200
> From: Przemysław Wojnowski <esperanto@cumego.com>
> Cc: emacs-devel@gnu.org
> 
>     I don't think this is possible.  Some slowdown will be inevitable.
> 
> IMHO the performance is overestimated.

I don't think it is (see below).  But in any case, I was simply
responding to a wish, which you deleted, that said:

> Or a more radical approach: a Lispy language, easy to learn by those who
> already know Elisp, but which can be translated all the way to something
> as performant as the current C code
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

My observation above simply says that "Lisp translated into C" will
always be somewhat slower than hand-written C.  That's all.

> Look at Java based IDEs - Idea, Eclipse, Netbeans, etc.
> All of them have been written in _slow_ Java (at least according
> to C standard) and run on _slow_ JVM. But all of them are able to
> attract more users that Emacs, which has core in blazing fast C.

Err... isn't "IDE" the subject of another thread?

In this thread, we are not talking about attracting more users, we are
talking about attracting more _developers_ for the core C code.

Anyway, we do have in Emacs a few known performance critical spots,
and if we switch to another language, we would like to avoid making
them slower in transition.

> The truth is that nobody cares about performance as long as it is
> _good enough_.

Sure.  And under some circumstances Emacs does show performance
problems precisely by that criterion.  For example, the display engine
is sometimes (fortunately, rarely) too slow to degrees that annoy
users.  Rewrite those parts in a slightly slower language, and you
have a larger problem; in particular, the rare situations where this
shows might become less rare.  We don't want that.

> IMHO C lacks basic abstractions that reduce conceptual weight of
> any subsystem/library/concept/etc. Interfaces and namespaces do.
> Static typing would also help.

As Daniel already wrote in this thread: don't underestimate the
complexity of Emacs that has nothing to do with the implementation
language.  Emacs solves quite a few very complex problems; the
solution is therefore complex, and will be so no matter in what
language you express it.  That complexity is IME the main obstacle for
newcomers to Emacs maintenance.  The rest is non-negligible, of
course, but good engineering practice is to solve the main problem
first, and go for the secondary one only afterwards.

Therefore, I think efforts to improve the documentation, commentary,
and readability of the code will yield higher gains faster than
switching to another language.

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> In this thread, we are not talking about attracting more users, we are
> talking about attracting more _developers_ for the core C code.

[...]

> As Daniel already wrote in this thread: don't underestimate the complexity
> of Emacs that has nothing to do with the implementation language. Emacs
> solves quite a few very complex problems; the solution is therefore complex,
> and will be so no matter in what language you express it. That complexity is
> IME the main obstacle for newcomers to Emacs maintenance. The rest is
> non-negligible, of course, but good engineering practice is to solve the
> main problem first, and go for the secondary one only afterwards.

> Therefore, I think efforts to improve the documentation, commentary, and
> readability of the code will yield higher gains faster than switching to
> another language.

I just want to give a huge +1 to this whole message by Eli. To anyone skimming
through this thread, it deserves a closer read.

John

Re: Emacs rewrite in a maintainable language

[Przemysław Wojnowski]

W dniu 13.10.2015 o 17:05, Eli Zaretskii pisze:
>> IMHO the performance is overestimated.
>
> I don't think it is (see below).  But in any case, I was simply
> responding to a wish, which you deleted, that said:
I deleted it on purpose, because I was referring to the "performance argument"
that is very often used in this thread. IMHO the argument is invalid when 
performance is good enough. In such case other things matter more.

>> Look at Java based IDEs - Idea, Eclipse, Netbeans, etc.
>> All of them have been written in _slow_ Java (at least according
>> to C standard) and run on _slow_ JVM. But all of them are able to
>> attract more users that Emacs, which has core in blazing fast C.
>
> Err... isn't "IDE" the subject of another thread?
>
> In this thread, we are not talking about attracting more users, we are
> talking about attracting more _developers_ for the core C code.
Not exactly, this (sub)thread is about validity of "performance" argument.
What I said was: users choose IDE using other criteria, when it performance is 
good enough.

> Therefore, I think efforts to improve the documentation, commentary,
> and readability of the code will yield higher gains faster than
> switching to another language.
I'm not sure about "higher gains" in longer perspective, but they definitely
will help.

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Przemysław Wojnowski <esperanto@cumego.com> writes:

> Look at Java based IDEs - Idea, Eclipse, Netbeans, etc. All of them have
> been written in _slow_ Java (at least according to C standard) and run on
> _slow_ JVM. But all of them are able to attract more users that Emacs, which
> has core in blazing fast C.

> The truth is that nobody cares about performance as long as it is _good
> enough_.

This is absolutely not true. Lagginess and lack of performance are the primary
reasons I avoid Eclipse and IntelliJ as much as possible -- even when their
IDE features would otherwise make them a better choice for some projects.

I will not consider regressing the pleasantly snappy feel of Emacs.

John

Re: Emacs rewrite in a maintainable language

[Przemysław Wojnowski]

W dniu 13.10.2015 o 18:06, John Wiegley pisze:
>> The truth is that nobody cares about performance as long as it is _good
>> enough_.
>
> This is absolutely not true. Lagginess and lack of performance are the primary
> reasons I avoid Eclipse and IntelliJ as much as possible -- even when their
> IDE features would otherwise make them a better choice for some projects.

But *you* are a very small sample compared to those users, who have chosen
those IDEs with *good enough* performance, but more features.

Simply, things have different value over time. It's also true for performance,
which has lower value when is good enough.

emacs IDE features (was: Emacs rewrite in a maintainable language)

[Andrés Ramírez]

Hi Przemysław.
> But *you* are a very small sample compared to those users, who have chosen
> those IDEs with *good enough* performance, but more features.

I left those IDE's because they freeze and when they do so. You need to
pray they would came back. On the contrary "Emacs is rock solid" and
just freezes when using the mail client (for a few seconds), surely Emacs
would come back after those few seconds. During
my 10 years (with the same machine) journey Emacs has crashed just a
couple of times and I have not been able to replicate the case when
Emacs has crashed.

Even if Emacs would add those required IDE's features. People are not
going to choose Emacs which has a different paradigm. Emacs users do not
tend to use the mouse a lot. Most People on the IDEs use a lot the
mouse. Besides that Emacs has a "steep learning curve". Emacs is an
extensible tool that's the reason people has different .emacs's
files. When You start with one of those predefined Emacs's setups, as
soon as you evolve your workflow You are going to start to modify your .emacs
according to your own personal needs.

Also IDE's are resource hungry. But this could be solved buying new
machines. I have been able of porting my Emacs setup to a 400 Mhz
machine and do some C work there. But probably most people won't do
that.

I think Cedet is a step in the Right direction. Cedet needs performance
improvements. And also more attention from the Emacs core. as John
pointed out "Let's setup an architecture first" for avoiding a lot of
duplicated effort and then start the work for on several iterations
adding more IDE's features to Emacs.

Then of course I should mention priorities:

- IDE's features
- Concurrency
- Guile Emacs future?. Guile Emacs has been on several Google summer of
code. Would it need to be on another one?. Or Emacs is going to move on
another direction?

And probably more features added to this list. Man power and team work
is not easy to find.

Regards

Re: emacs IDE features

[John Wiegley]

>>>>> Andrés Ramírez <sunshavi@fastmail.fm> writes:

> Then of course I should mention priorities:
> - IDE's features
> - Concurrency
> - Guile Emacs future?. Guile Emacs has been on several Google summer of
> code. Would it need to be on another one?. Or Emacs is going to move on
> another direction?

I have three major priorities in mind for the near-term, and you've named two.
I'm happy to see that we're on a similar wavelength.

But first, I want to know who will actually be the maintainer(s) before we
have that discussion. When/if that day comes, I'll present a high-level
proposal in collaboration with Richard and (hopefully) Eli, and then we can
all decide from there where to best spend our discussive energies.

John

Re: emacs IDE features

[Phillip Lord]

John Wiegley <johnw@newartisans.com> writes:

>>>>>> Andrés Ramírez <sunshavi@fastmail.fm> writes:
>
>> Then of course I should mention priorities:
>> - IDE's features
>> - Concurrency
>> - Guile Emacs future?. Guile Emacs has been on several Google summer of
>> code. Would it need to be on another one?. Or Emacs is going to move on
>> another direction?
>
> I have three major priorities in mind for the near-term, and you've named two.
> I'm happy to see that we're on a similar wavelength.
>
> But first, I want to know who will actually be the maintainer(s) before we
> have that discussion. When/if that day comes, I'll present a high-level
> proposal in collaboration with Richard and (hopefully) Eli, and then we can
> all decide from there where to best spend our discussive energies.


I'd add number 4. Support Java properly. It's one of the most common
languages in the world, and Emacs doesn't work with it.

There has been a little movement on the JDEE list recently. I still use
Emacs for editing Java, but only because I need to turn my code into a
slide show (for teaching) rather than actually running it. Even this is
going to be a problem in the future unless Emacs updates.

Phil

Re: emacs IDE features

[John Wiegley]

>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:

> I'd add number 4. Support Java properly. It's one of the most common
> languages in the world, and Emacs doesn't work with it.

As a core Emacs developer who doesn't use Java, it's hard to make that a
"mission statement" for the whole emacs-devel team.

However, I'd very much like it if someone could step up and volunteer as "Java
technical lead" to support this use case. I'd expect such a person to work
with the maintainers to imagine a better environment -- in the context of our
larger IDE discussion-- and engage others to help get the coding done. It is,
as you say, a large user base.

What do you think, Phillip? :)  Or anyone else?

In fact, we stand in need of several individuals, willing to take charge of
different aspects of Emacs where not all of us have expertise. I'd love to
find several such "group leads", so we ensure that every aspect of Emacs
receives constant attention. For example, we need people who are willing to
lead in the areas of:

    - Documentation
    - Each major language
    - Core libraries
    - External resources
    - Contributor well-being
    - Each supported platform
    - Performance
    - etc.

If you think you aren't up to it, you probably are, more than you think.
Dunning-Kruger strikes us all. :)

John

Re: emacs IDE features

[Eli Zaretskii]

> From: "John Wiegley" <johnw@newartisans.com>
> Date: Fri, 16 Oct 2015 16:13:54 -0700
> 
> In fact, we stand in need of several individuals, willing to take charge of
> different aspects of Emacs where not all of us have expertise. I'd love to
> find several such "group leads", so we ensure that every aspect of Emacs
> receives constant attention.

100% agreement.

>     - Core libraries

You mean "each core library", I presume.  IMO, it is impractical to
have one lead and one team for all, or even most of them.

>     - External resources

Not sure what that means.

>     - Each supported platform

This needs more detailed description.  What is a "platform"?  We have
Posix systems with and without GTK/Lucid/Motif, with and without
Cairo.  Are those the same platform, or do we need experts for each
toolkit?  Are all Posix systems the same "platforms", or do we need
separate experts on GNU/Linux, *BSD, Solaris, and Irix?

>     - Performance

I don't think this should be a separate team.  Each core team should
be responsible for performance in their area, because solution of any
performance problems in each area is specific to that area.

Re: emacs IDE features

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> - Core libraries

> You mean "each core library", I presume. IMO, it is impractical to have one
> lead and one team for all, or even most of them.

Right, I did mean that.

>> - External resources

> Not sure what that means.

Someone who is willing to keep an eye on things like reddit.com/r/emacs,
emacswiki.org, and other external resources, that really have nothing to do
with emacs-devel at all.

This would largely be a non-technical role: more a "community liaison" type of
thing. Are the informational and educational needs of the community being mit?

If I could pick anyone for this, it would be Sacha Chua. She has already been
serving in this capacity more or less, so I'd like to invite her to "report"
on the State of The Emacs Community with a touch more formality.

>> - Each supported platform

> This needs more detailed description. What is a "platform"? We have Posix
> systems with and without GTK/Lucid/Motif, with and without Cairo. Are those
> the same platform, or do we need experts for each toolkit? Are all Posix
> systems the same "platforms", or do we need separate experts on GNU/Linux,
> *BSD, Solaris, and Irix?

I suppose these dividing lines need to be drawn "as needed" and based on
available volunteers. If a group of platforms can be served by one person
(say, the *BSD family), great; if it becomes too big a job, or we've divided
it wrong, we make a change.

>> - Performance

> I don't think this should be a separate team. Each core team should be
> responsible for performance in their area, because solution of any
> performance problems in each area is specific to that area.

I agree, but I also don't agree.

I'd like to nominate a "performance czar", whose job is to construct and
maintain a performance benchmarking suite, run on all platforms as part of the
build, and whose output would be tracked over time to detect degradations and
inform us of the impact of changes.

In the Haskell world, we have exactly this for the GHC compiler:

    https://mail.haskell.org/pipermail/ghc-devs/2015-May/009032.html

The dashboard linked to in that message measures the performance impact of
every commit automatically, presents graphs, etc. The software behind it is
freely available, and its author is keen to receive usability suggestions.

I know that some don't believe performance is an issue in Emacs, but I would
counter with: maybe not the way *you* use it, or the environment you use it
in. There are performance issues in nextstep Emacs on OS X, for example, that
make it unusable for me. A benchmarking suite would help pinpoint why, and
ensure that once fixed, it wouldn't break again.

There's no harm in knowledge. If someone out there is interested, it really
could help improve our project. Plus, it doesn't require knowledge of either C
or Emacs Lisp. This could be a good starter project for someone who wants to
get involved, but doesn't feel confident submitting code or documentation just
yet.

John

Re: emacs IDE features

[Sacha Chua]

John Wiegley <johnw@newartisans.com> writes:

Hello, johnw, all!

> Someone who is willing to keep an eye on things like
> reddit.com/r/emacs, emacswiki.org, and other external resources, that
> really have nothing to do with emacs-devel at all. This would largely
> be a non-technical role: more a "community liaison" type of thing. Are
> the informational and educational needs of the community being mit?
> If I could pick anyone for this, it would be Sacha Chua. She has already been

I'd say jcs over at irreal.org does an even better job of pointing to
and summarizing cool stuff that people are sharing on Reddit and Planet
Emacsen. I'm totally happy to continue being an informal Emacs community
cheerleader, though.

It would be pretty cool to get something similar to Linux Weekly News
going, looking at new items coming in from:

- Emacs-related blog posts (on or off Planet Emacsen)
- Youtube, Vimeo, podcasts
- Meetups and other communities
- New packages
- Twitter
- Emacs StackExchange
- Newsgroups

The reddit.com/r/emacs folks do a good job at sharing interesting links
along those lines, actually.

I can see how far I get in terms of getting it up and running (and being
consistent with it, which could be the real challenge!) over on my blog.
It definitely makes sense as a web archive and/or a mailing list, and
maybe other people will want to pitch in too.

As for a summary of what people want: Daniel Gopar has mentioned the
need for more total-newbie-introduction tutorials and screencasts.
There's also the perpetual desire for Emacs 201-type information, or how
to go from the basic tutorial to actually setting things up for Emacs
awesomeness.

It feels like lots of people are sharing cool stuff in blog posts and
videos, which is definitely a step up from a few years ago. Of course,
people who are new don't know what to look for or where to go, and it
can be difficult to figure out a sequence for learning things or fill in
the gaps between pieces. We're missing the trails, the glue. EmacsWiki
used to be a little like that sort of a thing, and it still could become
that if people updated the resources that are on there. I might add that
to my list of little projects.

I'm guilty of this myself - I tend to focus more on weird idiosyncratic
hacks that make my life easier instead of making a concerted effort to
write coherent docs, although maybe I should play with that balance a
bit. I also try to nudge people to record screencasts (or conversations,
because those can sometimes be less intimidating) or write blog posts,
to get more stuff out there. I've been doing this in a rather haphazard
way, but I think we can figure out how to do this even better. =)

Sacha Chua

Re: emacs IDE features

[Xue Fuqiao]

On Sun, Oct 18, 2015 at 8:48 AM, Sacha Chua <sacha@sachachua.com> wrote:

> John Wiegley <johnw@newartisans.com> writes:
>> Someone who is willing to keep an eye on things like
>> reddit.com/r/emacs, emacswiki.org, and other external resources, that
>> really have nothing to do with emacs-devel at all. This would largely
>> be a non-technical role: more a "community liaison" type of thing. Are
>> the informational and educational needs of the community being mit?
>> If I could pick anyone for this, it would be Sacha Chua. She has already been

> It would be pretty cool to get something similar to Linux Weekly News
> going

I think something like Git's "Git Rev News"[1] and "What's cooking in
git.git"[2] would be great.

For the community outside of emacs-devel, it can provide useful
information about the development status (new features in master,
long-term branches progress, etc.), maintainer changes, Emacs internal
structure, and interesting discussions on emacs-devel.

For Emacs developers who don't follow emacs-devel and bug-gnu-emacs
closely, it can provide some important announcement about Emacs
development, e.g., feature freeze, workflow changes (VCS, ChangeLog,
commit message format, development branches), GNU ELPA development
process changes, new developers, and so on.

It can also provide some information for core developers about the rest
of the community, like downstream projects (Emacs Mac port, Aquamacs,
etc.) and interesting new packages.  It can even contain some cool new
features in the competitors of Emacs, for the sake of inspiration.

For the whole Emacs community, it can provide useful information by
linking to interesting articles and discussions about how to do various
things with Emacs.

2 cents, of course.

[1] A (monthly, IIRC) digest of all things Git, from core development to
    Git-related articles, tools, and projects.
[2] Reports about the current status and development direction of Git.

Re: emacs IDE features

[Sacha Chua]

Hi all!

Xue Fuqiao <xfq.free@gmail.com> writes:
> For Emacs developers who don't follow emacs-devel and bug-gnu-emacs
> closely, it can provide some important announcement about Emacs
> development, e.g., feature freeze, workflow changes (VCS, ChangeLog,
> commit message format, development branches), GNU ELPA development
> process changes, new developers, and so on.

I don't read emacs-devel or bug-gnu-emacs often, so someone else would
probably need to write that part. =) If someone would like to do that,
tell me the subject line to watch out for, and I'll link to those
summaries.

John Wiegley wrote:
>> I can see how far I get in terms of getting it up and running (and
>> being consistent with it, which could be the real challenge!) over on
>> my blog. It definitely makes sense as a web archive and/or a mailing
>> list, and maybe other people will want to pitch in too.
> Thanks, Sacha, I think your comments and energy demonstrate exactly
> what I was hoping for. You have the job! :) If you have any issues

Bwahahaha, as much of a "job" as these things are. <grin> Anyway, here's
my first attempt at it.

http://sachachua.com/blog/2015/10/2015-10-18-emacs-link-round-up/

Is that the sort of thing you had in mind?

It's in my usual Emacs category, so folks can read it on Planet Emacsen
as well. If I manage to keep it going for a few months, I might split it
off into its own category or domain. (Hmm, I haven't updated
emacslife.com in ages. I could probably put it there.)

Sacha Chua

Re: emacs IDE features

[John Wiegley]

>>>>> Sacha Chua <sacha@sachachua.com> writes:

> http://sachachua.com/blog/2015/10/2015-10-18-emacs-link-round-up/
> 
> Is that the sort of thing you had in mind?

You are awesome.

John

Re: emacs IDE features

[John Wiegley]

>>>>> Sacha Chua <sacha@sachachua.com> writes:

> I can see how far I get in terms of getting it up and running (and being
> consistent with it, which could be the real challenge!) over on my blog. It
> definitely makes sense as a web archive and/or a mailing list, and maybe
> other people will want to pitch in too.

Thanks, Sacha, I think your comments and energy demonstrate exactly what I was
hoping for. You have the job! :) If you have any issues with consistency, feel
free to confer with me if you'd like some encouragement or moral support from
time to time.

I'll put a separate call for technical leads later today in other areas we'd
like to start with, since I believe this sort of delegated structure could go
far toward reducing some of the technical burden we now face.

John

Re: emacs IDE features

[Christopher Allan Webber]

Sacha Chua writes:

> It feels like lots of people are sharing cool stuff in blog posts and
> videos, which is definitely a step up from a few years ago. Of course,
> people who are new don't know what to look for or where to go, and it
> can be difficult to figure out a sequence for learning things or fill in
> the gaps between pieces. We're missing the trails, the glue. EmacsWiki
> used to be a little like that sort of a thing, and it still could become
> that if people updated the resources that are on there. I might add that
> to my list of little projects.

Thanks for this email Sacha, your enthusiasm for learning and doing cool
things in Emacs and spreading that knowledge is, as always, infectious!

Re: emacs IDE features

[Eli Zaretskii]

> From: John Wiegley <johnw@newartisans.com>
> Date: Sat, 17 Oct 2015 16:51:48 -0700
> Cc: Sacha Chua <sacha@sachachua.com>
> 
> >> - Each supported platform
> 
> > This needs more detailed description. What is a "platform"? We have Posix
> > systems with and without GTK/Lucid/Motif, with and without Cairo. Are those
> > the same platform, or do we need experts for each toolkit? Are all Posix
> > systems the same "platforms", or do we need separate experts on GNU/Linux,
> > *BSD, Solaris, and Irix?
> 
> I suppose these dividing lines need to be drawn "as needed" and based on
> available volunteers. If a group of platforms can be served by one person
> (say, the *BSD family), great; if it becomes too big a job, or we've divided
> it wrong, we make a change.

I agree.  The experience till now, if I try to summarize it, was that
we are much more likely to find experts in specific aspects (like X,
GTK, etc.) than experts in everything that happens on some OS.

> >> - Performance
> 
> > I don't think this should be a separate team. Each core team should be
> > responsible for performance in their area, because solution of any
> > performance problems in each area is specific to that area.
> 
> I agree, but I also don't agree.
> 
> I'd like to nominate a "performance czar", whose job is to construct and
> maintain a performance benchmarking suite, run on all platforms as part of the
> build, and whose output would be tracked over time to detect degradations and
> inform us of the impact of changes.

I agree, but then we should probably strive to have a similar team for
the test suite, which already exists.

Re: emacs IDE features

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I agree, but then we should probably strive to have a similar team for the
> test suite, which already exists.

Yes!! My thoughts exactly. :)

John

Re: emacs IDE features

[Phillip Lord]

John Wiegley <johnw@newartisans.com> writes:

>>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> I'd add number 4. Support Java properly. It's one of the most common
>> languages in the world, and Emacs doesn't work with it.
>
> As a core Emacs developer who doesn't use Java, it's hard to make that a
> "mission statement" for the whole emacs-devel team.
>
> However, I'd very much like it if someone could step up and volunteer as "Java
> technical lead" to support this use case. I'd expect such a person to work
> with the maintainers to imagine a better environment -- in the context of our
> larger IDE discussion-- and engage others to help get the coding done. It is,
> as you say, a large user base.
>
> What do you think, Phillip? :)  Or anyone else?

There is a bit of a chicken and egg thing here. I don't use Java very
much these days either, and my own Emacs time is currently taken up with
other things.

We have seen some movement on the JDE mailing list recently, which has
been good, and we have the links between JDE and emacs-devel. So, we
already have this in place to some extent.

I don't mind trying to support any movement of code that needs to happen
from the various Java modes out there into core that I can. I think that
is the most I could commit to.

Phil

Re: emacs IDE features

[John Wiegley]

>>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:

> I don't mind trying to support any movement of code that needs to happen
> from the various Java modes out there into core that I can. I think that is
> the most I could commit to.

Thanks, Phillip. I think we should wait until we determine if that's what is
needed then.

John

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Przemysław Wojnowski <esperanto@cumego.com> writes:

> Simply, things have different value over time. It's also true for
> performance, which has lower value when is good enough.

When it comes to Emacs, "good enough" is just not good enough, in my book. We
must want it to be great, for it to become so.

Everyone can live with an Emacs that lacks some feature. Just don't use it for
that feature: it's still great at everything else it does. But make Emacs
sluggish and unresponsive, and you've killed its usability for everyone who
doesn't value the features you just traded it for.

A fancy moped would be "good enough" to get me around town; but just because
it fulfills the requirements of travel and can go places I can't go now,
doesn't mean I'd ever choose it over my car. "Good enough" is not an argument
I can buy when it comes to a thing I love.

John

Re: Emacs rewrite in a maintainable language

[Camm Maguire]

Greetings!

Óscar Fuentes <ofv@wanadoo.es> writes:

> "John Wiegley" <johnw@newartisans.com> writes:
>
> [snip]
>
>> A subset of Elisp that doesn't have dynamic binding might well produce
>> code that could be of use to us,
>
> That's what I had in mind all the time. We could aim to the subset of
> Elisp that can be translated to a fast enough target code, while
> remaining superior to C in convenience.
>
> Or a more radical approach: a Lispy language, easy to learn by those who
> already know Elisp, but which can be translated all the way to something
> as performant as the current C code, while being capable of interacting
> with it. This way, whenever someone needs fast code, he would use this
> other dialect instead of writing C, or the slow Elisp (or avoiding the
> task altogether.)
>

Has anyone ever considered using GNU gcl to translate lisp into (very
fast) C?

Take care,
-- 
Camm Maguire			     		    camm@maguirefamily.org
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

Re: Emacs rewrite in a maintainable language

[Alexis]

Camm Maguire <camm@maguirefamily.org> writes:

> Has anyone ever considered using GNU gcl to translate lisp into 
> (very fast) C?

Well, a couple of months ago, Pascal Bourguignon wrote:

    I'm writing a C parser in CL, so that I may read the C sources 
    of CL, and then translate them into maintainable CL.  Then I 
    will be able to compile a CL GNU emacs core, that would be 
    100% bug-for-bug compatible with GNU emacs, and on which you 
    could run all the .el (and even .elc) you'd want.

https://lists.gnu.org/archive/html/help-gnu-emacs/2015-08/msg00137.html


Alexis.

Re: Emacs rewrite in a maintainable language

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On 10/12/2015 06:38 PM, Alexis wrote:
> 
> Camm Maguire <camm@maguirefamily.org> writes:
> 
>> Has anyone ever considered using GNU gcl to translate lisp into (very
>> fast) C?
> 
> Well, a couple of months ago, Pascal Bourguignon wrote:
> 
>    I'm writing a C parser in CL, so that I may read the C sources    of
> CL, and then translate them into maintainable CL.  Then I    will be
> able to compile a CL GNU emacs core, that would be    100% bug-for-bug
> compatible with GNU emacs, and on which you    could run all the .el
> (and even .elc) you'd want.

So then, instead of C, we'll have a C AST represented in CL. Isn't one
custom language enough for Emacs?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs rewrite in a maintainable language

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Or a more radical approach: a Lispy language, easy to learn by those who
  > already know Elisp, but which can be translated all the way to something
  > as performant as the current C code, while being capable of interacting
  > with it.

This would be a tremendous amount of work, for a mere cleanup:
it would not make Emacs any better to use, for anyone.
It would not greatly help future development either.

I urge people to think instead about implementing useful features,
such as handling more kinds of formatted text, or various features
for a more powerful IDE (many of these are lacking).

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

>> Or a more radical approach: a Lispy language, easy to learn by those who
>> already know Elisp, but which can be translated all the way to something as
>> performant as the current C code, while being capable of interacting with
>> it.

> This would be a tremendous amount of work, for a mere cleanup: it would not
> make Emacs any better to use, for anyone. It would not greatly help future
> development either.

We suppose that it might be bad, and we suppose that it might be good. I'd
like to see some experimentation before calling it either way.

> I urge people to think instead about implementing useful features, such as
> handling more kinds of formatted text, or various features for a more
> powerful IDE (many of these are lacking).

The point is to discuss what is of possible interest to us. If others don't
care for this thread, let them ignore it; if they want to start a new thread
for more productive endeavors, may it be so.

John

Re: Emacs rewrite in a maintainable language

[Ken Raeburn]

> The thought of an Elisp->C compiler that could produce well-performing code is
> very interesting. I starting working on an LLVM JIT for Emacs bytecode at one
> point -- got as far as hello world -- and it was not terribly difficult. The
> main reason I stopped is that it didn't help performance enough. The slowness
> imposed by constantly looking up symbol names at global scope, due to dynamic
> binding, dominated the profiling results.
> 
> A subset of Elisp that doesn't have dynamic binding might well produce code
> that could be of use to us, and wouldn't need any of the macros we currently
> use to communicate between the two languages.

Do you think the introduction of lexical binding support changes the balance much?
(Or was your work after lexical binding was added?)

Ken

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Ken Raeburn <raeburn@raeburn.org> writes:

>> The main reason I stopped is that it didn't help performance enough. The
>> slowness imposed by constantly looking up symbol names at global scope, due
>> to dynamic binding, dominated the profiling results.

> Do you think the introduction of lexical binding support changes the balance
> much? (Or was your work after lexical binding was added?)

I think it should really help performance, but I haven't measured yet.

John

Re: Emacs rewrite in a maintainable language

[David Kastrup]

John Wiegley <johnw@newartisans.com> writes:

>>>>>> Ken Raeburn <raeburn@raeburn.org> writes:
>
>>> The main reason I stopped is that it didn't help performance enough. The
>>> slowness imposed by constantly looking up symbol names at global scope, due
>>> to dynamic binding, dominated the profiling results.
>
>> Do you think the introduction of lexical binding support changes the balance
>> much? (Or was your work after lexical binding was added?)
>
> I think it should really help performance, but I haven't measured yet.

I don't think it will really help performance all that much if you
measure it now since the inescapable cost of establishing and resolving
dynamic bindings at every let or function boundary has both made certain
optimizations mostly pointless as well as lending preference to a
programming style using iteration and setq rather than recursive
techniques.

Also, code like

`(lambda (...) ...)

does not magically rewrite itself into a compiled form just because
Elisp now got closures.

So there is a lot of trickle-down into the code base to wait for until
one can really make a statement.  And then the other reference point is
gone.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1137 bytes --]

On 10/13/2015 03:13 AM, David Kastrup wrote:
> John Wiegley <johnw@newartisans.com> writes:
> 
>>>>>>> Ken Raeburn <raeburn@raeburn.org> writes:
>>
>>>> The main reason I stopped is that it didn't help performance enough. The
>>>> slowness imposed by constantly looking up symbol names at global scope, due
>>>> to dynamic binding, dominated the profiling results.
>>
>>> Do you think the introduction of lexical binding support changes the balance
>>> much? (Or was your work after lexical binding was added?)
>>
>> I think it should really help performance, but I haven't measured yet.
> 
> I don't think it will really help performance all that much if you
> measure it now since the inescapable cost of establishing and resolving
> dynamic bindings at every let or function boundary has both made certain
> optimizations mostly pointless as well as lending preference to a
> programming style using iteration and setq rather than recursive
> techniques.

Maybe it's blasphemous to say so, but I often find iterate code much
more comprehensible and easily modified than equivalent logic written
recursively.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs rewrite in a maintainable language

[Stephen J. Turnbull]

Ken Raeburn writes:

 > Do you think the introduction of lexical binding support changes
 > the balance much?

FWIW, I was thinking that same thing but decided I am sufficiently
inexpert that I'd add more noise than signal.  Now that Ken's posted,
I'll stick my neck out and propose an empirical definition of RELisp
(that's not "Lisp with two problems", that's "restricted Emacs Lisp").

Specifically, have a Lisp-to-C (or even gcl-to-Evers-to-Chance-to-C,
as somebody-whose-name-I-should-remember-mea-maxima-culpa suggested)
compiler that just barfs on any non-lexical reference (and on defvar).

Learning it would be really easy: Just learn Emacs Lisp and then by
trial and error learn how to placate the compiler. :-)

Of course, I suppose that lexical refs probably already use a different
bytecode and an optimized code path, so compiling pure lexical code to
C would buy only a small integer factor of speedup at best.  But that
could be enough to make some mildly annoying delays fall within human
JND.

And the idea could easily be weakened to a bytecompiler mode that
refuses to emit dynamic refs, to teach people to avoid dynrefs on pain
of a cranky compiler. ;-)

Steve

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Replacing C with a language that helps people to write readable,
>> safe, efficient and maintainable code without having to study the
>> existing code base for months looks quite appealing to me. That
>> replacement could be gradual (no "stopping the world while The Big
>> Rewrite is underway.")  If that language is not so popular as C,
>> that's no big problem as long as it makes things less daunting for
>> those willing to work on the Emacs core. As already said, there is no
>> dramatic shortage of people who learn Elisp just for hacking Emacs.
>
> I think slowly moving the Emacs C core to C++ is a good idea.

I don't, and the reasons are similar to why Linux on C++ crashed and
burned.

The theory is very nice, but C++ is not a stable basis.  Every 5 years
it gets completely redefined, and the Elisp semantics don't magically
move with that.

> I've been trying to get into understanding Emacs C core, but it's just
> so hard with all those macros around. I think even replacing `#define`
> with `const` would be a huge improvement, worthwhile of the switch.
> The second step would be to replace some macros with type checked
> template functions.

"template functions"?  Seriously?  Template mechanisms are compile-time
mechanisms.  They would not correspond to anything in Elisp, ever.  So
they'd end up in parallel construction, with a construction on the C++
side of things that the C++ standard committee changes around every 5
years.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

David Kastrup <dak@gnu.org> writes:

> Oleh Krehel <ohwoeowho@gmail.com> writes:
>> I think slowly moving the Emacs C core to C++ is a good idea.
>
> I don't, and the reasons are similar to why Linux on C++ crashed and
> burned.

I haven't investigated that topic. Could you give the summary of the
reason?

> The theory is very nice, but C++ is not a stable basis.  Every 5 years
> it gets completely redefined, and the Elisp semantics don't magically
> move with that.

Not completely. I'm thinking of using only a very small subset of C++ /
superset of C:

- the C features
- classes with getters/setters (no need for inheritance)
- typed constants
- templated functions where appropriate

Essentially what good C programs do anyway, but with proper support from
the compiler. These features are supported by any version of any C++
compiler.

>> I've been trying to get into understanding Emacs C core, but it's just
>> so hard with all those macros around. I think even replacing `#define`
>> with `const` would be a huge improvement, worthwhile of the switch.
>> The second step would be to replace some macros with type checked
>> template functions.
>
> "template functions"?  Seriously?

Yes, very.

Quoting the Emacs sources:

    # define MIN(a,b) ((a) < (b) ? (a) : (b))
    #define streq(X, Y) (*(X) == *(Y) && strcmp ((X) + 1, (Y) + 1) == 0)
    #define SET_FLAG(F, FLAG)	((F) |= (FLAG))
    #define HAS_FLAG(F, FLAG)	(((F) & (FLAG)) != 0)
    #define xnew(n, Type)      ((Type *) xmalloc ((n) * sizeof (Type)))
    #define xrnew(op, n, Type) ((op) = (Type *) xrealloc (op, (n) * sizeof (Type)))
    #define switch_line_buffers() (curndx = 1 - curndx)
    #define curlinepos (lbs[curndx].linepos)
    #define BVAR(buf, field) ((buf)->field ## _)

Some of these can be replaced with plain type checked C++
functions. Others can be replaced with templated C++ functions, that are
still type checked. Still others with public getters and setters.

    Oleh

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Oleh Krehel <ohwoeowho@gmail.com> writes:

>
>> "template functions"?  Seriously?
>
> Yes, very.
>
> Quoting the Emacs sources:
>
>     # define MIN(a,b) ((a) < (b) ? (a) : (b))
>     #define streq(X, Y) (*(X) == *(Y) && strcmp ((X) + 1, (Y) + 1) == 0)
>     #define SET_FLAG(F, FLAG)	((F) |= (FLAG))
>     #define HAS_FLAG(F, FLAG)	(((F) & (FLAG)) != 0)
>     #define xnew(n, Type)      ((Type *) xmalloc ((n) * sizeof (Type)))
>     #define xrnew(op, n, Type) ((op) = (Type *) xrealloc (op, (n) *
> sizeof (Type)))
>     #define switch_line_buffers() (curndx = 1 - curndx)
>     #define curlinepos (lbs[curndx].linepos)
>     #define BVAR(buf, field) ((buf)->field ## _)
>
> Some of these can be replaced with plain type checked C++
> functions. Others can be replaced with templated C++ functions, that are
> still type checked.

"type checked" in the meaning that they will just grab suitable
operators for the given types and create code from them, just like
macros do just with much more complex semantics.  And you finally get to
write the stuff just like you did before.  The code using the stuff will
be just as readable, but figuring out what happens becomes harder.

> Still others with public getters and setters.

Shrug.  That looks a lot like C++ for C++' sake.  As I said: we do use
C++ in LilyPond, a high-level application built atop of the Scheme
interpreter GUILE (written in C itself).  And we get sort of a
reasonable payoff.  But the Elisp implementation is not a complex
high-level application.  It's a low-level wrapping akin to how GUILE
wraps Scheme in C.  And cobbling higher-level stuff together should
rather happen at the Elisp level than in a semi-parallel C/C++ level.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Date: Mon, 12 Oct 2015 17:17:52 +0200
> Cc: Óscar Fuentes <ofv@wanadoo.es>, emacs-devel@gnu.org
> 
>     # define MIN(a,b) ((a) < (b) ? (a) : (b))
>     #define streq(X, Y) (*(X) == *(Y) && strcmp ((X) + 1, (Y) + 1) == 0)
>     #define SET_FLAG(F, FLAG)	((F) |= (FLAG))
>     #define HAS_FLAG(F, FLAG)	(((F) & (FLAG)) != 0)
>     #define xnew(n, Type)      ((Type *) xmalloc ((n) * sizeof (Type)))
>     #define xrnew(op, n, Type) ((op) = (Type *) xrealloc (op, (n) * sizeof (Type)))
>     #define switch_line_buffers() (curndx = 1 - curndx)
>     #define curlinepos (lbs[curndx].linepos)
>     #define BVAR(buf, field) ((buf)->field ## _)
> 
> Some of these can be replaced with plain type checked C++
> functions. Others can be replaced with templated C++ functions, that are
> still type checked. Still others with public getters and setters.

Yes, but why should they be replaced?  Isn't it clear what SET_FLAG
does, just from its name?

Re: Emacs rewrite in a maintainable language

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

On 10/12/2015 08:17 AM, Oleh Krehel wrote:
> Quoting the Emacs sources:
>
>      # define MIN(a,b) ((a) < (b) ? (a) : (b))
>      #define streq(X, Y) (*(X) == *(Y) && strcmp ((X) + 1, (Y) + 1) == 0)
>      #define SET_FLAG(F, FLAG)	((F) |= (FLAG))
>      #define HAS_FLAG(F, FLAG)	(((F) & (FLAG)) != 0)
>      #define xnew(n, Type)      ((Type *) xmalloc ((n) * sizeof (Type)))
>      #define xrnew(op, n, Type) ((op) = (Type *) xrealloc (op, (n) * sizeof (Type)))
>      #define switch_line_buffers() (curndx = 1 - curndx)
>      #define curlinepos (lbs[curndx].linepos)
>      #define BVAR(buf, field) ((buf)->field ## _)
>
> Some of these can be replaced with plain type checked C++

Only BVAR belongs to the Emacs core. The other macros belong to 
auxiliary executables that presumably would be unaffected by any rewrite 
of the core. Anyway, for fun I just now replaced streq, SET_FLAG, and 
HAS_FLAG with functions, in the attached patch. More work needs to be 
done in this area but one step at a time....

[-- Attachment #2: 0001-Unmacroize-ebrowse.c-and-etags.c-a-bit.patch --]
[-- Type: text/x-patch, Size: 8095 bytes --]

From ddeec8ae168382bc16658f6836489ad26d7cf755 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Mon, 12 Oct 2015 15:30:18 -0700
Subject: [PATCH] Unmacroize ebrowse.c and etags.c a bit

* lib-src/ebrowse.c (READ_CHUNK_SIZE): Now an enum constant.
(streq, filename_eq, set_flag, has_flag): Now inline functions.
(set_flag): First arg is now an address, not an lvalue.
All callers changed.
(filename_eq, set_flag, has_flag):
Rename from FILENAME_EQ, SET_FLAG, HAS_FLAG.
All callers changed.
* lib-src/etags.c (streq, strcaseeq, strneq, strncaseeq):
Now inline functions.  Remove asserts that are unnecessary these
days (and in some cases were too-generous anyway).
---
 lib-src/ebrowse.c | 84 ++++++++++++++++++++++++++++++++-----------------------
 lib-src/etags.c   | 27 +++++++++++++++---
 2 files changed, 72 insertions(+), 39 deletions(-)

diff --git a/lib-src/ebrowse.c b/lib-src/ebrowse.c
index 09edc7d..f2093ae 100644
--- a/lib-src/ebrowse.c
+++ b/lib-src/ebrowse.c
@@ -32,27 +32,32 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #define SEEK_END 2
 #endif
 
-/* Conditionalize function prototypes.  */
-
-/* Value is non-zero if strings X and Y compare equal.  */
-
-#define streq(X, Y) (*(X) == *(Y) && strcmp ((X) + 1, (Y) + 1) == 0)
-
 #include <min-max.h>
 
 /* Files are read in chunks of this number of bytes.  */
 
-#define READ_CHUNK_SIZE (100 * 1024)
+enum { READ_CHUNK_SIZE = 100 * 1024 };
 
-#if defined (__MSDOS__)
-#define FILENAME_EQ(X,Y)    (strcasecmp (X,Y) == 0)
-#else
-#if defined (WINDOWSNT)
-#define FILENAME_EQ(X,Y)    (stricmp (X,Y) == 0)
+/* Value is true if strings X and Y compare equal.  */
+
+static bool
+streq (char const *x, char const *y)
+{
+  return strcmp (x, y) == 0;
+}
+
+static bool
+filename_eq (char const *x, char const *y)
+{
+#ifdef __MSDOS__
+  return strcasecmp (x, y) == 0;
+#elif defined WINDOWSNT
+  return stricmp (x, y) == 0;
 #else
-#define FILENAME_EQ(X,Y)    (streq (X,Y))
-#endif
+  return streq (x, y);
 #endif
+}
+
 /* The default output file name.  */
 
 #define DEFAULT_OUTFILE "BROWSE"
@@ -217,10 +222,19 @@ enum visibility
 #define F_EXTERNC	256	/* Is declared extern "C".  */
 #define F_DEFINE	512	/* Is a #define.  */
 
-/* Two macros to set and test a bit in an int.  */
+/* Set and test a bit in an int.  */
 
-#define SET_FLAG(F, FLAG)	((F) |= (FLAG))
-#define HAS_FLAG(F, FLAG)	(((F) & (FLAG)) != 0)
+static void
+set_flag (int *f, int flag)
+{
+  *f |= flag;
+}
+
+static bool
+has_flag (int f, int flag)
+{
+  return (f & flag) != 0;
+}
 
 /* Structure describing a class member.  */
 
@@ -682,7 +696,7 @@ add_member_decl (struct sym *cls, char *name, char *regexp, int pos, unsigned in
     m = add_member (cls, name, var, sc, hash);
 
   /* Have we seen a new filename?  If so record that.  */
-  if (!cls->filename || !FILENAME_EQ (cls->filename, filename))
+  if (!cls->filename || !filename_eq (cls->filename, filename))
     m->filename = filename;
 
   m->regexp = regexp;
@@ -745,7 +759,7 @@ add_member_defn (struct sym *cls, char *name, char *regexp, int pos, unsigned in
   if (!cls->sfilename)
     cls->sfilename = filename;
 
-  if (!FILENAME_EQ (cls->sfilename, filename))
+  if (!filename_eq (cls->sfilename, filename))
     m->def_filename = filename;
 
   m->def_regexp = regexp;
@@ -830,7 +844,7 @@ add_global_decl (char *name, char *regexp, int pos, unsigned int hash, int var,
   if (!found)
     {
       if (!global_symbols->filename
-	  || !FILENAME_EQ (global_symbols->filename, filename))
+	  || !filename_eq (global_symbols->filename, filename))
 	m->filename = filename;
 
       m->regexp = regexp;
@@ -931,11 +945,11 @@ mark_virtual (struct sym *r)
   for (p = r->subs; p; p = p->next)
     {
       for (m = r->fns; m; m = m->next)
-        if (HAS_FLAG (m->flags, F_VIRTUAL))
+        if (has_flag (m->flags, F_VIRTUAL))
           {
             for (m2 = p->sym->fns; m2; m2 = m2->next)
               if (m->param_hash == m2->param_hash && streq (m->name, m2->name))
-                SET_FLAG (m2->flags, F_VIRTUAL);
+                set_flag (&m2->flags, F_VIRTUAL);
           }
 
       mark_virtual (p->sym);
@@ -1159,7 +1173,7 @@ sym_scope_1 (struct sym *p)
   strcpy (scope_buffer + scope_buffer_len, p->name);
   scope_buffer_len += len;
 
-  if (HAS_FLAG (p->flags, F_TEMPLATE))
+  if (has_flag (p->flags, F_TEMPLATE))
     {
       ensure_scope_buffer_room (3);
       strcpy (scope_buffer + scope_buffer_len, "<>");
@@ -2435,7 +2449,7 @@ parm_list (int *flags)
         {
           /* We can overload the same function on `const' */
           hash = (hash << 1) ^ CONST;
-          SET_FLAG (*flags, F_CONST);
+          set_flag (flags, F_CONST);
           MATCH ();
         }
 
@@ -2443,7 +2457,7 @@ parm_list (int *flags)
         {
           MATCH ();
           SKIP_MATCHING_IF ('(');
-          SET_FLAG (*flags, F_THROW);
+          set_flag (flags, F_THROW);
         }
 
       if (LOOKING_AT ('='))
@@ -2452,7 +2466,7 @@ parm_list (int *flags)
           if (LOOKING_AT (CINT) && yyival == 0)
             {
               MATCH ();
-              SET_FLAG (*flags, F_PURE);
+              set_flag (flags, F_PURE);
             }
         }
     }
@@ -2505,25 +2519,25 @@ member (struct sym *cls, int vis)
           /* A function or class may follow.  */
         case TEMPLATE:
           MATCH ();
-          SET_FLAG (flags, F_TEMPLATE);
+          set_flag (&flags, F_TEMPLATE);
           /* Skip over template argument list */
           SKIP_MATCHING_IF ('<');
           break;
 
         case EXPLICIT:
-          SET_FLAG (flags, F_EXPLICIT);
+          set_flag (&flags, F_EXPLICIT);
           goto typeseen;
 
         case MUTABLE:
-          SET_FLAG (flags, F_MUTABLE);
+          set_flag (&flags, F_MUTABLE);
           goto typeseen;
 
         case T_INLINE:
-          SET_FLAG (flags, F_INLINE);
+          set_flag (&flags, F_INLINE);
           goto typeseen;
 
         case VIRTUAL:
-          SET_FLAG (flags, F_VIRTUAL);
+          set_flag (&flags, F_VIRTUAL);
           goto typeseen;
 
         case '[':
@@ -2761,7 +2775,7 @@ parse_classname (void)
       if (LOOKING_AT ('<'))
         {
           skip_matching ();
-          SET_FLAG (last_class->flags, F_TEMPLATE);
+          set_flag (&last_class->flags, F_TEMPLATE);
         }
 
       if (!LOOKING_AT (DCOLON))
@@ -3189,7 +3203,7 @@ declaration (int flags)
           break;
 
         case T_INLINE:
-          SET_FLAG (flags, F_INLINE);
+          set_flag (&flags, F_INLINE);
           MATCH ();
           break;
 
@@ -3335,14 +3349,14 @@ globals (int start_flags)
                   MATCH_IF ('}');
                 }
               else
-                SET_FLAG (flags, F_EXTERNC);
+                set_flag (&flags, F_EXTERNC);
             }
           break;
 
         case TEMPLATE:
           MATCH ();
           SKIP_MATCHING_IF ('<');
-          SET_FLAG (flags, F_TEMPLATE);
+          set_flag (&flags, F_TEMPLATE);
           break;
 
         case CLASS: case STRUCT: case UNION:
diff --git a/lib-src/etags.c b/lib-src/etags.c
index 791722d..8b980d3 100644
--- a/lib-src/etags.c
+++ b/lib-src/etags.c
@@ -150,10 +150,29 @@ char pot_etags_version[] = "@(#) pot revision number is 17.38.1.4";
 # define CTAGS false
 #endif
 
-#define streq(s,t)	(assert ((s)!=NULL || (t)!=NULL), !strcmp (s, t))
-#define strcaseeq(s,t)	(assert ((s)!=NULL && (t)!=NULL), !c_strcasecmp (s, t))
-#define strneq(s,t,n)	(assert ((s)!=NULL || (t)!=NULL), !strncmp (s, t, n))
-#define strncaseeq(s,t,n) (assert ((s)!=NULL && (t)!=NULL), !c_strncasecmp (s, t, n))
+static bool
+streq (char const *s, char const *t)
+{
+  return strcmp (s, t) == 0;
+}
+
+static bool
+strcaseeq (char const *s, char const *t)
+{
+  return c_strcasecmp (s, t) == 0;
+}
+
+static bool
+strneq (char const *s, char const *t, size_t n)
+{
+  return strncmp (s, t, n) == 0;
+}
+
+static bool
+strncaseeq (char const *s, char const *t, size_t n)
+{
+  return c_strncasecmp (s, t, n) == 0;
+}
 
 /* C is not in a name.  */
 static bool
-- 
2.1.0

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 10/12/2015 08:17 AM, Oleh Krehel wrote:
>> Quoting the Emacs sources:
>>
>>      # define MIN(a,b) ((a) < (b) ? (a) : (b))
>>      #define streq(X, Y) (*(X) == *(Y) && strcmp ((X) + 1, (Y) + 1) == 0)
>>      #define SET_FLAG(F, FLAG)	((F) |= (FLAG))
>>      #define HAS_FLAG(F, FLAG)	(((F) & (FLAG)) != 0)
>>      #define xnew(n, Type)      ((Type *) xmalloc ((n) * sizeof (Type)))
>>      #define xrnew(op, n, Type) ((op) = (Type *) xrealloc (op, (n) * sizeof (Type)))
>>      #define switch_line_buffers() (curndx = 1 - curndx)
>>      #define curlinepos (lbs[curndx].linepos)
>>      #define BVAR(buf, field) ((buf)->field ## _)
>>
>> Some of these can be replaced with plain type checked C++
>
> Only BVAR belongs to the Emacs core. The other macros belong to
> auxiliary executables that presumably would be unaffected by any
> rewrite of the core. Anyway, for fun I just now replaced streq,
> SET_FLAG, and HAS_FLAG with functions, in the attached patch. More
> work needs to be done in this area but one step at a time....

Thanks, Paul.

May I ask if there's a reason to not have a single style for const
pointers? I get 2162 occurrences of "const char *", and 357 occurrences
of "char const *" in the C sources.

I have to admit that I haven't seen the "char const *" variation
anywhere before. Just to get another sample, I looked at the Linux
sources; their ratio is 31961:189 in favor of "const char *".

If no one deems it counter-productive, I'd like to switch all const
pointers to a single "const char *" style. If we want more people to be
comfortable with the C sources, style and consistency is important.

    Oleh

Re: Emacs rewrite in a maintainable language

[Alan Mackenzie]

Hello, Oleh.

On Tue, Oct 13, 2015 at 01:27:49PM +0200, Oleh Krehel wrote:

> May I ask if there's a reason to not have a single style for const
> pointers? I get 2162 occurrences of "const char *", and 357 occurrences
> of "char const *" in the C sources.

Yes there is.  The two are semantically different.  In one of them, it is
the pointer which is declared constant.  In the other, it's what's
pointed at that's constant.  Though I can never remember which is which.
:-(

>     Oleh

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Oleh.
>
> On Tue, Oct 13, 2015 at 01:27:49PM +0200, Oleh Krehel wrote:
>
>> May I ask if there's a reason to not have a single style for const
>> pointers? I get 2162 occurrences of "const char *", and 357 occurrences
>> of "char const *" in the C sources.
>
> Yes there is.  The two are semantically different.  In one of them, it is
> the pointer which is declared constant.  In the other, it's what's
> pointed at that's constant.  Though I can never remember which is which.
> :-(

Hi Alan,

Actually, "const char *" and "char const *" are absolutely equivalent in
all regards, both in C and C++.

I think you meant the difference between "const char *" and "char * const".

For exactly this reason (the confusion over 3 notations, 2 of which are
the same), I propose that we stick only to "const char *" and "char * const",
and don't use "char const *".

    Oleh

Re: Emacs rewrite in a maintainable language

[Alan Mackenzie]

Hello, Oleh.

On Tue, Oct 13, 2015 at 02:02:48PM +0200, Oleh Krehel wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > On Tue, Oct 13, 2015 at 01:27:49PM +0200, Oleh Krehel wrote:

> >> May I ask if there's a reason to not have a single style for const
> >> pointers? I get 2162 occurrences of "const char *", and 357 occurrences
> >> of "char const *" in the C sources.

> > Yes there is.  The two are semantically different.  In one of them, it is
> > the pointer which is declared constant.  In the other, it's what's
> > pointed at that's constant.  Though I can never remember which is which.
> > :-(

> Hi Alan,

> Actually, "const char *" and "char const *" are absolutely equivalent in
> all regards, both in C and C++.

> I think you meant the difference between "const char *" and "char * const".

Yes, you're right.  Sorry about that.  It's one of the few flaws in C's
syntax that there are here two confusing ways of writing the same thing.

> For exactly this reason (the confusion over 3 notations, 2 of which are
> the same), I propose that we stick only to "const char *" and "char * const",
> and don't use "char const *".

No disagreement here.  But the deprecated form will keep creeping back,
over the coming years.

>     Oleh

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Emacs rewrite in a maintainable language

[Mathieu Lirzin]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> For exactly this reason (the confusion over 3 notations, 2 of which are
> the same), I propose that we stick only to "const char *" and "char * const",
> and don't use "char const *".

This seems a good idea to me.  It would be nice for this convention to
be documented somewhere.  I think this can be defined in GNU coding
standards.  Can you send this suggestion to bugs-standards@gnu.org?

Thanks,

--
Mathieu Lirzin

Re: Emacs rewrite in a maintainable language

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]

On Tue, Oct 13, 2015 at 8:22 AM, Mathieu Lirzin <mthl@openmailbox.org>
wrote:

> Oleh Krehel <ohwoeowho@gmail.com> writes:
>
> > For exactly this reason (the confusion over 3 notations, 2 of which are
> > the same), I propose that we stick only to "const char *" and "char *
> const",
> > and don't use "char const *".
>
> This seems a good idea to me.  It would be nice for this convention to
> be documented somewhere.  I think this can be defined in GNU coding
> standards.  Can you send this suggestion to bugs-standards@gnu.org?
>

Standardizing is a good thing. Basing the choice purely on subjective
preference is less so. Many who have looked at the C and C++ grammars
notice that

    const char *

is an historical irregularity that has been folded into the grammar as
a special case. Embracing and standardizing on that special case
detracts from learning a consistent rule and typically leads to
misreading of more complex declarators. The alternative is to leverage
the fact that a const or volatile qualifier always applies to the
object to its left so cv-qualifiers uniformly appear on the right. No
need to remember special case rules or the relative precedence of
various type constructs.  Many C++ well-regarded coding standards
have adopted such a rule (along with moving '*' and '&' leftward).

/john

[-- Attachment #2: Type: text/html, Size: 4056 bytes --]

Re: Emacs rewrite in a maintainable language

[David Kastrup]

John Yates <john@yates-sheets.org> writes:

> On Tue, Oct 13, 2015 at 8:22 AM, Mathieu Lirzin <mthl@openmailbox.org>
> wrote:
>
>> Oleh Krehel <ohwoeowho@gmail.com> writes:
>>
>> > For exactly this reason (the confusion over 3 notations, 2 of which are
>> > the same), I propose that we stick only to "const char *" and "char *
>> const",
>> > and don't use "char const *".
>>
>> This seems a good idea to me.  It would be nice for this convention to
>> be documented somewhere.  I think this can be defined in GNU coding
>> standards.  Can you send this suggestion to bugs-standards@gnu.org?
>>
>
> Standardizing is a good thing. Basing the choice purely on subjective
> preference is less so. Many who have looked at the C and C++ grammars
> notice that
>
>     const char *
>
> is an historical irregularity that has been folded into the grammar as
> a special case. Embracing and standardizing on that special case
> detracts from learning a consistent rule and typically leads to
> misreading of more complex declarators. The alternative is to leverage
> the fact that a const or volatile qualifier always applies to the
> object to its left so cv-qualifiers uniformly appear on the right. No
> need to remember special case rules or the relative precedence of
> various type constructs.  Many C++ well-regarded coding standards
> have adopted such a rule (along with moving '*' and '&' leftward).

I don't get the preoccupation with the order and the talk about
"historical irregularity".  I can perfectly well write

unsigned const int extern x;

and that is a well-formed declaration.  It doesn't matter in which order
the various type declarators appear: you can reorder at will as long as
you don't move across * or equivalent.

I am not even sure _historically_ (meaning K&R C), storage class
specifiers would have had to come before the rest.  After all,

    extern x;

was and still is a perfectly valid declaration in C, so a storage class
declaration can fill in for a type declaration.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Andreas Schwab]

David Kastrup <dak@gnu.org> writes:

> I don't get the preoccupation with the order and the talk about
> "historical irregularity".  I can perfectly well write
>
> unsigned const int extern x;
>
> and that is a well-formed declaration.

Though obsolescent.  (6.11.5)

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Andreas Schwab <schwab@suse.de> writes:

> David Kastrup <dak@gnu.org> writes:
>> I don't get the preoccupation with the order and the talk about "historical
>> irregularity". I can perfectly well write
>> 
>> unsigned const int extern x;
>> 
>> and that is a well-formed declaration.

> Though obsolescent.  (6.11.5)

Please move this to another thread. It has zero relevance to the subject.

John

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

John Yates <john@yates-sheets.org> writes:

> Standardizing is a good thing. Basing the choice purely on subjective
> preference is less so. Many who have looked at the C and C++ grammars
> notice that
>
> const char *
>
> is an historical irregularity that has been folded into the grammar as
> a special case. Embracing and standardizing on that special case
> detracts from learning a consistent rule and typically leads to
> misreading of more complex declarators. The alternative is to leverage
> the fact that a const or volatile qualifier always applies to the
> object to its left so cv-qualifiers uniformly appear on the right. No
> need to remember special case rules or the relative precedence of
> various type constructs. Many C++ well-regarded coding standards
> have adopted such a rule (along with moving '*' and '&' leftward).

I'm fine with either way, as long as it's one and not two, while
preferring "const char *" since that's how I learned C++.

Just stating the data points for "const char *" vs. "char const *":

- Linux: 99.4%
- GCC: 99.2%
- Boost: 66.1%
- Emacs: 85.9%

Here, I think Linux and GCC are in a good spot style-wise, while both
Emacs and Boost suffer from mixed styles.

    Oleh

Re: Emacs rewrite in a maintainable language

[Sergey Organov]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> Alan Mackenzie <acm@muc.de> writes:
>
>> Hello, Oleh.
>>
>> On Tue, Oct 13, 2015 at 01:27:49PM +0200, Oleh Krehel wrote:
>>
>>> May I ask if there's a reason to not have a single style for const
>>> pointers? I get 2162 occurrences of "const char *", and 357 occurrences
>>> of "char const *" in the C sources.
>>
>> Yes there is.  The two are semantically different.  In one of them, it is
>> the pointer which is declared constant.  In the other, it's what's
>> pointed at that's constant.  Though I can never remember which is which.
>> :-(
>
> Hi Alan,
>
> Actually, "const char *" and "char const *" are absolutely equivalent in
> all regards, both in C and C++.
>
> I think you meant the difference between "const char *" and "char * const".
>
> For exactly this reason (the confusion over 3 notations, 2 of which are
> the same), I propose that we stick only to "const char *" and "char * const",
> and don't use "char const *".

People write "char const *" because it uses the general and simple rule
"'const' applies to what is to the left", that makes it easy to figure
what "char * const" means and why ('const' applies to '*' here, to
the pointer).

OTOH, "const char *", while looks more "natural", relies on additional
rule "when nothing is to the left, 'const' applies to what is to the
right".

Dunno if enforcing one form or another makes sense.

-- Sergey.

Re: Emacs rewrite in a maintainable language

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 233 bytes --]

On 13 Oct 2015 2:06 pm, "Sergey Organov" <sorganov@gmail.com> wrote:
> Dunno if enforcing one form or another makes sense.

It makes sense to prevent inconsistencies and confusion of new contributors
(specially at almost zero cost).

[-- Attachment #2: Type: text/html, Size: 334 bytes --]

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 13 Oct 2015 2:06 pm, "Sergey Organov" <sorganov@gmail.com> wrote:
>> Dunno if enforcing one form or another makes sense.
>
> It makes sense to prevent inconsistencies and confusion of new
> contributors (specially at almost zero cost).

Well, people who cannot figure out that "const char" and "char const"
are the same are not likely to find their way across our code base.  At
any rate, "const" in C is nuisance-only and not meaning-conveying like
in C++ where it may take part in disambiguation as well as semantics
(copy constructor calls behave specially and are very much const &).

So the "confusion" here is restricted to "oh, the compiler does not
complain?".

And indeed in C,

    #define const /*empty*/

is a reasonably reliable thing to do with almost no adverse effects.
Well, it renders

    const x = 4;

invalid.  But people rarely do that in C.  In particular since C++ does
not like it from the start.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Artur Malabarba]

2015-10-13 15:39 GMT+01:00 David Kastrup <dak@gnu.org>:
> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>
>> On 13 Oct 2015 2:06 pm, "Sergey Organov" <sorganov@gmail.com> wrote:
>>> Dunno if enforcing one form or another makes sense.
>>
>> It makes sense to prevent inconsistencies and confusion of new
>> contributors (specially at almost zero cost).
>
> Well, people who cannot figure out that "const char" and "char const"
> are the same are not likely to find their way across our code base.  At
> any rate, "const" in C is nuisance-only and not meaning-conveying like
> in C++ where it may take part in disambiguation as well as semantics
> (copy constructor calls behave specially and are very much const &).
>
> So the "confusion" here is restricted to "oh, the compiler does not
> complain?".

No, the confusion is "which one of these does what I want?".
I can perfectly figure the answer to that question on my own by
googling (which is "either"). That doesn't mean it's a good thing.
It's yet another small obstacle/time-waste/distraction I would have to
overcome.

The point is not that people can't figure it out. The point is that
the fewer useless things to figure out, the better.

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> 2015-10-13 15:39 GMT+01:00 David Kastrup <dak@gnu.org>:
>> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>>
>>> On 13 Oct 2015 2:06 pm, "Sergey Organov" <sorganov@gmail.com> wrote:
>>>> Dunno if enforcing one form or another makes sense.
>>>
>>> It makes sense to prevent inconsistencies and confusion of new
>>> contributors (specially at almost zero cost).
>>
>> Well, people who cannot figure out that "const char" and "char const"
>> are the same are not likely to find their way across our code base.  At
>> any rate, "const" in C is nuisance-only and not meaning-conveying like
>> in C++ where it may take part in disambiguation as well as semantics
>> (copy constructor calls behave specially and are very much const &).
>>
>> So the "confusion" here is restricted to "oh, the compiler does not
>> complain?".
>
> No, the confusion is "which one of these does what I want?".

The answer in C is "if the compiler does not complain, it does what I
want".

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

David Kastrup <dak@gnu.org> writes:

>>> Well, people who cannot figure out that "const char" and "char const"
>>> are the same are not likely to find their way across our code base.  At
>>> any rate, "const" in C is nuisance-only and not meaning-conveying like
>>> in C++ where it may take part in disambiguation as well as semantics
>>> (copy constructor calls behave specially and are very much const &).
>>>
>>> So the "confusion" here is restricted to "oh, the compiler does not
>>> complain?".
>>
>> No, the confusion is "which one of these does what I want?".
>
> The answer in C is "if the compiler does not complain, it does what I
> want".

The C compiler doesn't complain:

    mark_object (KVAR (kb, Voverriding_terminal_local_map));
    /* David Kastrup rulez! */
    mark_object (KVAR (kb, Vlast_command));
    /* David Kastrup rulez! */
    mark_object (KVAR (kb, Vreal_last_command));
    /* David Kastrup rulez! */
    mark_object (KVAR (kb, Vkeyboard_translate_table));

It doesn't mean that the above code is pleasant to read, or that anyone
should write this way.

See https://www.python.org/dev/peps/pep-0008/ for example. Just because
a code runs good doesn't mean it looks good. Whenever there's ambiguity
it's best to resolve it by deciding on the preferred alternative. For
instance the GNU braces style is used and enforced, instead of just
putting the braces wherever.

    Oleh

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>>>> Well, people who cannot figure out that "const char" and "char const"
>>>> are the same are not likely to find their way across our code base.  At
>>>> any rate, "const" in C is nuisance-only and not meaning-conveying like
>>>> in C++ where it may take part in disambiguation as well as semantics
>>>> (copy constructor calls behave specially and are very much const &).
>>>>
>>>> So the "confusion" here is restricted to "oh, the compiler does not
>>>> complain?".
>>>
>>> No, the confusion is "which one of these does what I want?".
>>
>> The answer in C is "if the compiler does not complain, it does what I
>> want".
>
> The C compiler doesn't complain:
>
>     mark_object (KVAR (kb, Voverriding_terminal_local_map));

Are we still talking about "const char" vs "char const" or is this just
another parade of the impressive Emacs-devel straw man army?

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Date: Tue, 13 Oct 2015 18:09:23 +0200
> Cc: Sergey Organov <sorganov@gmail.com>,
> 	Artur Malabarba <bruce.connor.am@gmail.com>,
> 	emacs-devel <emacs-devel@gnu.org>
> 
>     mark_object (KVAR (kb, Voverriding_terminal_local_map));
>     /* David Kastrup rulez! */
>     mark_object (KVAR (kb, Vlast_command));
>     /* David Kastrup rulez! */
>     mark_object (KVAR (kb, Vreal_last_command));
>     /* David Kastrup rulez! */
>     mark_object (KVAR (kb, Vkeyboard_translate_table));
> 
> It doesn't mean that the above code is pleasant to read, or that anyone
> should write this way.

Err... what exactly is wrong with this code, in your opinion?

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Oleh Krehel <ohwoeowho@gmail.com>
>> Date: Tue, 13 Oct 2015 18:09:23 +0200
>> Cc: Sergey Organov <sorganov@gmail.com>,
>> 	Artur Malabarba <bruce.connor.am@gmail.com>,
>> 	emacs-devel <emacs-devel@gnu.org>
>> 
>>     mark_object (KVAR (kb, Voverriding_terminal_local_map));
>>     /* David Kastrup rulez! */
>>     mark_object (KVAR (kb, Vlast_command));
>>     /* David Kastrup rulez! */
>>     mark_object (KVAR (kb, Vreal_last_command));
>>     /* David Kastrup rulez! */
>>     mark_object (KVAR (kb, Vkeyboard_translate_table));
>> 
>> It doesn't mean that the above code is pleasant to read, or that anyone
>> should write this way.
>
> Err... what exactly is wrong with this code, in your opinion?

The comments appear redundant.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> Date: Tue, 13 Oct 2015 11:46:30 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: Paul Eggert <eggert@cs.ucla.edu>, emacs-devel@gnu.org
> 
> > May I ask if there's a reason to not have a single style for const
> > pointers? I get 2162 occurrences of "const char *", and 357 occurrences
> > of "char const *" in the C sources.
> 
> Yes there is.  The two are semantically different.  In one of them, it is
> the pointer which is declared constant.  In the other, it's what's
> pointed at that's constant.  Though I can never remember which is which.
> :-(

Use 'cdecl':

  cdecl> explain const char *p
  declare p as pointer to const char
  cdecl> explain char const *p
  syntax error
  cdecl> explain char * const p
  declare p as const pointer to char

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 13 Oct 2015 11:46:30 +0000
>> From: Alan Mackenzie <acm@muc.de>
>> Cc: Paul Eggert <eggert@cs.ucla.edu>, emacs-devel@gnu.org
>> 
>> > May I ask if there's a reason to not have a single style for const
>> > pointers? I get 2162 occurrences of "const char *", and 357 occurrences
>> > of "char const *" in the C sources.
>> 
>> Yes there is.  The two are semantically different.  In one of them, it is
>> the pointer which is declared constant.  In the other, it's what's
>> pointed at that's constant.  Though I can never remember which is which.
>> :-(
>
> Use 'cdecl':
>
>   cdecl> explain const char *p
>   declare p as pointer to const char
>   cdecl> explain char const *p
>   syntax error
>   cdecl> explain char * const p
>   declare p as const pointer to char

Neither C compilers nor C++ compilers nor C standards nor C++ standards
agree with that diagnosis.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Paul Eggert]

Oleh Krehel wrote:
> I'd like to switch all const
> pointers to a single "const char *" style.

Let's not. It's more consistent to put 'const' after the type it modifies, and 
if we're going to have a "standard" style, that's the one we should have. But 
really, we have better things to do.

(I assume we can all agree, though, to avoid 'int const long'. :-)

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Oleh Krehel wrote:
>> I'd like to switch all const
>> pointers to a single "const char *" style.
>
> Let's not. It's more consistent to put 'const' after the type it
> modifies, and if we're going to have a "standard" style, that's the
> one we should have.

"const char *" style is used in 85.9% of the cases. If you want to
revert those uses to "char const *" I'll support you, since I want the
consistency of a single style.

> But really, we have better things to do.

I don't. I have no problem spending time to replace "char const *" into
"const char *". Browsing through a code base that I want to get familiar
with and fixing a small annoyance doesn't sound bad to me.

    Oleh

Re: Emacs rewrite in a maintainable language

[Andy Moreton]

On Tue 13 Oct 2015, Oleh Krehel wrote:

> Paul Eggert <eggert@cs.ucla.edu> writes:
>
>> Oleh Krehel wrote:
>>> I'd like to switch all const
>>> pointers to a single "const char *" style.
>>
>> Let's not. It's more consistent to put 'const' after the type it
>> modifies, and if we're going to have a "standard" style, that's the
>> one we should have.
>
> "const char *" style is used in 85.9% of the cases. If you want to
> revert those uses to "char const *" I'll support you, since I want the
> consistency of a single style.

Consistency is helpful, but pointless churn makes version history less
useful, so please don't make this kind of change unless you are also
making a semantic change to that area of the code.

>> But really, we have better things to do.
>
> I don't. I have no problem spending time to replace "char const *" into
> "const char *". Browsing through a code base that I want to get familiar
> with and fixing a small annoyance doesn't sound bad to me.

Pointless churn. Find something that fixes an bug, or adds a useful new
feature, and work on that. You will find it more rewarding, and other
users and developers will find your contribution more helpful.

    AndyM

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Tue 13 Oct 2015, Oleh Krehel wrote:
>
>> Paul Eggert <eggert@cs.ucla.edu> writes:
>>
>>> Oleh Krehel wrote:
>>>> I'd like to switch all const
>>>> pointers to a single "const char *" style.
>>>
>>> Let's not. It's more consistent to put 'const' after the type it
>>> modifies, and if we're going to have a "standard" style, that's the
>>> one we should have.
>>
>> "const char *" style is used in 85.9% of the cases. If you want to
>> revert those uses to "char const *" I'll support you, since I want the
>> consistency of a single style.
>
> Consistency is helpful, but pointless churn makes version history less
> useful, so please don't make this kind of change unless you are also
> making a semantic change to that area of the code.

One commit is pointless churn?

>>> But really, we have better things to do.
>>
>> I don't. I have no problem spending time to replace "char const *" into
>> "const char *". Browsing through a code base that I want to get familiar
>> with and fixing a small annoyance doesn't sound bad to me.
>
> Pointless churn. Find something that fixes an bug, or adds a useful new
> feature, and work on that. You will find it more rewarding, and other
> users and developers will find your contribution more helpful.

I don't care about pleasing the user or developer gods. I want to do
what's helpful for me, and I think that change would be. I'm offering to
do the work for free, and currently I'm not offering any other work as
an alternative. You either want the improvement or not. The concerns for
opportunity cost are irrelevant here.

Very often I see /harmless/ suggestions turned down as not worth it,
both on this list and on debbugs. This isn't a good way to move forward
if we want to expand the base of developers and bug-reporting users.

Re: Emacs rewrite in a maintainable language

[Andy Moreton]

On Wed 14 Oct 2015, Oleh Krehel wrote:

> Andy Moreton <andrewjmoreton@gmail.com> writes:
>
>> On Tue 13 Oct 2015, Oleh Krehel wrote:
>>
>>> Paul Eggert <eggert@cs.ucla.edu> writes:
>>>
>>>> Oleh Krehel wrote:
>>>>> I'd like to switch all const
>>>>> pointers to a single "const char *" style.
>>>>
>>>> Let's not. It's more consistent to put 'const' after the type it
>>>> modifies, and if we're going to have a "standard" style, that's the
>>>> one we should have.
>>>
>>> "const char *" style is used in 85.9% of the cases. If you want to
>>> revert those uses to "char const *" I'll support you, since I want the
>>> consistency of a single style.
>>
>> Consistency is helpful, but pointless churn makes version history less
>> useful, so please don't make this kind of change unless you are also
>> making a semantic change to that area of the code.
>
> One commit is pointless churn?
>
>>>> But really, we have better things to do.
>>>
>>> I don't. I have no problem spending time to replace "char const *" into
>>> "const char *". Browsing through a code base that I want to get familiar
>>> with and fixing a small annoyance doesn't sound bad to me.
>>
>> Pointless churn. Find something that fixes an bug, or adds a useful new
>> feature, and work on that. You will find it more rewarding, and other
>> users and developers will find your contribution more helpful.
>
> I don't care about pleasing the user or developer gods. I want to do
> what's helpful for me, and I think that change would be. I'm offering to
> do the work for free, and currently I'm not offering any other work as
> an alternative. You either want the improvement or not. The concerns for
> opportunity cost are irrelevant here.

Changes of this nature often turn into bikeshedding (where developers
disagree about the one true style), or an endless series of small
changes to modify code style (if style checker tools are not used).

All of this make it harder to follow the history when trying to find how
the behaviour has changed across versions. That is a cost that comes
with an otherwise harmless change.

> Very often I see /harmless/ suggestions turned down as not worth it,
> both on this list and on debbugs. This isn't a good way to move forward
> if we want to expand the base of developers and bug-reporting users.

It is not cost-free, as noted above. By all means update the style to be
consistent across the codebase, as long as you introduce a style checker
that can automatically enforce the agreed style.

As there have been dissenting voices in this thread, it seems your
preferred style does not meet with universal approval.

    AndyM

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Date: Wed, 14 Oct 2015 10:15:40 +0200
> Cc: emacs-devel@gnu.org
> 
> I don't care about pleasing the user or developer gods. I want to do
> what's helpful for me, and I think that change would be. I'm offering to
> do the work for free, and currently I'm not offering any other work as
> an alternative. You either want the improvement or not. The concerns for
> opportunity cost are irrelevant here.

If you feel like scratching this itch, please feel free.  TIA.

Re: Emacs rewrite in a maintainable language

[Przemysław Wojnowski]

W dniu 2015-10-13 23:02, Andy Moreton napisał(a):
[...]
>> I don't. I have no problem spending time to replace "char const *" 
>> into
>> "const char *". Browsing through a code base that I want to get 
>> familiar
>> with and fixing a small annoyance doesn't sound bad to me.
> 
> Pointless churn. Find something that fixes an bug, or adds a useful new
> feature, and work on that. You will find it more rewarding, and other
> users and developers will find your contribution more helpful.

Refactorings that increase maintainability (like this one) are very 
useful to other developers.

Such changes, even small, accumulate towards something that can be 
called a "maintainable system".
In (wannabe) healthy systems it is everyday practice.

Re: Emacs rewrite in a maintainable language

[Tassilo Horn]

Przemysław Wojnowski <esperanto@cumego.com> writes:

>>> I don't. I have no problem spending time to replace "char const *"
>>> into "const char *". Browsing through a code base that I want to get
>>> familiar with and fixing a small annoyance doesn't sound bad to me.
>>
>> Pointless churn. Find something that fixes an bug, or adds a useful
>> new feature, and work on that. You will find it more rewarding, and
>> other users and developers will find your contribution more helpful.
>
> Refactorings that increase maintainability (like this one) are very
> useful to other developers.
>
> Such changes, even small, accumulate towards something that can be
> called a "maintainable system".  In (wannabe) healthy systems it is
> everyday practice.

OTOH, they reduce the helpfulness version control offers to new and old
developers.  I.e., it's good when "git blame" shows you the last changes
which actually changed the code, and switching "char const *" to "const
char *" or vice versa are no real changes.

In the same vein, it's of course good to have one consistent indentation
style, one consistent style of setting braces, one consistent style of
naming variables, etc.  But I'd suggest to clean up the non-conforming
parts only when you do significant changes in that area anyhow.

Bye,
Tassilo

Re: Emacs rewrite in a maintainable language

[Przemysław Wojnowski]

W dniu 2015-10-14 12:56, Tassilo Horn napisał(a):
>> Refactorings that increase maintainability (like this one) are very
>> useful to other developers.
>> 
>> Such changes, even small, accumulate towards something that can be
>> called a "maintainable system".  In (wannabe) healthy systems it is
>> everyday practice.
> 
> OTOH, they reduce the helpfulness version control offers to new and old
> developers.  I.e., it's good when "git blame" shows you the last 
> changes
> which actually changed the code, and switching "char const *" to "const
> char *" or vice versa are no real changes.
Why? In git you've got full file history. Besides the latest message 
gives
you nothing if it doesn't explain why something was changed. "Fixed the
issue" types of messages are basically useless.

Important is: write to a person in the future why you do what you do.

> In the same vein, it's of course good to have one consistent 
> indentation
> style, one consistent style of setting braces, one consistent style of
> naming variables, etc.
This is the first thing every team does after deciding that they want to
have a maintainable system. In teams that are not used to constant
refactoring, people always start with "But history!", "Don't fix what's 
not
broken!", etc. But after sometime they admit that it was a good change.
In theory it's scary, in practice it works.

In maintainable (readable, understandable) systems developers can read
many things from the code and don't have to look into VC history all
the time.

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

Tassilo Horn <tsdh@gnu.org> writes:

> Przemysław Wojnowski <esperanto@cumego.com> writes:
>
>>>> I don't. I have no problem spending time to replace "char const *"
>>>> into "const char *". Browsing through a code base that I want to get
>>>> familiar with and fixing a small annoyance doesn't sound bad to me.
>>>
>>> Pointless churn. Find something that fixes an bug, or adds a useful
>>> new feature, and work on that. You will find it more rewarding, and
>>> other users and developers will find your contribution more helpful.
>>
>> Refactorings that increase maintainability (like this one) are very
>> useful to other developers.
>>
>> Such changes, even small, accumulate towards something that can be
>> called a "maintainable system".  In (wannabe) healthy systems it is
>> everyday practice.
>
> OTOH, they reduce the helpfulness version control offers to new and old
> developers.  I.e., it's good when "git blame" shows you the last changes
> which actually changed the code, and switching "char const *" to "const
> char *" or vice versa are no real changes.

I've seen this excuse before. Doesn't it mean that there's a flaw in the
"git blame" workflow rather than in the refactoring commits?  Besides,
"git blame" provides a way to specify a range of revisions that you're
interested in.

Personally, looking through the revision history in order to understand
code is a last resort for me. The code should be written and documented
well enough to be understandable at the current revision. This is
further magnified by the fact that we release source tarballs, which
should be self-containing.

> In the same vein, it's of course good to have one consistent indentation
> style, one consistent style of setting braces, one consistent style of
> naming variables, etc.  But I'd suggest to clean up the non-conforming
> parts only when you do significant changes in that area anyhow.

You mean:

    /* TODO: change "char const *" to "const char *" when an opportunity arises
      to change foo */
    void foo (char const *bar);

This doesn't sound like a sustainable and healthy approach.

In the same vein, concerning the matter of one consistent indentation
style, on occasion I've received complaints of producing too large
whitespace diffs when changing Elisp functions. On those occasions I
had to twist myself into never using `indent-sexp' and inserting the
appropriate amount of mixed tabs and spaces with "C-q". It wasn't fun.

    Oleh

Re: Emacs rewrite in a maintainable language

[Tassilo Horn]

Oleh Krehel <ohwoeowho@gmail.com> writes:

>> OTOH, they reduce the helpfulness version control offers to new and
>> old developers.  I.e., it's good when "git blame" shows you the last
>> changes which actually changed the code, and switching "char const *"
>> to "const char *" or vice versa are no real changes.
>
> I've seen this excuse before. Doesn't it mean that there's a flaw in
> the "git blame" workflow rather than in the refactoring commits?

No.  That's a very effective workflow when it works, and it works quite
often with mature code.  For example, there are tons of lines in AUCTeX
which haven't been changed for more than a decade and `vc-annotate'
still suffices to get to know why some specific piece of code is like it
is.

> Besides, "git blame" provides a way to specify a range of revisions
> that you're interested in.

Yes, and we have `vc-region-history' which is fantastic.

> Personally, looking through the revision history in order to
> understand code is a last resort for me.  The code should be written
> and documented well enough to be understandable at the current
> revision.  This is further magnified by the fact that we release
> source tarballs, which should be self-containing.

Yes, obviously that's ideal.  Now of course the current version might
have introduced some new problem, so you can't always take it as a
trusty source of information.

>> In the same vein, it's of course good to have one consistent
>> indentation style, one consistent style of setting braces, one
>> consistent style of naming variables, etc.  But I'd suggest to clean
>> up the non-conforming parts only when you do significant changes in
>> that area anyhow.
>
> You mean:
>
>     /* TODO: change "char const *" to "const char *" when an opportunity arises
>       to change foo */
>     void foo (char const *bar);

Yes, exactly that but without the TODO.  You just do it when you change
foo().

And don't get me wrong, I'm all for refactoring.  But IMHO switching
from one thing to a synonym of it or changing indentation from tabs to
spaces doesn't outweight the disadvantages of such "nothing really done"
commits.

> In the same vein, concerning the matter of one consistent indentation
> style, on occasion I've received complaints of producing too large
> whitespace diffs when changing Elisp functions.  On those occasions I
> had to twist myself into never using `indent-sexp' and inserting the
> appropriate amount of mixed tabs and spaces with "C-q". It wasn't fun.

For that reason, it's best to just go with the standard intentation
settings even when there's potential to improve them.  When I contribute
to your projects (and I like doing that, you do a great job!), I always
have to remember to turn off `aggressive-indent-mode' or later stage my
changes hunk-wise omitting hunks where only whitespace changes are
in. ;-)

Bye,
Tassilo

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Tassilo Horn <tsdh@gnu.org> writes:

> Przemysław Wojnowski <esperanto@cumego.com> writes:
>
>>>> I don't. I have no problem spending time to replace "char const *"
>>>> into "const char *". Browsing through a code base that I want to get
>>>> familiar with and fixing a small annoyance doesn't sound bad to me.
>>>
>>> Pointless churn. Find something that fixes an bug, or adds a useful
>>> new feature, and work on that. You will find it more rewarding, and
>>> other users and developers will find your contribution more helpful.
>>
>> Refactorings that increase maintainability (like this one) are very
>> useful to other developers.
>>
>> Such changes, even small, accumulate towards something that can be
>> called a "maintainable system".  In (wannabe) healthy systems it is
>> everyday practice.
>
> OTOH, they reduce the helpfulness version control offers to new and
> old developers.  I.e., it's good when "git blame" shows you the last
> changes which actually changed the code, and switching "char const *"
> to "const char *" or vice versa are no real changes.

On the other hand, the blame for declarations is mostly less interesting
than the blame for the actual use of such variables, so the attribution
damage is not all that important in this case.

> In the same vein, it's of course good to have one consistent indentation
> style,

git blame -w

helps skipping across spacing changes.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Tassilo Horn]

David Kastrup <dak@gnu.org> writes:

>> OTOH, they reduce the helpfulness version control offers to new and
>> old developers.  I.e., it's good when "git blame" shows you the last
>> changes which actually changed the code, and switching "char const *"
>> to "const char *" or vice versa are no real changes.
>
> On the other hand, the blame for declarations is mostly less
> interesting than the blame for the actual use of such variables, so
> the attribution damage is not all that important in this case.

On yet another hand, the declaration might be the right-hand side of an
assignment with an extremely complex left-hand side.

>> In the same vein, it's of course good to have one consistent
>> indentation style,
>
> git blame -w
>
> helps skipping across spacing changes.

Yes, I know.  But I'm assuming the default values of
`vc-*-annotate-switches' / `magit-blame-arguments'.  Oh, for the latter
"-w" is actually the default.

Bye,
Tassilo

Re: Emacs rewrite in a maintainable language

[Mathieu Lirzin]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Tue 13 Oct 2015, Oleh Krehel wrote:
[...]
>> "const char *" style is used in 85.9% of the cases. If you want to
>> revert those uses to "char const *" I'll support you, since I want the
>> consistency of a single style.
>
> Consistency is helpful, but pointless churn makes version history less
> useful, so please don't make this kind of change unless you are also
> making a semantic change to that area of the code.
[...]
>> I don't. I have no problem spending time to replace "char const *" into
>> "const char *". Browsing through a code base that I want to get familiar
>> with and fixing a small annoyance doesn't sound bad to me.
>
> Pointless churn. Find something that fixes an bug, or adds a useful new
> feature, and work on that. You will find it more rewarding, and other
> users and developers will find your contribution more helpful.
>
>     AndyM

Please express your opinion in a different way.  This kind of language
is direspectful to Oleh.  "other users and developers" have various
interests so please don't tell what others will find useful or not.

--
Mathieu Lirzin

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Andy Moreton <andrewjmoreton@gmail.com> writes:

>> I don't. I have no problem spending time to replace "char const *" into
>> "const char *". Browsing through a code base that I want to get familiar
>> with and fixing a small annoyance doesn't sound bad to me.

> Pointless churn. Find something that fixes an bug, or adds a useful new
> feature, and work on that. You will find it more rewarding, and other users
> and developers will find your contribution more helpful.

I've asked that we move this to another thread or stop the discussion, as it
detracts from the main topic.

Since people have ignored that request, in future I will create new threads to
continue discussion on the topic of interest. It's either that, or we move to
a semi-moderated list, if participants are unwilling to stay on topic or honor
requests to do so.

John

Re: Emacs rewrite in a maintainable language

[John Wiegley]

>>>>> Oleh Krehel <ohwoeowho@gmail.com> writes:

> May I ask if there's a reason to not have a single style for const pointers?
> I get 2162 occurrences of "const char *", and 357 occurrences of "char const
> *" in the C sources.

Hi Oleh,

In response to this and some other recent messages by others: If people have a
tangential question within the context of a larger thread, please start a new
thread by changing the subject line. A question about C pointers shouldn't be
part of a discussion on whether Emacs should be rewritten in another language.

It makes it hard to determine which messages to pay attention to, and that
slows down my ability to keep on top of relevant questions.

Thank you,
  John

Re: Emacs rewrite in a maintainable language

[Paul Eggert]

On 10/12/2015 05:48 AM, Oleh Krehel wrote:
> I've been trying to get into understanding Emacs C core, but it's just
> so hard with all those macros around.

I dislike those macros too, and have gotten rid of more than my share of 
them, typically replacing them with inline functions (e.g., XFLOAT) or 
with enum constants (e.g., BITS_PER_CHAR). Much more work could be done 
in this area. This part largely doesn't need C++, and the few places 
where it might benefit from C++ typically require so much C++ foo that 
it's not clear it's worth the hassle.

I suppose the main advantage of C++ over C, for something like the Emacs 
core, would be encapsulation. But here again, it's not clear that C++ 
would be enough of a win. The ways in which the Emacs core are poorly 
encapsulated are largely independent of C vs C++. Changing it to be 
better-encapsulated C++ would not be that much of a win over changing 
the current Emacs to be better-encapsulated C.

Of course all of this is sheer speculation. I haven't attempted 
rewriting major sections of the Emacs core into C++ and I suspect nobody 
else has either. Until that is attempted, we're just guessing. (And I'm 
not volunteering!)

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 10/12/2015 05:48 AM, Oleh Krehel wrote:
>> I've been trying to get into understanding Emacs C core, but it's just
>> so hard with all those macros around.
>
> I dislike those macros too, and have gotten rid of more than my share
> of them, typically replacing them with inline functions (e.g., XFLOAT)
> or with enum constants (e.g., BITS_PER_CHAR). Much more work could be
> done in this area. This part largely doesn't need C++, and the few
> places where it might benefit from C++ typically require so much C++
> foo that it's not clear it's worth the hassle.
>
> I suppose the main advantage of C++ over C, for something like the
> Emacs core, would be encapsulation. But here again, it's not clear
> that C++ would be enough of a win. The ways in which the Emacs core
> are poorly encapsulated are largely independent of C vs C++. Changing
> it to be better-encapsulated C++ would not be that much of a win over
> changing the current Emacs to be better-encapsulated C.
>
> Of course all of this is sheer speculation. I haven't attempted
> rewriting major sections of the Emacs core into C++ and I suspect
> nobody else has either. Until that is attempted, we're just
> guessing. (And I'm not volunteering!)

LilyPond is using C++ and I replaced a macro system for defining GUILE's
Scheme objects (smobs) and a macro system for defining message-passing
protocols with C++ constructs.  Part of the reason was that when using
this macro system in connection with C++ inheritance, the code just
bombed out with indecipherable symptoms either at compilation or, worse,
execution time and it was quite impossible to figure out what the code
actually did after preprocessing.

The main incentive here was to replace large conceptual complexity (not
adequately represented by C macros) with constructs more accessible to
the average programmer, and constructs that were understandable to GDB
and C++ and that were not bypassable with other ad-hoc code (C macros
obviously can just be spelled out).  In the context of doing that, there
were a few deadlocks in our development because the compiler
requirements increased and we had to update all of our cross-compiling
environment.  There also were some language and crosscompiling issues,
as well as some hiccups with people using Clang.

The C macros used within Emacs are not of a similar complexity and
type-madness.  I don't see that changing them to C++ constructs will
deliver enough of a return to justify the price.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Oleh Krehel]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 10/12/2015 05:48 AM, Oleh Krehel wrote:
>> I've been trying to get into understanding Emacs C core, but it's just
>> so hard with all those macros around.
>
> I dislike those macros too, and have gotten rid of more than my share
> of them, typically replacing them with inline functions (e.g., XFLOAT)
> or with enum constants (e.g., BITS_PER_CHAR). Much more work could be
> done in this area. This part largely doesn't need C++, and the few
> places where it might benefit from C++ typically require so much C++
> foo that it's not clear it's worth the hassle.

Thanks for your effort. It's very much needed and appreciated.

> I suppose the main advantage of C++ over C, for something like the
> Emacs core, would be encapsulation. But here again, it's not clear
> that C++ would be enough of a win. The ways in which the Emacs core
> are poorly encapsulated are largely independent of C vs C++. Changing
> it to be better-encapsulated C++ would not be that much of a win over
> changing the current Emacs to be better-encapsulated C.

Better encapsulation is definitely needed, and using C++ allows the
compiler to enforce this encapsulation, instead of just grepping
manually to make sure each private isn't used outside of
getters/setters.

For example, I was trying to write some code to make the mark be
window-local and not buffer-local, just like the point.  And it's hard
to see all uses of point wrt window-point, because of insufficient
encapsulation.

    Oleh

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Date: Mon, 12 Oct 2015 17:24:26 +0200
> Cc: emacs-devel@gnu.org
> 
> Better encapsulation is definitely needed, and using C++ allows the
> compiler to enforce this encapsulation, instead of just grepping
> manually to make sure each private isn't used outside of
> getters/setters.
> 
> For example, I was trying to write some code to make the mark be
> window-local and not buffer-local, just like the point.  And it's hard
> to see all uses of point wrt window-point, because of insufficient
> encapsulation.

Maybe a more adequate tool will help.  Did you try ID-Utils (already
integrated into Semantic to some extent)?

Re: Emacs rewrite in a maintainable language

[Stephen J. Turnbull]

Paul Eggert writes:

 > Of course all of this is sheer speculation. I haven't attempted
 > rewriting major sections of the Emacs core into C++ and I suspect
 > nobody else has either.

Ben Wing did a certain kind of rewriting into C++ with XEmacs (a few
thousand lines' worth, IIRC), mostly turning typedefs for objects of
several kinds that resolve to some kind of int-ish into proper classes
that contained an int-ish member.  This caught a number of bugs via
syntax errors ("this function takes an unsigned byte by the name of
Bufbyte and the actual argument is an unsigned byte by the name of
Ascbyte, so you lose!"), but more readable it isn't.  The client code
doesn't even change, it just bloats the implementation sources by a
factor of four or five or fifty depending on what methods are
necessary to allow class instances to interact normally with the other
types that are around.

I really don't think there are readability benefits to be gotten from
a change of language -- unless you're willing to sacrifice a *lot* of
performance -- although better type checking might help catch some
bugs.  OTOH, XEmacs is written in homebrew dialect called "Clean C",
which is the intersection of C89 and C++, as of 2003 or so, and that
already catches quite a few bugs if you compile both as C and as C++,
but I don't know how that would compare to taking advantage of other
features of C++.

So I'm with David K.  See also http://www.jwz.org/doc/cadt.html (if
you're a rewrite advocate, change into flame-retardant clothes first).

Re: Emacs rewrite in a maintainable language

[Marcus Harnisch]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Ben Wing did a certain kind of rewriting into C++ with XEmacs (a few
> thousand lines' worth, IIRC), mostly turning typedefs for objects of
> several kinds that resolve to some kind of int-ish into proper classes
> that contained an int-ish member.  This caught a number of bugs via
> syntax errors ("this function takes an unsigned byte by the name of
> Bufbyte and the actual argument is an unsigned byte by the name of
> Ascbyte, so you lose!"), but more readable it isn't.

Just to experiment with templates and partial specialization, I redid
part of his implementation where he used tons of macros and still
ended up with a lot of repetitive code for all sorts of EMACS_INT
derived position types. While the template-code is arguably nicer to
look at, it also helped avoiding a bug in his macro monster where
invoking non-existent macro silently expanded to nothing.

His much appreciated C++-based contribution appears rather
conservative by todays standards, which is understandable considering
the state of C++ compilers at the time of contribution (ca
2002). Given the time it would take any hypothetical C++ rewrite to
mature I wouldn't even bother starting with anything less than
C++2014.

Another potential I would see is the availability of higher-level
(STL) data structures w/o dependencies on external libraries, thus
allowing Emacsen to reduce their number of home-grown data
structures. Seeing comments in ancient code (I think it was related to
memory allocation) that translate to something along the lines of:

  “this magic number used to be value X in an even more ancient
   version, chosen as the result of trial-and-error on a small number
   of architectures at hand at the time of writing, and we have decided
   to increase it for totally non-scientific reasons, purely based on
   application of Moore's Law.”

make me cringe.

Best regards
Marcus

Re: Emacs rewrite in a maintainable language

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Seeing comments in ancient code (I think it was related to
  > memory allocation) that translate to something along the lines of:

  >   “this magic number used to be value X in an even more ancient
  >    version, chosen as the result of trial-and-error on a small number
  >    of architectures at hand at the time of writing, and we have decided
  >    to increase it for totally non-scientific reasons, purely based on
  >    application of Moore's Law.”

Could you show us what actually occurs in the Emacs sources,
rather than this paraphrase?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs rewrite in a maintainable language

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]

On 10/13/2015 05:02 AM, Marcus Harnisch wrote:
> Another potential I would see is the availability of higher-level
> (STL) data structures w/o dependencies on external libraries, thus
> allowing Emacsen to reduce their number of home-grown data
> structures. Seeing comments in ancient code

Similar comments apply to STL data structure implementations. We can
always do better with application-specific data structures than we can
with one-size-fits-all C++ data structures designed as reasonable
defaults, not optimal implementations. I'd hate for someone to think
that std::unordered_map, say, is somehow better than the data structures
we've used for a long time merely because the standard stipulates that
the runtime must provide it.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs rewrite in a maintainable language

[Marcus Harnisch]

Daniel Colascione <dancol@dancol.org> writes:

> We can
> always do better with application-specific data structures than we can
> with one-size-fits-all C++ data structures designed as reasonable
> defaults, not optimal implementations.

With all due respect, that is a weak argument[1]. Having things under
your own control does not guarantee better (see below) performance. At
best it may carry the potential of delivering better performance.

> I'd hate for someone to think
> that std::unordered_map, say, is somehow better than the data structures
> we've used for a long time merely because the standard stipulates that
> the runtime must provide it.

I guess that depends on the definition of “better”. Regarding run-time
and/or memory usage, it is much more likely that, say, a processor
vendor will contribute a clever, architecture specific implementation
to a compiler's standard library than to Emacsen.

But then I was just raising this point as another potential in the
unlikely event of someone converting the code base to C++, and not in
order to convince anyone.

Regards
Marcus

Footnotes: 
[1] Just as weak as my own attempt to paraphrase a comment whose
source I have been unable to locate all morning :(

Re: Emacs rewrite in a maintainable language

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Date: Mon, 12 Oct 2015 14:48:47 +0200
> Cc: emacs-devel@gnu.org
> 
> I think slowly moving the Emacs C core to C++ is a good idea.  The two
> languages are supposed to be largely compile-compatible, i.e. g++ can
> understand what was meant for gcc.

C++ is a dying language.  On my daytime job we have difficulty
recruiting C++ programmers for the last 2-3 years.  Why should we
invest a non-trivial amount of effort, just to get to another language
that is foreign to most contributors?  It sounds like waste of
precious resources.

> I've been trying to get into understanding Emacs C core, but it's just
> so hard with all those macros around.

A macro is just like a function when used.

Maybe adding more comments will help.  Don't hesitate to ask questions
and point out places where things are hard to understand.

> I think even replacing `#define`
> with `const` would be a huge improvement, worthwhile of the switch.

Standard C supports 'const', and we use it quite a lot in Emacs
sources.

Re: Emacs rewrite in a maintainable language

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> C++ is a dying language.  On my daytime job we have difficulty
> recruiting C++ programmers for the last 2-3 years.

Heh... I remember trying to teach some Java trained summer students
about C++ smart pointers back in 2000 and getting blank stares in
return...

Re: Emacs rewrite in a maintainable language

[David Kastrup]

Steinar Bang <sb@dod.no> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org>:
>
>> C++ is a dying language.  On my daytime job we have difficulty
>> recruiting C++ programmers for the last 2-3 years.
>
> Heh... I remember trying to teach some Java trained summer students
> about C++ smart pointers back in 2000 and getting blank stares in
> return...

"smart pointers"?  If the Oxymoron Series of XEmacs is still ongoing,
that sounds like a worthy release name.

-- 
David Kastrup

Re: Emacs rewrite in a maintainable language

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]

On 10/12/2015 05:48 AM, Oleh Krehel wrote:
> Óscar Fuentes <ofv@wanadoo.es> writes:
> 
>> Replacing C with a language that helps people to write readable, safe,
>> efficient and maintainable code without having to study the existing
>> code base for months looks quite appealing to me. That replacement could
>> be gradual (no "stopping the world while The Big Rewrite is underway.")
>> If that language is not so popular as C, that's no big problem as long
>> as it makes things less daunting for those willing to work on the Emacs
>> core. As already said, there is no dramatic shortage of people who learn
>> Elisp just for hacking Emacs.
> 
> I think slowly moving the Emacs C core to C++ is a good idea.  The two
> languages are supposed to be largely compile-compatible, i.e. g++ can
> understand what was meant for gcc.

Not generally --- C++ has no implicit void cast. I very much think that
this whole thread is a giant waste of time. We have a core that works
perfectly well. We should focus on things that actually matter, not
arguing about things that will never change, or worse, changing massive
amounts of code for no good reason.

I'm firmly opposed to a move to C++. If you want a higher-level
language, use elisp.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs rewrite in a maintainable language

[Przemysław Wojnowski]

W dniu 11.10.2015 o 19:25, John Wiegley pisze:
>>>>>> Przemysław Wojnowski <esperanto@cumego.com> writes:
>
>> Recently I was going through Emacs C code and immediately one thing come to
>> my mind: "Did anyone consider rewrite it a more maintainable language?" :-)
>
> Hi Przemysław,
[...]
> If C presented difficulties in terms of capability, efficiency, contribution,
> debugging, etc., that would naturally drive us toward another language. But as
> it stands, C is well understood, easy to resolve performance issues, there are
> superb debugging and analysis tools available, and the runtime is exceedingly
> mature.
Thank you for comprehensive answer.
I was just asking, not proposing as some people misread.

> And, as some have said, C code represents only a fraction of what Emacs is to
> the user. Switching from Emacs Lisp to another high-level language might be a
> more interesting discussion, which you could take up with the people working
> on such projects, like Guile Emacs. If someone can prove to me that we gain a
> lot and lose a little from such a switch, it's definitely worth considering!

It is an interesting topic, but I was not asking about that. But thanks, anyway!