Change of Lisp syntax for "fancy" quotes in Emacs 27?

Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Noam Postavsky]

In Emacs 26 and earlier the following is valid lisp code:

(setq ’bar 42)
(setq foo ’bar)

In the current master branch, this will signal (invalid-read-syntax
"strange quote" "’"). To write the equivalent the ’ must be backslash
escaped:

(setq \’bar 42)
(setq foo \’bar)

(the backslash escaping also works in earlier Emacs versions).

The point of this change is to give a more straightforward error in
cases where a plain straight quote is accidentally written instead of
a curved one.

In Bug#30217, Drew Adams strongly objects to this change. I don't want
to "sneak" this in, so I'm asking here for people's thoughts on this.

References:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30217
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=2967

PS In case anyone has trouble reading the example code (e.g., due to
some email encoding failure), evaluating

   (insert "(setq \u2019bar 42)\n(setq foo \u2019bar)")

will write it into your current buffer.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

On 02/02/2018 02:24 PM, Noam Postavsky wrote:
> In Bug#30217, Drew Adams strongly objects to this change. I don't want
> to "sneak" this in, so I'm asking here for people's thoughts on this.

I see two main categories of users here, with different needs. 
Less-expert users are likely to run into problems with quotes and other 
characters (that's why we got bug reports), and appreciate diagnostics 
pinpointing the problems; also, programmers concerned about security are 
likely to want these confusing characters to be diagnosed, to prevent an 
attacker from sending code that is easily read one way but actually 
operates in a different way. On the other hand, programs that generate 
Elisp code might prefer not having to special-case these characters. So 
perhaps there should be a buffer-local variable that controls which 
behavior is selected. The default behavior should be the one that caters 
better to general users and is safer.

While we're on the topic, I suggest using the Unicode confusables list 
<http://www.unicode.org/Public/security/10.0.0/confusables.txt> to come 
up with a list of confusing alternatives for each character that has a 
special meaning in Emacs Lisp. This should be better than our trying to 
come up with our own, ad-hoc list. For example, U+A78C LATIN SMALL 
LETTER SALTILLO (ꞌ) looks almost exactly like an apostrophe on my screen 
and is in the confusables list, but is not a character that Emacs 
currently checks for.

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

> I see two main categories of users here, with different needs.
> Less-expert users are likely to run into problems with quotes
> and other characters (that's why we got bug reports), and
> appreciate diagnostics pinpointing the problems; also,
> programmers concerned about security are likely to want these
> confusing characters to be diagnosed, to prevent an attacker
> from sending code that is easily read one way but actually
> operates in a different way.
>
> On the other hand, programs that generate Elisp code might
> prefer not having to special-case these characters. So
> perhaps there should be a buffer-local variable that controls
> which behavior is selected. The default behavior should be
> the one that caters better to general users and is safer.

The distinction I think needs to be made is between:

1. Trying to _warn users_ (all users, less-expert or not)
   about possible misuse of particularly confusable chars.
   This just warns about possible pilot error.

2. _Changing Lisp_ reading and evaluating, to treat some
   (all?) confusable characters specially, changing their
   syntax and requiring them to be escaped in order to be
   treated normally (i.e., as they have been treated so far).

I object to #2, NOT to #1.

#1: By all means, we should try to help users.  We can
    issue byte-compilation warnings and some interactive
    warnings - provided we can helpfully and unambiguously
    distinguish the right situations.

#2 changes Lisp in non-neglible, non-helpful ways.
   See bug #30217 for more.

----

There are lots more characters to which the same
non-bug "fix" of changing Lisp might be applied (which
means that users will wonder why this confusable char
is treated specially, and not that one).

Such chars include pretty much anything that could be
confused with anything that is ever used as a delimiter
in Emacs Lisp: brackets (in the British sense) of all
sorts: parens, square, angle, curly.  There are really
quite a few such bracket-confusables.

Such chars also include pretty much anything that could
be confused with any other chars that are used specially
in Lisp: period, comma, quote, backquote, colon.  Again:
there are quite a few such confusables.

They even include chars that could be confused with the
directory separators used in Emacs Lisp.

Finally (?), they include chars that could be confused
with the ASCII-digit numerals 0123456789.  There are
lots of these confusables too.

(Even with just ASCII there are confusables.  Think of
what some use in passwords or leet: zero vs uppercase
letter O, digit 1 vs lowercase letter l, etc.  We've
just gotten used to carefully distinguishing such chars.
Now there are many more, and slighter, differences to
get used to.)

----

Beyond the question of which chars to treat specially,
there's the question of where - in which contexts -
to try to distinguish them.

Contexts include such places as sexps being evaluated,
doc strings, and comments.

They can also include fonts: a given character might
be confusable, or more confusable, in one font than
in another.  Even font size can make a difference
(with some fonts I find myself zooming in to see
whether a quote-thingy might really be a curly quote).

The questions of which chars and where (context) are
both relevant even if we only warn users (#1) and do
not change Lisp syntax (#2).

----

At the very least, I would hope that if we do anything
at all about this we would start by only warning.
I really hope we will not change Lisp syntax for this,
i.e., I hope we revert the change that has been made so
far for Emacs 27.

> While we're on the topic, I suggest using the Unicode
> confusables list ... to come up with a list of confusing
> alternatives for each character that has a special meaning
> in Emacs Lisp. This should be better than our trying to
> come up with our own, ad-hoc list.
>
> For example, U+A78C LATIN SMALL LETTER SALTILLO (ꞌ) looks
> almost exactly like an apostrophe on my screen and is in
> the confusables list, but is not a character that Emacs
> currently checks for.

Yup, and that's just one tiny tip of this terribly
tippy iceberg.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

On 02/02/2018 04:00 PM, Drew Adams wrote:
> The distinction I think needs to be made is between:
>
> 1. Trying to_warn users_  (all users, less-expert or not)
>     about possible misuse of particularly confusable chars.
>     This just warns about possible pilot error.
>
> 2._Changing Lisp_  reading and evaluating, to treat some
>     (all?) confusable characters specially, changing their
>     syntax and requiring them to be escaped in order to be
>     treated normally (i.e., as they have been treated so far).
>
> I object to #2, NOT to #1.

I don't see a clear distinction between #1 and #2. For example, in an 
adversarial environment, users who get warned about suspicious 
characters in their incoming source files will most likely type "no" 
when asked to run such code. In that case, if you want your audience to 
include users who care even a smidgen about security, you'll need to 
escape confusable characters in the business parts of your Emacs Lisp 
code. Effectively that will be a change to Emacs Lisp, even if its 
formal syntax does not change.

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

> > The distinction I think needs to be made is between:
> >
> > 1. Trying to_warn users_  (all users, less-expert or not)
> >     about possible misuse of particularly confusable chars.
> >     This just warns about possible pilot error.
> >
> > 2._Changing Lisp_  reading and evaluating, to treat some
> >     (all?) confusable characters specially, changing their
> >     syntax and requiring them to be escaped in order to be
> >     treated normally (i.e., as they have been treated so far).
> >
> > I object to #2, NOT to #1.
> 
> I don't see a clear distinction between #1 and #2.

That's too bad.  They are really quite different.

In the first case, you get a warning.  In the second case
your code breaks.

> For example, in an adversarial environment...

I don't think that's the reason for this change at all.
It was not mentioned in the bug thread, AFAIK.

The motivation was to prevent confusion on the part of
users, not to prevent or avoid malevolent behavior.
Please see the bug thread (#30217).

The idea was to improve convenience and reduce confusion
by someone who copy+pastes code from a web page (for
example), when (for example) that page renders a normal
quote as a curly quote.

You want to introduce a security aspect here.  I can't
speak much to that.  I'll simply ask whether other Lisps
(e.g. Common Lisp) worry about such a risk?  What does
Clojure do about confusables in Lisp symbols?  Does any
other Lisp change the Lisp syntax and behavior to require
special escaping of such chars in symbols (or elsewhere)?

Sure, even if no other Lisp worries about this or takes
the same approach as that proposed, that's not a proof
that Emacs Lisp shouldn't.  Still...

Given enough motivation, you can already, today, create
Lisp code (confusing, confusable, or otherwise) that is
evil, even without using any consusable Unicode chars.

When I was a kid we would play tricks on each other,
changing a character somewhere in a friend's large deck
of punched Hollerith cards - e.g., insert or remove a
decimal point.  You had to wait a full day to get back
the result of your program run, and the result would
only be a pretty cryptic error msg.  Argggh!

It was just good-natured fun - a game among friends.
And that was only with assembler and Fortran, and we
were just newbie kids.  Imagine what you can do today,
without bothering to rely on close Unicode confusables.

Sorry, but your "security" argument just doesn't pass
muster, for me.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Fri, 2 Feb 2018 17:24:43 -0500
> Cc: Drew Adams <drew.adams@oracle.com>
> 
> In Emacs 26 and earlier the following is valid lisp code:
> 
> (setq ’bar 42)
> (setq foo ’bar)
> 
> In the current master branch, this will signal (invalid-read-syntax
> "strange quote" "’"). To write the equivalent the ’ must be backslash
> escaped:
> 
> (setq \’bar 42)
> (setq foo \’bar)
> 
> (the backslash escaping also works in earlier Emacs versions).
> 
> The point of this change is to give a more straightforward error in
> cases where a plain straight quote is accidentally written instead of
> a curved one.

The bug reports which triggered the above changes are bug#2967 and
bug#23425.  So any proposal to remove those changes should also
suggest an alternative for handling those bug reports.

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

> The bug reports which triggered the above changes are bug#2967 and
> bug#23425.  So any proposal to remove those changes should also
> suggest an alternative for handling those bug reports.

For "handling those bug reports"?  Are we to add
more cans of worms to this question, obscuring it?

AFAICT, no alternatives to handling those bugs
are needed because of reverting the Lisp syntax
change made for bug #30217.  Can you point to
how/why reverting that change would necessitate
alternative fixes for those bugs?

Bug #2967 just asked for a warning, e.g. during
byte-compilation or loading.  There's no
objection here to warning.

Bug #2967 did not ask for (or get) a change in
Lisp syntax.  I see no negative impact on #2967
from reverting the Lisp-syntax "fix" to #30217.

Even #30217 did not ask for such a syntax change.
Warning is sufficient for fixing #30217 too.

Bug #23425, on the other hand, is a gigantic
stream-of-consciousness about anything and
everything to do with Paul's changes to Emacs
over the last few years wrt curly quotes.
It's not a single bug report thread - it's
all over the map.

In any case, #23425, like #2967 (and even
#30217), is not about what was done to "fix"
#30217 - changing Lisp syntax for fancy quotes.

How is it helpful to throw all of #23425 into
this Lisp syntax-change question, as if the
present issue puts into question everything
ever discussed about curly quotes?

Or do you have something specific in mind here
wrt #23425 - some part of it?  Something that
would actually be impacted negatively by
reverting the Lisp syntax changes for #30217?
If so, please identify it.

But if you mean only the ability to get confused
by copy+pasting Lisp code that has a fancy quote
mark somewhere in place of ordinary ASCII
apostrophe ('), e.g., (setq foo ’bar), then
that's just the same pilot-error gotcha as for
bug #30217.

There are many gotchas in Lisp.  You can see
repeated postings of some at various places
(e.g., help-gnu-emacs, emacs.stackexchange).
E.g., the error that a given Lisp function is
not defined (because its library was not loaded).

The pilot error described in bug #30217 is not
even a commonly reported one.  The "fix" made
in #30217 is an overreaction.

So one solution to #30217 is to do nothing - just
revert the misguided Lisp syntax change.  Users
will learn that gotcha the same way they learn
others.  Not every report of a gotcha needs to
lead to changes to Emacs.

If we do nothing there will continue to be some
such pilot errors, of course.  But we already
raise an error if the code leads to a problem.

And the original error message from bug #23425
is _more_ meaningful and helpful, not less,
than the new one after the "fix".

The original error msg of #23425:
  (wrong-number-of-arguments setq 31)

tells you pretty much that setq is missing an
argument or it has too many, which makes you
look at its arguments.  Not so obscure.  And
accurate.

The new error msg:
  (invalid-read-syntax "strange quote" "’")

is obscure.  Invalid read syntax when reading
what?  What's invalid about it?

Confusion - not understanding an accurate error
msg, is not the same thing as Lisp itself having
a bug because such a character is included in a
symbol name.

Another solution is to try to warn users about
the use of confusables.

That's actually many solutions, because it
requires handling different chars and different
gotcha contexts differently, and carefully.
But unlike a syntax change it's not an
all-or-nothing thing: we could add warnings here
and there, as something might be better than
nothing.

Either doing nothing or trying to warn about such
gotchas is right.  Changing Lisp syntax here is
not right.  Lisp doesn't have a bug here.

This is all about pilot error - the same kind of
thing that happens when someone mistypes `,' for
`.' for dotted-pair syntax, or types `.' in `a.b'
intending dotted-pair syntax but getting a symbol
instead, or quotes a sexp expecting the sexp to
be evaluated.

Yes, a user might scratch her head when seeing
the error message from such a mistake, but the
error message is right, not wrong, and eventually
the light turns on.

And this enlightenment is aided by the fact that
Lisp syntax is so simple.  The "fix" for bug
#30217 goes in the opposite direction.  It makes
Lisp syntax more complex and makes understanding
syntax mistakes more difficult.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> Date: Sat, 3 Feb 2018 08:16:15 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: emacs-devel@gnu.org
> 
> > The bug reports which triggered the above changes are bug#2967 and
> > bug#23425.  So any proposal to remove those changes should also
> > suggest an alternative for handling those bug reports.
> 
> For "handling those bug reports"?  Are we to add
> more cans of worms to this question, obscuring it?
> 
> AFAICT, no alternatives to handling those bugs
> are needed because of reverting the Lisp syntax
> change made for bug #30217.  Can you point to
> how/why reverting that change would necessitate
> alternative fixes for those bugs?

Those bug reports complained about obscure error messages that are
unhelpful when a Lisp programmer tries to figure out the root cause.
I'm saying that we should find an alternative way of making clear,
helpful error messages in those special cases where characters which
display similarly might make the error message confusing if it just
cites the symbol's name.

For example, suppose you have a Lisp program that produces the
following error message when compiled/executed:

  Symbol's value as variable is void: 'аbbrevs-changed

You then type "C-h v abbrevs-changed RET" and get the expected result,
meaning that the variable is known to Emacs.  How quickly will you be
able to spot the cause of the error message?

The change that got reverted from the emacs-26 branch was about a
similar case, but for a character that's much more important for Lisp
than 'a': it's about the character used to quote symbol names.  But
the essence is the same: due to how characters are displayed, some
characters can be confused for others.

We want to find a way of identifying such situation and telling the
Lisp programmer about that in clear and easily understandable ways.
One way, perhaps too radical one, is to reject such "confusable"
characters outright.  We could decide that we don't want such a
radical solution, but that doesn't mean we should give up on the
attempt to find some other solution for the problem.  Neither does it
mean we should proclaim people who installed the change as enemies of
the society.

> Bug #23425, on the other hand, is a gigantic
> stream-of-consciousness about anything and
> everything [...]
> [...]
> How is it helpful to throw all of #23425 into
> this Lisp syntax-change question, as if the
> present issue puts into question everything
> ever discussed about curly quotes?

I could turn the table and ask you how is it helpful to dump on us all
your random thoughts about this, instead of simply saying you didn't
understand the relevance and asking for more explanations.  Which I
just provided.

I hope now the issue is clear enough.

> And the original error message from bug #23425
> is _more_ meaningful and helpful, not less,
> than the new one after the "fix".
> 
> The original error msg of #23425:
>   (wrong-number-of-arguments setq 31)
> 
> tells you pretty much that setq is missing an
> argument or it has too many, which makes you
> look at its arguments.  Not so obscure.  And
> accurate.
> 
> The new error msg:
>   (invalid-read-syntax "strange quote" "’")
> 
> is obscure.  Invalid read syntax when reading
> what?  What's invalid about it?

I think you are so eager to make your point that you are willing to
claim that black is white and vice versa.  Any objective person would
agree that the new error message is more directly pointing to the root
cause, which is the syntax of specifying a quoted symbol name using a
"strange quote".  If we are good in writing and indexing our ELisp
manual, then I'd expect to find there an index entry for "strange
quote", which will land me where this issue is explained.  Case
closed.

Once again, I can agree that this measure might be too harsh, but I
would still like to see clear diagnostics of such typos, and like
Paul, I thing we should take our inspiration from the Unicode
Standard's notion of "confusables".  Ideas and proposals for patches
along those lines are welcome.  Ignoring the problem, or trying to
convince us that it doesn't exist, is not.

> Either doing nothing or trying to warn about such
> gotchas is right.  Changing Lisp syntax here is
> not right.

Doing nothing would be ignoring the problem.  That changing Lisp
syntax is not right is your opinion: legitimate, but clearly not
shared by at least some.

> Lisp doesn't have a bug here.

That's a strawman, and you know it.  We are talking about diagnostics
for bugs in Lisp programs.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Michael Heerdegen]

Hello,

Helpfulness of error messages surely depends on the beholder, and on
expectations.  In my eyes,

> Symbol's value as variable is void: 'аbbrevs-changed

is quite clear: you think this        ^^^^^^^^^^^^^^^^ is a quoted
thing, but the error message calls it a symbol.  So there must be a
problem with that quote, it has obviously gotten read as part of the
symbol.  Sure, you have still to find out why.  OTOH

> >   (invalid-read-syntax "strange quote" "’")

also doesn't say what's wrong with that quote.  It even calls something
a quote where there is none.  The error message is confusing.  Repeating
the pseudo quote character in the error message doesn't make it look
less like a quote.

> I think you are so eager to make your point that you are willing to
> claim that black is white and vice versa.  Any objective person would
> agree that the new error message is more directly pointing to the root
> cause

Are you really sure that every Emacs user would expect that we modify
the Lisp reader to catch typos?

FWIW, we already modified the Lisp reader to catch another style issue
(to get rid of old-style backquotes) and made it error.  It broke my
stuff (el-search) horribly - though I don't use old-style backquotes,
and for code that also doesn't use them.  Now I need to work around
`read' and define my own `read' function.  I also need to remember for a
long time that using `read' is forbidden in my library.  I even
implemented a minor mode to warn me just about that: it warns me that I
use `read' and it's forbidden.  Otherwise, I would get strange errors
when using my stuff, from time to time, whenever I added a `read' by
accident.  All other users of my package, too.  And believe me, _these_
error messages are then less understandable than

> Symbol's value as variable is void: 'аbbrevs-changed.

Misusing something fundamental as the Lisp reader to catch such stuff
should be the very last resort.  The result can get much more confusing
in situations we now don't think about.

> > Lisp doesn't have a bug here.
> That's a strawman, and you know it.  We are talking about diagnostics
> for bugs in Lisp programs.

I think it's a eligible argument.  Drew just thinks it's the wrong fix.
He may also think that no fix would maybe suffice.  That's ok, and I
think he made some good points.

We should discuss about alternative approaches to move forward.  People
often paste stuff into scratch or the M-: prompt that they copied from
elsewhere.  Maybe we could make M-: and C-x C-e check for this problem.
These could also check for other, similar frequent problems.  Any better
suggestions?


Michael.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Clément Pit-Claudel]

On 2018-02-03 20:16, Michael Heerdegen wrote:
> Helpfulness of error messages surely depends on the beholder, and on
> expectations.  In my eyes,
> 
>> Symbol's value as variable is void: 'аbbrevs-changed
> is quite clear: you think this        ^^^^^^^^^^^^^^^^ is a quoted
> thing, but the error message calls it a symbol.  So there must be a
> problem with that quote, it has obviously gotten read as part of the
> symbol.  Sure, you have still to find out why.

I think you're making Eli's point, actually :)

The problem isn't the quote: it's the CYRILLIC SMALL LETTER A instead of LATIN SMALL LETTER A.  IOW, (string= "аbbrevs-changed" "abbrevs-changed") is nil.

I think Eli was illustrating the confusion that can stem from Unicode confusables (and I must agree that the error message could be much better ^^)

Clément.

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

> > Helpfulness of error messages surely depends on the beholder, and on
> > expectations.  In my eyes,
> >
> >> Symbol's value as variable is void: 'аbbrevs-changed
> > is quite clear: you think this        ^^^^^^^^^^^^^^^^ is a quoted
> > thing, but the error message calls it a symbol.  So there must be a
> > problem with that quote, it has obviously gotten read as part of the
> > symbol.  Sure, you have still to find out why.
> 
> I think you're making Eli's point, actually :)
> 
> The problem isn't the quote: it's the CYRILLIC SMALL LETTER A instead of
> LATIN SMALL LETTER A.  IOW, (string= "аbbrevs-changed" "abbrevs-
> changed") is nil.
> 
> I think Eli was illustrating the confusion that can stem from Unicode
> confusables (and I must agree that the error message could be much
> better ^^)

I too misread Eli's example as being about using a
curly quote instead of an apostrophe.  You're right
that it's an ordinary apostrophe and the first `a'
is the letter you mention.

But then why would anyone ever see the quote mark
in such a message?  Was the message artificially
configured?

In any case, if that example, without the quote, say,
is trying to make Eli's point, then he must be arguing
for warning about using such confusables also - `а'
as a confusable for `a'.

That's a monumental undertaking (take a look at the
confusables.txt list).  And the messages (warning or
error) would need to be pretty darn clear about just
what char was used and where, in order not to sow
even more confusion.  It sure won't cut the mustard
to just say "Invalid read syntax"!

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Michael Heerdegen]

Clément Pit-Claudel <cpitclaudel@gmail.com> writes:

> On 2018-02-03 20:16, Michael Heerdegen wrote:
> > Helpfulness of error messages surely depends on the beholder, and on
> > expectations.  In my eyes,
> > 
> >> Symbol's value as variable is void: 'аbbrevs-changed
> > is quite clear: you think this        ^^^^^^^^^^^^^^^^ is a quoted
> > thing, but the error message calls it a symbol.  So there must be a
> > problem with that quote, it has obviously gotten read as part of the
> > symbol.  Sure, you have still to find out why.
>
> I think you're making Eli's point, actually :)
>
> The problem isn't the quote: it's the CYRILLIC SMALL LETTER A instead
> of LATIN SMALL LETTER A.  IOW, (string= "аbbrevs-changed"
> "abbrevs-changed") is nil.

Oh.  Why is then there a quote in this error message?

FWIW, I'm not against doing something that helps the user in such
situations.  But these are problems in the interaction between the user
and Emacs, so we should care about it on that (the interface) level.
And keep Lisp, the language, simple.


Michael.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Alan Third]

On Sat, Feb 03, 2018 at 08:25:01PM -0500, Clément Pit-Claudel wrote:
> On 2018-02-03 20:16, Michael Heerdegen wrote:
> > Helpfulness of error messages surely depends on the beholder, and on
> > expectations.  In my eyes,
> > 
> >> Symbol's value as variable is void: 'аbbrevs-changed
> > is quite clear: you think this        ^^^^^^^^^^^^^^^^ is a quoted
> > thing, but the error message calls it a symbol.  So there must be a
> > problem with that quote, it has obviously gotten read as part of the
> > symbol.  Sure, you have still to find out why.
> 
> I think you're making Eli's point, actually :)
> 
> The problem isn't the quote: it's the CYRILLIC SMALL LETTER A
> instead of LATIN SMALL LETTER A. IOW, (string= "аbbrevs-changed"
> "abbrevs-changed") is nil.
> 
> I think Eli was illustrating the confusion that can stem from
> Unicode confusables (and I must agree that the error message could
> be much better ^^)

Something like:

Symbol's value as variable is void: 'аbbrevs-changed
Did you mean `abbrevs-changed'?
Symbol contains `а' (CYRILLIC SMALL LETTER A) at character 0, did you
mean `a' (LATIN SMALL LETTER A)?

The middle line would require Emacs to do a fuzzy search for similar
symbols, which may be too much. Something like that could be helpful
even in cases where the name has been mistyped (abbrev-changed instead
of abbrevs-changed, for example).

-- 
Alan Third

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Clément Pit-Claudel]

On 2018-02-04 05:34, Alan Third wrote:
> Symbol's value as variable is void: 'аbbrevs-changed
> Did you mean `abbrevs-changed'?
> Symbol contains `а' (CYRILLIC SMALL LETTER A) at character 0, did you
> mean `a' (LATIN SMALL LETTER A)?

That would be pretty cool.

> The middle line would require Emacs to do a fuzzy search for similar
> symbols, which may be too much.

OCaml does this (but at compile time).  Do we have a way to delay the fuzzy search to the point when the error message is displayed?  Otherwise we'll pay the price of the search even if the error is then swallowed by a condition-case.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> Cc: Michael Heerdegen <michael_heerdegen@web.de>, Eli Zaretskii
>  <eliz@gnu.org>, emacs-devel@gnu.org, Drew Adams <drew.adams@oracle.com>,
>  npostavs@users.sourceforge.net
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Sun, 4 Feb 2018 10:36:49 -0500
> 
> > The middle line would require Emacs to do a fuzzy search for similar
> > symbols, which may be too much.
> 
> OCaml does this (but at compile time).  Do we have a way to delay the fuzzy search to the point when the error message is displayed?  Otherwise we'll pay the price of the search even if the error is then swallowed by a condition-case.

Isn't this premature optimization?  We aren't even sure yet that such
a fuzzy search will be too expensive.  We could, for example,
implement the confusables as a char-table, which would make it fast
enough, I think.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 804 bytes --]

On Sun, Feb 4, 2018 at 12:37 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Sun, 4 Feb 2018 10:36:49 -0500
>>
>> > The middle line would require Emacs to do a fuzzy search for similar
>> > symbols, which may be too much.
>>
>> OCaml does this (but at compile time).  Do we have a way to delay the fuzzy search to the point when the error message is displayed?  Otherwise we'll pay the price of the search even if the error is then swallowed by a condition-case.
>
> Isn't this premature optimization?

I think the check fits nicely into command-error-default-function
though. Attaching a quick proof-of-concept (handles only a single
curved quote at the beginning of symbol name). We would want something
also for the byte-compiler.

[-- Attachment #2: v1-0001-sketch-Catch-strange-quotes-on-error-time.patch --]
[-- Type: text/x-diff, Size: 3166 bytes --]

From c9d1e761cea56e94d9ad3d783c8ed7fcf448b082 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Sun, 4 Feb 2018 16:20:32 -0500
Subject: [PATCH v1] [sketch] Catch strange quotes on error time

* src/keyboard.c (Fcommand_error_default_function): Check for RIGHT
SINGLE QUOTATION MARK and give a more detailed message.  TODO: check
for other confusables.
* src/lread.c (read1): Don't signal error on strange quotes.
* src/eval.c (Fsetq): Pass full arglist in error data.  TODO: the same
for all the other Qwrong_number_of_arguments cases.
---
 src/eval.c     |  3 ++-
 src/keyboard.c | 20 ++++++++++++++++++++
 src/lread.c    |  7 -------
 3 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/src/eval.c b/src/eval.c
index 7db4dbcf18..db61c0421f 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -507,7 +507,8 @@ DEFUN ("setq", Fsetq, Ssetq, 0, UNEVALLED, 0,
       Lisp_Object sym = XCAR (tail), lex_binding;
       tail = XCDR (tail);
       if (!CONSP (tail))
-	xsignal2 (Qwrong_number_of_arguments, Qsetq, make_number (nargs + 1));
+        xsignal3 (Qwrong_number_of_arguments, Qsetq,
+                  make_number (nargs + 1), args);
       Lisp_Object arg = XCAR (tail);
       tail = XCDR (tail);
       val = eval_sub (arg);
diff --git a/src/keyboard.c b/src/keyboard.c
index 4324991da4..24c5f66934 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -1047,6 +1047,26 @@ DEFUN ("command-error-default-function", Fcommand_error_default_function,
       bitch_at_user ();
 
       print_error_message (data, Qt, SSDATA (context), signal);
+
+      Lisp_Object errname = Fcar (data);
+      /* TODO: Add arglist to Qwrong_number_of_arguments errors, and
+         check those too.  */
+      if (EQ (errname, Qvoid_variable))
+        {
+          Lisp_Object void_symname = Fsymbol_name (Fnth (make_number (1), data));
+          if (SCHARS (void_symname) > 0 &&
+              /* TODO: check all confusables.  */
+              EQ (Faref (void_symname, make_number (0)), make_number (0x2019)))
+            {
+              Lisp_Object msg = CALLN
+                (Fformat_message,
+                 build_string ("\nSymbol has with `%c' (%s) at character 0,"
+                               " did you mean `%c' (%s)"),
+                 make_number (0x2019), build_string ("RIGHT SINGLE QUOTATION MARK"),
+                 make_number ('\''), build_string ("APOSTROPHE"));
+              Fprinc (msg, Qt);
+            }
+        }
     }
   return Qnil;
 }
diff --git a/src/lread.c b/src/lread.c
index 3b0a17c90b..ee08902f81 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -3470,13 +3470,6 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 	    if (! NILP (result))
 	      return unbind_to (count, result);
 	  }
-        if (!quoted && multibyte)
-          {
-            int ch = STRING_CHAR ((unsigned char *) read_buffer);
-            if (confusable_symbol_character_p (ch))
-              xsignal2 (Qinvalid_read_syntax, build_string ("strange quote"),
-                        CALLN (Fstring, make_number (ch)));
-          }
 	{
 	  Lisp_Object result;
 	  ptrdiff_t nbytes = p - read_buffer;
-- 
2.11.0

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Alan Mackenzie]

Hello, Michael.

On Sun, Feb 04, 2018 at 02:16:52 +0100, Michael Heerdegen wrote:
> Hello,

> Helpfulness of error messages surely depends on the beholder, and on
> expectations.  In my eyes,

> > Symbol's value as variable is void: 'аbbrevs-changed

> is quite clear: you think this        ^^^^^^^^^^^^^^^^ is a quoted
> thing, but the error message calls it a symbol.  So there must be a
> problem with that quote, it has obviously gotten read as part of the
> symbol.  Sure, you have still to find out why.  OTOH

This has actually happened to me.  In the error message, I didn't see
the quote as part of the symbol, I subconsciously dismissed it as a
quoting convention in the error message.  So what my brain saw was

    Symbol's value as variable is void: abbrevs-changed

.  This puzzled me a long time.

> > >   (invalid-read-syntax "strange quote" "’")

> also doesn't say what's wrong with that quote.  It even calls something
> a quote where there is none.

Perhaps "strange quasi quote" would be more emphatic and clearer.

> The error message is confusing.  Repeating the pseudo quote character
> in the error message doesn't make it look less like a quote.

Agreed, on both points.

> > I think you are so eager to make your point that you are willing to
> > claim that black is white and vice versa.  Any objective person would
> > agree that the new error message is more directly pointing to the root
> > cause

> Are you really sure that every Emacs user would expect that we modify
> the Lisp reader to catch typos?

We're not talking about typos here.  The curly quotes aren't present on
typical keyboard layouts (though I'm informed they are present on
Finnish keyboards), so nobody who isn't Finnish will type one of these
characters by accident.  We're talking about Emacs itself corrupting
ASCII quotes into curly quotes in a `message' call because of the
default setting of `text-quoting-style', and so on.

Because of this, the error message should concentrate on that quote, not
the strange symbol, which Emacs itself created.

[ .... ]

> > Symbol's value as variable is void: 'аbbrevs-changed.

> Misusing something fundamental as the Lisp reader to catch such stuff
> should be the very last resort.  The result can get much more confusing
> in situations we now don't think about.

Maybe we're already at the last resort for this problem.  Or maybe not.
Maybe an error message for unknown symbols should check for them
beginning with a curly quote.

> > > Lisp doesn't have a bug here.
> > That's a strawman, and you know it.  We are talking about diagnostics
> > for bugs in Lisp programs.

> I think it's a eligible argument.  Drew just thinks it's the wrong fix.
> He may also think that no fix would maybe suffice.  That's ok, and I
> think he made some good points.

> We should discuss about alternative approaches to move forward.  People
> often paste stuff into scratch or the M-: prompt that they copied from
> elsewhere.  Maybe we could make M-: and C-x C-e check for this problem.
> These could also check for other, similar frequent problems.  Any better
> suggestions?

I think that's a good suggestion.

> Michael.

-- 
Alan Mackenzie (Nuremberg, Germany).

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

> We're not talking about typos here.  The curly quotes aren't present on
> typical keyboard layouts (though I'm informed they are present on
> Finnish keyboards), so nobody who isn't Finnish will type one of these
> characters by accident.  We're talking about Emacs itself corrupting
> ASCII quotes into curly quotes in a `message' call because of the
> default setting of `text-quoting-style', and so on.
> 
> Because of this, the error message should concentrate on that quote, not
> the strange symbol, which Emacs itself created.

Not necessarily.  Although I share your concern about
Emacs promulgating curly quotes, there is a real usage
problem akin to "typos": users copying text, including
Lisp code, from a web page or elsewhere, and pasting
it into Emacs as code to be evaluated at some point.

If the source of the copy has already changed simple
apostrophe to a curly quote (as one example) then that's
what gets passed to Emacs.  The person copy+pasting
may well have no clue about just which characters are
being copied.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Noam Postavsky]

On Sat, Feb 3, 2018 at 8:16 PM, Michael Heerdegen
<michael_heerdegen@web.de> wrote:

> FWIW, we already modified the Lisp reader to catch another style issue
> (to get rid of old-style backquotes) and made it error.  It broke my
> stuff (el-search) horribly - though I don't use old-style backquotes,
> and for code that also doesn't use them.

That backquote change made `read' signal errors when reading
subexpressions of otherwise valid code.

The change under discussion changes what is valid code, so you won't
have the problem of getting read errors for valid code.
(changing what is valid Lisp has other drawbacks, as Drew has
repeatedly pointed out)

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

> Those bug reports complained about obscure error messages that are
> unhelpful when a Lisp programmer tries to figure out the root cause.
> I'm saying that we should find an alternative way of making clear,
> helpful error messages in those special cases where characters which
> display similarly might make the error message confusing if it just
> cites the symbol's name.

OK.  Except I would say warnings, not error messages, at
least in most cases.

But even if we have an error message, that's not a call
to change the syntax of Lisp.  User errors happen.  We
should just want to help users avoid making such errors.

> For example, suppose you have a Lisp program that produces
> the following error message when compiled/executed:
> 
>   Symbol's value as variable is void: 'аbbrevs-changed
> 
> You then type "C-h v abbrevs-changed RET" and get the expected result,
> meaning that the variable is known to Emacs.  How quickly will you be
> able to spot the cause of the error message?

Some people will wonder for a while.  Others, perhaps
already bitten by this gotcha, will notice the quote
mark there right away.

One thing that would help, I think, and which should
be done in general, would be to put the offending
thingie between `...':

 Symbol's value as variable is void: `'аbbrevs-changed'

That makes it more obvious that the symbol name
includes that fancy quote char.

Still, all of this is pilot error, where "pilot" can
include the user who wrote the code but more likely
means a user who copy+pasted it.

> The change that got reverted from the emacs-26 branch was about a
> similar case, but for a character that's much more important for Lisp
> than 'a': it's about the character used to quote symbol names.  But
> the essence is the same: due to how characters are displayed, some
> characters can be confused for others.
> 
> We want to find a way of identifying such situation and telling the
> Lisp programmer about that in clear and easily understandable ways.
> One way, perhaps too radical one, is to reject such "confusable"
> characters outright.  We could decide that we don't want such a
> radical solution, but that doesn't mean we should give up on the
> attempt to find some other solution for the problem.  Neither does it
> mean we should proclaim people who installed the change as enemies of
> the society.

Agreed.  As I've said, I'm in favor of providing
friendly warnings/reminders that point out that
such a character is present.

I think that should be enough.

There are lots of potential confusables, and lots
of different use contexts.  But if we start with
just one or two such chars and one or two common
and clear contexts where a warning might help, that
would be good.  We can always add more such warnings
as cases come up (get reported or otherwise become
obvious).

It would be an overreaction, IMO, to jump to
changing the existing Lisp syntax to raise errors
when someone uses such a character in, say, a symbol
name.  We should not require such chars to be
escaped in a symbol name.  Such chars have no special
meaning for Lisp (unlike `.', `,' `'', ``', `(', `)',
`[', `]', `"', `<', `>', `#' `;', and perhaps some more).

> > Bug #23425, on the other hand, is a gigantic
> > stream-of-consciousness about anything and
> > everything [...]
> > [...]
> > How is it helpful to throw all of #23425 into
> > this Lisp syntax-change question, as if the
> > present issue puts into question everything
> > ever discussed about curly quotes?
> 
> I could turn the table and ask you how is it helpful
> to dump on us all your random thoughts about this,
> instead of simply saying you didn't understand the
> relevance and asking for more explanations.  Which I
> just provided.

Whoa!  I don't see a connection between the current
issue and the many things discussed in #23425.  And
I don't think I dumped any random thoughts on anyone.

> I hope now the issue is clear enough.

No idea what your point is there.

If there is some part of bug #23425 that you think
is relevant here, and you think it will be UNfixed by
reverting the Lisp-syntax change made for bug #30217,
please tell us what that part is.

I don't see anything in #23425 that needs the change
in Lisp syntax made for #30217.  And I don't see that
Lisp change being necessary to fix #30217 either.
It wasn't requested by the bug filer, AFAIK.  Same
for the other bugs you mentioned.  The filers just
asked for warnings, AFAICT.

> > And the original error message from bug #23425
> > is _more_ meaningful and helpful, not less,
> > than the new one after the "fix".
>
> I think you are so eager to make your point that you are willing to
> claim that black is white and vice versa.  Any objective person would
> agree that the new error message is more directly pointing to the root
> cause, which is the syntax of specifying a quoted symbol name using a
> "strange quote".  If we are good in writing and indexing our ELisp
> manual, then I'd expect to find there an index entry for "strange
> quote", which will land me where this issue is explained.  Case
> closed.

We can perhaps agree to disagree about that.
But of course if you say the case is closed then
it's closed.

> Once again, I can agree that this measure might be too harsh, but I
> would still like to see clear diagnostics of such typos, and like
> Paul, I thing we should take our inspiration from the Unicode
> Standard's notion of "confusables".

I've agreed about that from the beginning.  It can
be helpful to warn users about possible confusion
when they use confusables.  And I agree that clear
diagnostics are needed - that was one of my points.

That's different from changing the syntax of Lisp.

> Ideas and proposals for patches along those lines
> are welcome.

Ditto.

> Ignoring the problem, or trying to convince us
> that it doesn't exist, is not.

I recognize the problems of confusable characters.
Not all such possible confusions are equally likely,
in practice.

Recognizing contexts where something might well be
a typo, and warrants a helpful reminder/warning, is
what's needed - case by case.

What's not needed, IMO (and probably the only place
where I differ from you on this, even if you don't
want to recognize it) is a change in Lisp syntax,
making it a read error not to escape such a character.

> > Either doing nothing or trying to warn about such
> > gotchas is right.  Changing Lisp syntax here is
> > not right.
> 
> Doing nothing would be ignoring the problem.

Yes.  It's maybe not the best help for users, but
it would be one way to handle those few reports of
confusion.  We get a lot more questions due to
other confusions wrt Lisp than we do such questions
due to confusing one char for another.

I didn't, and don't, say that doing nothing is the
best approach.  I said it's one way to deal with
such reports.  Unlike changing Lisp syntax, it at
least doesn't introduce new problems.

> That changing Lisp syntax is not right is your
> opinion: legitimate, but clearly not shared by at
> least some.

That's why we're having this discussion.

I have yet to hear a reason why it is right to
change Lisp syntax for this - why a simple warning
is not sufficient and we need to also make Lisp
raise an error.

> > Lisp doesn't have a bug here.
> 
> That's a strawman, and you know it.  We are talking
> about diagnostics for bugs in Lisp programs.

I have no objection to diagnostics.  Add warnings
for byte-compilation, loading, whatever.

Make sure the warnings are clear.  Say, for instance
that a curly quote was used in sexp `...'.  Don't
just say that invalid syntax was read (somewhere).
Clearly pointing out the confusable char in the
possibly confused sexp should go a long way to
making things clear.

My objection is to making such chars be escaped to
prevent Lisp from raising an error.  I don't put
`a’b' in the same class as, say, `a,b'.

`,' is special in Lisp, and (setq a,b 42) should
(and does) raise an error.  `’' is not special in
Lisp, and (setq a’b 42) should not raise an error (IMO).
Likewise, (setq ,b 42) (yes) and (setq ’b 42) (no).

If you want to argue for this syntax change, why
not address some of my arguments against it?  Where
will you draw the line, for instance?  There are
_lots_ of possible confusables.

I'd say start with only the few that have actually
been reported (is there only one reported?), trying
to come up with reasonable warnings in particular
contexts of use.  That would be a good start.

We might even have a user option that lists the
confusables to check/warn for, with whatever
default value people here think is best (it might
be only `’', to start with - or both left and
right curly quotes).

Are you thinking instead (since both you and Paul
mentioned the Unicode list of confusables) of
starting with _all_ characters in that list?

http://www.unicode.org/Public/security/8.0.0/confusables.txt

I won't argue about which chars should be warned
about, though I might be interested to see what
contexts we warn for and what the messages will be.

My objection is not about detecting this or that
use of this or that character and warning/reminding
users about it.

My objection is to making Lisp require escaping of
such characters.  That's all.  I think I've made
that as clear as I possibly can.

But you seem to want to paint my objection as
being against helping users know about accidental
use of confusables, e.g., `’' instead of `''.  Why?

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Noam Postavsky]

On Sat, Feb 3, 2018 at 8:55 PM, Drew Adams <drew.adams@oracle.com> wrote:

> My objection is to making Lisp require escaping of
> such characters.  That's all.  I think I've made
> that as clear as I possibly can.

I think your position is indeed quite clear by now. In fact, I think
the length and frequency of your posts are going to make it harder for
other people to participate, so could you dial it back it a bit.
Please?

Why "symbol's value" error about a list?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For example, suppose you have a Lisp program that produces the
  > following error message when compiled/executed:

  >   Symbol's value as variable is void: 'аbbrevs-changed

Does that error message really happen?  If so, how can I reproduce it?

I understand that the character 'а' is not ASCII a.  That explains why
'аbbrevs-changed' is not known as a variable.  But I'm talking about
a different issue, which has nothing to do with character coding.

Suppose it were 'foobaz', all ASCII, and we got an error such as

  >   Symbol's value as variable is void: 'foobaz

That still seems wrong.

If the error was that foobaz was void, the error message should not
include a quote.  It should say

  >   Symbol's value as variable is void: foobaz

Or if the error was that 'foobaz is used instead of a symbol, the error
message should say

  >   Wrong type argument: symbolp, (quote foobaz)


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Why "symbol's value" error about a list?

[Alan Mackenzie]

Hello, Richard.

On Sun, Feb 04, 2018 at 20:06:39 -0500, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > For example, suppose you have a Lisp program that produces the
>   > following error message when compiled/executed:

>   >   Symbol's value as variable is void: 'аbbrevs-changed

> Does that error message really happen?  If so, how can I reproduce it?

In Emacs-25.3 -Q, do

    M-: (message "(setq foo 'bar)") RET

, followed by getting the output from *Messages* into the kill ring with
M-w, followed by

    M-: C-y RET

.  You might think you are executing (setq foo 'bar).  You're not.
You're executing (setq foo ’bar), where the ’ is a Unicode curly quote.

The error message given out is:

    Symbol's value as variable is void: ’bar

.  If you're like me, you will read that as the symbol "bar" is void,
rather than the symbol "’bar" is void.

This is a result of the change in `message', silently to convert ' to a
curly quote, by default.  Some of us were unhappy at this change and
protested against it.

> I understand that the character 'а' is not ASCII a.  That explains why
> 'аbbrevs-changed' is not known as a variable.  But I'm talking about
> a different issue, which has nothing to do with character coding.

> Suppose it were 'foobaz', all ASCII, and we got an error such as

>   >   Symbol's value as variable is void: 'foobaz

> That still seems wrong.

Again "’foobaz", not "foobaz" is the symbol, here.

> If the error was that foobaz was void, the error message should not
> include a quote.  It should say

>   >   Symbol's value as variable is void: foobaz

Yes.

> Or if the error was that 'foobaz is used instead of a symbol, the error
> message should say

>   >   Wrong type argument: symbolp, (quote foobaz)

In the recent pretest, Emacs-26.0.91, when a curly quote appears at the
start of a symbol, the reader rejects it, giving the error message:

    read--expression: Invalid read syntax: "strange quote", "'"

.  This is somewhat controversional, and is what the recent discussion
has been about.

> -- 
> Dr Richard Stallman
> President, Free Software Foundation (https://gnu.org, https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
> Skype: No way! See https://stallman.org/skype.html.

-- 
Alan Mackenzie (Nuremberg, Germany).

RE: Why "symbol's value" error about a list?

[Drew Adams]

> >   > For example, suppose you have a Lisp program that produces the
> >   > following error message when compiled/executed:
> >   >   Symbol's value as variable is void: 'аbbrevs-changed
> 
> > Does that error message really happen?  If so, how can I reproduce it?
> 
> In Emacs-25.3 -Q, do M-: (message "(setq foo 'bar)") RET
> followed by getting the output from *Messages* into the kill ring with
> M-w, followed by M-: C-y RET.
> 
> You might think you are executing (setq foo 'bar).  You're not.
> You're executing (setq foo ’bar), where the ’ is a Unicode curly quote.
> 
> The error message given out is:
>     Symbol's value as variable is void: ’bar

That was the old, and legitimate, error message, yes.  It
accurately describes what is really going on (as you describe
well, below).

Now the message is instead (invalid-read-syntax "strange quote"
"’").  Is that better?  That's part of what this discussion is
about.

I suggested that the variable name be enclosed in `...'.  That
would make the original message clearer, I think:

  Symbol's value as variable is void: `’bar'

At least it could make it more likely that you would think about
looking at that quote mark.

> This is a result of the change in `message', silently to convert ' to a
> curly quote, by default.  Some of us were unhappy at this change and
> protested against it.

Count me as one of those "some of us".  Echoing Lisp code
should do just that - no fiddling to "prettify" apostrophe to
curly quote etc.

> > Suppose it were 'foobaz', all ASCII, and we got an error such as
> >   Symbol's value as variable is void: 'foobaz
> > That still seems wrong.

Here's the thing: There _is_ a Lisp error - no doubt.

But for Lisp the error is not that a curly quote was read
as part of a symbol name.  That's not a Lisp error (at
least it has not been, until now.)

The error is using a symbol as a variable, when it is not
defined as a variable.  Which is exactly what the original
error message said.

That's the LISP error.  Is there a _user error_ here?
Yes, it's the mistake of copying and pasting what was
printed in *Messages*.

That user mistake is excusable.  And we would want to
inform the user about it, if we can't prevent it.  But
changing Lisp read syntax to guess what might be the
most helpful thing to tell a user here is NOT the solution.

Should this Lisp syntax change be reverted?  That's the
question being discussed here.

Changing the read syntax is a general, Lisp-level change.
We should instead prevent this user mistake by removing
its cause.

The real error here is (IMO) a design error by Emacs: The
expression read and copied to *Messages* should not have
been "helpfully" translated to use a curly quote instead
of an apostrophe.

Emacs shot Lisp in the foot on this one.  It's not the
fault of Lisp and its reader (syntax).  It's the fault of
some misguided "modernization" of Emacs gone amuck.  Users
should not find the input (setq foo 'bar) transformed to
(setq foo ’bar), i.e., APOSTROPHE replaced by RIGHT SINGLE
QUOTATION MARK.

> Again "’foobaz", not "foobaz" is the symbol, here.

Yes, and that's a legitimate symbol name.  Nothing wrong
with Lisp telling us that that symbol is undefined as a
variable.  That's exactly what the _LISP_ problem is here.

That's just not the symbol that was passed to Lisp
originally.  That's a non-variable symbol name copied from
*Messages*.  The mistake was putting that in *Messages* in
the first place.

Where was the mistake?  Lisp claiming that you used a
symbol as an undefined variable?  The user copying that
symbol name from *Messages* and trying to evaluate its
symbol as a var?  Or Emacs inserting a different symbol
name in *Messages*, by substituting the text "’bar" for
the text "'bar"?

The original Lisp expression was a Lisp expression, not
just text.  A quote mark (apostrophe) in Lisp has special
meaning, special syntax.  That shouldn't be ignored by
some dumb (yes) substitution of curly quotes for straight
quotes.

> > If the error was that foobaz was void, the error message
> > should not include a quote.  It should say
> >   Symbol's value as variable is void: foobaz
> 
> Yes.

No.  Only if `foobaz' were indeed the symbol that was an
undefined variable.  But that's NOT the case here.  The
undefined variable here is the symbol `’foobaz' (from
*Messages*) - it really is.

The underlying mistake took place long before Lisp
evaluation of the pasted sexp.

Re: Why "symbol's value" error about a list?

[Eli Zaretskii]

> Date: Mon, 5 Feb 2018 13:46:38 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, npostavs@users.sourceforge.net,
>         emacs-devel@gnu.org
> 
> > The error message given out is:
> >     Symbol's value as variable is void: ’bar
> 
> That was the old, and legitimate, error message, yes.  It
> accurately describes what is really going on (as you describe
> well, below).
> 
> Now the message is instead (invalid-read-syntax "strange quote"
> "’").  Is that better?

I think it's somewhat better, because it talks about "strange quote",
which is a hint for the user about the actual problem.

> I suggested that the variable name be enclosed in `...'.  That
> would make the original message clearer, I think:
> 
>   Symbol's value as variable is void: `’bar'

That might make things even more confusing, because the text actually
displayed will be this:

    Symbol’s value as variable is void: ‘’bar’

which loses all hints of what is being quoted here.

> > This is a result of the change in `message', silently to convert ' to a
> > curly quote, by default.  Some of us were unhappy at this change and
> > protested against it.
> 
> Count me as one of those "some of us".  Echoing Lisp code
> should do just that - no fiddling to "prettify" apostrophe to
> curly quote etc.

That ship has sailed two Emacs releases ago.  We are trying to fix the
fallout.

And strange quotes is only one situation where confusingly similar
characters can be presented in error messages, making it hard for
users to spot the real problem.  We are trying to find ways of making
such "typos" more evident in error messages.

> The error is using a symbol as a variable, when it is not
> defined as a variable.  Which is exactly what the original
> error message said.
> 
> That's the LISP error.  Is there a _user error_ here?
> Yes, it's the mistake of copying and pasting what was
> printed in *Messages*.
> 
> That user mistake is excusable.  And we would want to
> inform the user about it, if we can't prevent it.  But
> changing Lisp read syntax to guess what might be the
> most helpful thing to tell a user here is NOT the solution.

The issue is what _would_ be a helpful message in these cases.  You
are just saying what should _not_ be done (repeatedly), but that
doesn't advance us towards the solution.

> Should this Lisp syntax change be reverted?  That's the
> question being discussed here.

No, that's only part of the question.  The other, no less important
part is if we revert that change, how to make the confusing error
message less so and more helpful in understanding the user error.

Re: Why "symbol's value" error about a list?

[Tim Cross]

[-- Attachment #1: Type: text/plain, Size: 3814 bytes --]

It seems there are two issues here - they are not completely separate, but
do seem to be distinct and probably need to be addressed in two steps.

If the statement

> Count me as one of those "some of us".  Echoing Lisp code
> should do just that - no fiddling to "prettify" apostrophe to
> curly quote etc.

is correct, then I would agree it was a bad design decision. The *Messages*
buffer should display lisp code exactly as it is read and not try to
'prettify' it.

The second issue seems to be more about how to make the error message more
informative. I suspect this is a much harder problem to resolve. I don't
know what the right solution is for that, but I do know that I would have
more chance of recognising my error if the message displayed in the buffer
displays the lisp code exactly as it was read by the reader.

Tim


On 6 February 2018 at 15:13, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Mon, 5 Feb 2018 13:46:38 -0800 (PST)
> > From: Drew Adams <drew.adams@oracle.com>
> > Cc: Eli Zaretskii <eliz@gnu.org>, npostavs@users.sourceforge.net,
> >         emacs-devel@gnu.org
> >
> > > The error message given out is:
> > >     Symbol's value as variable is void: ’bar
> >
> > That was the old, and legitimate, error message, yes.  It
> > accurately describes what is really going on (as you describe
> > well, below).
> >
> > Now the message is instead (invalid-read-syntax "strange quote"
> > "’").  Is that better?
>
> I think it's somewhat better, because it talks about "strange quote",
> which is a hint for the user about the actual problem.
>
> > I suggested that the variable name be enclosed in `...'.  That
> > would make the original message clearer, I think:
> >
> >   Symbol's value as variable is void: `’bar'
>
> That might make things even more confusing, because the text actually
> displayed will be this:
>
>     Symbol’s value as variable is void: ‘’bar’
>
> which loses all hints of what is being quoted here.
>
> > > This is a result of the change in `message', silently to convert ' to a
> > > curly quote, by default.  Some of us were unhappy at this change and
> > > protested against it.
> >
> > Count me as one of those "some of us".  Echoing Lisp code
> > should do just that - no fiddling to "prettify" apostrophe to
> > curly quote etc.
>
> That ship has sailed two Emacs releases ago.  We are trying to fix the
> fallout.
>
> And strange quotes is only one situation where confusingly similar
> characters can be presented in error messages, making it hard for
> users to spot the real problem.  We are trying to find ways of making
> such "typos" more evident in error messages.
>
> > The error is using a symbol as a variable, when it is not
> > defined as a variable.  Which is exactly what the original
> > error message said.
> >
> > That's the LISP error.  Is there a _user error_ here?
> > Yes, it's the mistake of copying and pasting what was
> > printed in *Messages*.
> >
> > That user mistake is excusable.  And we would want to
> > inform the user about it, if we can't prevent it.  But
> > changing Lisp read syntax to guess what might be the
> > most helpful thing to tell a user here is NOT the solution.
>
> The issue is what _would_ be a helpful message in these cases.  You
> are just saying what should _not_ be done (repeatedly), but that
> doesn't advance us towards the solution.
>
> > Should this Lisp syntax change be reverted?  That's the
> > question being discussed here.
>
> No, that's only part of the question.  The other, no less important
> part is if we revert that change, how to make the confusing error
> message less so and more helpful in understanding the user error.
>
>


-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 5199 bytes --]

Re: Why "symbol's value" error about a list?

[Eli Zaretskii]

On February 6, 2018 9:32:21 AM GMT+02:00, Tim Cross <theophilusx@gmail.com> wrote:
> It seems there are two issues here - they are not completely separate,
> but
> do seem to be distinct and probably need to be addressed in two steps.
> 
> If the statement
> 
> > Count me as one of those "some of us".  Echoing Lisp code
> > should do just that - no fiddling to "prettify" apostrophe to
> > curly quote etc.
> 
> is correct, then I would agree it was a bad design decision. The
> *Messages*
> buffer should display lisp code exactly as it is read and not try to
> 'prettify' it.
> 
> The second issue seems to be more about how to make the error message
> more
> informative. I suspect this is a much harder problem to resolve. I
> don't
> know what the right solution is for that, but I do know that I would
> have
> more chance of recognising my error if the message displayed in the
> buffer
> displays the lisp code exactly as it was read by the reader.
> 
> Tim
> 
> 
> On 6 February 2018 at 15:13, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > Date: Mon, 5 Feb 2018 13:46:38 -0800 (PST)
> > > From: Drew Adams <drew.adams@oracle.com>
> > > Cc: Eli Zaretskii <eliz@gnu.org>, npostavs@users.sourceforge.net,
> > >         emacs-devel@gnu.org
> > >
> > > > The error message given out is:
> > > >     Symbol's value as variable is void: ’bar
> > >
> > > That was the old, and legitimate, error message, yes.  It
> > > accurately describes what is really going on (as you describe
> > > well, below).
> > >
> > > Now the message is instead (invalid-read-syntax "strange quote"
> > > "’").  Is that better?
> >
> > I think it's somewhat better, because it talks about "strange
> quote",
> > which is a hint for the user about the actual problem.
> >
> > > I suggested that the variable name be enclosed in `...'.  That
> > > would make the original message clearer, I think:
> > >
> > >   Symbol's value as variable is void: `’bar'
> >
> > That might make things even more confusing, because the text
> actually
> > displayed will be this:
> >
> >     Symbol’s value as variable is void: ‘’bar’
> >
> > which loses all hints of what is being quoted here.
> >
> > > > This is a result of the change in `message', silently to convert
> ' to a
> > > > curly quote, by default.  Some of us were unhappy at this change
> and
> > > > protested against it.
> > >
> > > Count me as one of those "some of us".  Echoing Lisp code
> > > should do just that - no fiddling to "prettify" apostrophe to
> > > curly quote etc.
> >
> > That ship has sailed two Emacs releases ago.  We are trying to fix
> the
> > fallout.
> >
> > And strange quotes is only one situation where confusingly similar
> > characters can be presented in error messages, making it hard for
> > users to spot the real problem.  We are trying to find ways of
> making
> > such "typos" more evident in error messages.
> >
> > > The error is using a symbol as a variable, when it is not
> > > defined as a variable.  Which is exactly what the original
> > > error message said.
> > >
> > > That's the LISP error.  Is there a _user error_ here?
> > > Yes, it's the mistake of copying and pasting what was
> > > printed in *Messages*.
> > >
> > > That user mistake is excusable.  And we would want to
> > > inform the user about it, if we can't prevent it.  But
> > > changing Lisp read syntax to guess what might be the
> > > most helpful thing to tell a user here is NOT the solution.
> >
> > The issue is what _would_ be a helpful message in these cases.  You
> > are just saying what should _not_ be done (repeatedly), but that
> > doesn't advance us towards the solution.
> >
> > > Should this Lisp syntax change be reverted?  That's the
> > > question being discussed here.
> >
> > No, that's only part of the question.  The other, no less important
> > part is if we revert that change, how to make the confusing error
> > message less so and more helpful in understanding the user error.
> >
> >

Lisp code is not changed in messages; only quoted plain text is.

RE: Why "symbol's value" error about a list?

[Drew Adams]

> It seems there are two issues here - they are not completely
> separate, but do seem to be distinct and probably need to be
> addressed in two steps. 
>
> If the statement 
>
> > Count me as one of those "some of us".  Echoing Lisp code
> > should do just that - no fiddling to "prettify" apostrophe to
> > curly quote etc.
>
> is correct, then I would agree it was a bad design decision.
> The *Messages* buffer should display lisp code exactly as it
> is read and not try to 'prettify' it.

Yes.

> The second issue seems to be more about how to make the error
> message more informative.

Yes, but it's not just about that error message.  That Lisp
error is about an undefined variable.  But there are plenty
of other contexts where users can be confused by such a
gotcha.

If code or a user did in fact define a variable named ’bar
then there would be no Lisp error (prior to the recent change).

Such a symbol name could nevertheless be confusing in some
contexts.  But it's not about how best to present that
undefined-variable error message.

That message was telling the truth, even if in the particular
context presented it might not be immediately clear to a user
what the undefined symbol name is (i.e., that the name contains
a curly quote).

> I suspect this is a much harder problem to resolve. I don't
> know what the right solution is for that, but I do know that
> I would have more chance of recognising my error if the
> message displayed in the buffer displays the lisp code
> exactly as it was read by the reader.

Precisely.

That Lisp error really is about using symbol `’bar' as
a variable.  Such code can exist in different contexts,
only some of which have anything to do with a user
mistaking a curly quote for an apostrophe.

Just as some Lisp code can mistakenly use symbol `abc'
as a variable apart from any binding of it as a variable,
and so provoking the undefined-variable error, so can
code mistakenly use symbol `’bar' as an undefined variable.

Such a context would have nothing to do with the newly
fabricated error (invalid-read-syntax "strange quote" "’").
That's just the wrong error for Lisp to raise here.

RE: Why "symbol's value" error about a list?

[Drew Adams]

> > > The error message given out is:
> > >     Symbol's value as variable is void: ’bar
> >
> > That was the old, and legitimate, error message, yes.  It
> > accurately describes what is really going on (as you describe
> > well, below).
> >
> > Now the message is instead (invalid-read-syntax "strange quote"
> > "’").  Is that better?
> 
> I think it's somewhat better, because it talks about "strange quote",
> which is a hint for the user about the actual problem.

The actual problem is the use of a non-variable symbol
as a variable.  At least that has been the problem in
this example, until the recent change in Lisp syntax.

There's no problem using a symbol as a variable if its
name is ’bar.  You've just made it necessary now to
escape that curly quote when defining and using the
symbol:

 (defvar \’bar 42 "...")

And if a variable in fact has that name you still raise
a Lisp read-syntax error if the quote is not escaped.

> > I suggested that the variable name be enclosed in `...'.  That
> > would make the original message clearer, I think:
> >   Symbol's value as variable is void: `’bar'
> 
> That might make things even more confusing, because the text actually
> displayed will be this:
>     Symbol’s value as variable is void: ‘’bar’
> which loses all hints of what is being quoted here.

I wrote `’bar'.

> > > This is a result of the change in `message', silently to
> > > convert ' to a curly quote, by default.  Some of us were
> > > unhappy at this change and protested against it.
> >
> > Count me as one of those "some of us".  Echoing Lisp code
> > should do just that - no fiddling to "prettify" apostrophe to
> > curly quote etc.
> 
> That ship has sailed two Emacs releases ago.  We are trying to
> fix the fallout.

Two releases ago and still reaping the fallout rewards...
Time to call back that ship or try to redirect it?

> And strange quotes is only one situation where confusingly similar
> characters can be presented in error messages, making it hard for
> users to spot the real problem.  We are trying to find ways of making
> such "typos" more evident in error messages.

Where's the error in  (defvar ’bar 42 "...")?
You've introduced Lisp read errors where there were
none.

The error here is the automatic translation of a Lisp
sexp that uses an ordinary quote mark (apostrophe) to
a curly quote by `message' (?), so that the wrong sexp
gets logged to *Messages*.

The second error is trying to fix that error by changing
Lisp syntax so that an error is raised, instead of just
(optionally) displaying a warning message.  There's no
reason to stop Lisp evaluation just because we want to
inform a user about a possible misunderstanding (gotcha).

> > The error is using a symbol as a variable, when it is not
> > defined as a variable.  Which is exactly what the original
> > error message said.
> >
> > That's the LISP error.  Is there a _user error_ here?
> > Yes, it's the mistake of copying and pasting what was
> > printed in *Messages*.
> >
> > That user mistake is excusable.  And we would want to
> > inform the user about it, if we can't prevent it.  But
> > changing Lisp read syntax to guess what might be the
> > most helpful thing to tell a user here is NOT the solution.
> 
> The issue is what _would_ be a helpful message in these cases.  You
> are just saying what should _not_ be done (repeatedly), but that
> doesn't advance us towards the solution.

I've said (repeatedly, as you like to repeat) that we can
display all the warnings you like.  What we should not do
is change Lisp syntax to raise an artificial error.

There is no Lisp error in evaluating (setq ’bar 42),
regardless of how or why someone might do that.  It's
fine to let someone know that s?he did it, pointing
to the curly quote.  It's wrong to raise a Lisp error.

> > Should this Lisp syntax change be reverted?  That's the
> > question being discussed here.
> 
> No, that's only part of the question.  The other, no less important
> part is if we revert that change, how to make the confusing error
> message less so and more helpful in understanding the user error.

Agreed.  The first step is to revert the change in Lisp
syntax.  The second step is to design aids for users to
recognize such gotchas.

The zeroth step is to realize that the Lisp change should
be reverted.

Re: Why "symbol's value" error about a list?

[Eli Zaretskii]

> Date: Tue, 6 Feb 2018 07:45:55 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: acm@muc.de, rms@gnu.org, npostavs@users.sourceforge.net,
>         emacs-devel@gnu.org
> 
> > > > The error message given out is:
> > > >     Symbol's value as variable is void: ’bar
> > >
> > > That was the old, and legitimate, error message, yes.  It
> > > accurately describes what is really going on (as you describe
> > > well, below).
> > >
> > > Now the message is instead (invalid-read-syntax "strange quote"
> > > "’").  Is that better?
> > 
> > I think it's somewhat better, because it talks about "strange quote",
> > which is a hint for the user about the actual problem.
> 
> The actual problem is the use of a non-variable symbol
> as a variable.

That's one possibility, yes.  But a much more probable possibility is
that the user mistyped the quote, either because she copy/pasted it
from some text, or because she turned on the Electric Quote mode, or
for some other reason.

A useful error message should consider this latter probable cause and
help the user correct it, if indeed that was the reason.  Many tools
do similar second-guessing for frequent mistakes.  For example. GNU
Make detects when a line in a Makefile starts with 8 SPC characters
instead of a mandatory TAB, and says:

 *** missing separator (did you mean TAB instead of 8 spaces?).

The first part is the "dumb" error message, based on the syntax error,
the part in parentheses is a helpful hint for the user, based on many
such user errors seen in the past.

Latest versions of GCC also provide similar hints.

> You've just made it necessary now to escape that curly quote when
> defining and using the symbol:

You are changing the subject.  I just wrote that an error message
which mentions "strange quotes" is somewhat better than one which just
states the syntax error.  I said nothing about anything else.

> > >   Symbol's value as variable is void: `’bar'
> > 
> > That might make things even more confusing, because the text actually
> > displayed will be this:
> >     Symbol’s value as variable is void: ‘’bar’
> > which loses all hints of what is being quoted here.
> 
> I wrote `’bar'.

Yes, but the Lisp function 'message' has its own ideas regarding
quoting text, as you well know.  Try evaluating this:

  (message "Symbol's value as variable is void: `%s'" "’bar")

> > That ship has sailed two Emacs releases ago.  We are trying to
> > fix the fallout.
> 
> Two releases ago and still reaping the fallout rewards...
> Time to call back that ship or try to redirect it?

You can try fighting this Quixotic battle on and on, but I don't
recommend that.

> You've introduced Lisp read errors where there were
> none.

No, I didn't do anything of the kind.
> > > Should this Lisp syntax change be reverted?  That's the
> > > question being discussed here.
> > 
> > No, that's only part of the question.  The other, no less important
> > part is if we revert that change, how to make the confusing error
> > message less so and more helpful in understanding the user error.
> 
> Agreed.  The first step is to revert the change in Lisp
> syntax.  The second step is to design aids for users to
> recognize such gotchas.

I think both steps should be made together, otherwise we would be
making a change for the worse.

Re: Why "symbol's value" error about a list?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   >   Symbol's value as variable is void: 'foobaz

  > > That still seems wrong.

  > Again "’foobaz", not "foobaz" is the symbol, here.

This makes sense, as an issue about quotes.

But why, then, did the other message talk about using a confusable
letter and say that was the ONLY problem.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Why "symbol's value" error about a list?

[Noam Postavsky]

On Sun, Feb 4, 2018 at 8:06 PM, Richard Stallman <rms@gnu.org> wrote:

>   > For example, suppose you have a Lisp program that produces the
>   > following error message when compiled/executed:
>
>   >   Symbol's value as variable is void: 'аbbrevs-changed
>
> Does that error message really happen?  If so, how can I reproduce it?

I don't think there is a way to get this particular message (with an
ascii apostrophe and cyrillic a).

Re: Why "symbol's value" error about a list?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > For example, suppose you have a Lisp program that produces the
  > >   > following error message when compiled/executed:
  > >
  > >   >   Symbol's value as variable is void: 'аbbrevs-changed
  > >
  > > Does that error message really happen?  If so, how can I reproduce it?

  > I don't think there is a way to get this particular message (with an
  > ascii apostrophe and cyrillic a).

So it was purely hypothetical?

In that case, I wish the person who wrote that had made it clear
it was not a real, existing problem.  The failure to do this
made us waste our time.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Why "symbol's value" error about a list?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, drew.adams@oracle.com, emacs-devel@gnu.org
> Date: Tue, 06 Feb 2018 09:53:16 -0500
> 
>   > I don't think there is a way to get this particular message (with an
>   > ascii apostrophe and cyrillic a).
> 
> So it was purely hypothetical?
> 
> In that case, I wish the person who wrote that had made it clear
> it was not a real, existing problem.

That person did:

  For example, suppose you have a Lisp program that produces the
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  following error message when compiled/executed:

    Symbol's value as variable is void: 'аbbrevs-changed

Re: Why "symbol's value" error about a list?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > In that case, I wish the person who wrote that had made it clear
  > > it was not a real, existing problem.

  > That person did:

  >   For example, suppose you have a Lisp program that produces the
  >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  >   following error message when compiled/executed:

Those words do NOT say that this is an unreal example.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Why "symbol's value" error about a list?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: npostavs@users.sourceforge.net, drew.adams@oracle.com,
> 	emacs-devel@gnu.org
> Date: Tue, 06 Feb 2018 21:40:50 -0500
> 
>   > > In that case, I wish the person who wrote that had made it clear
>   > > it was not a real, existing problem.
> 
>   > That person did:
> 
>   >   For example, suppose you have a Lisp program that produces the
>   >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   >   following error message when compiled/executed:
> 
> Those words do NOT say that this is an unreal example.

They do for me.  Of course, my command of English is not perfect, so
maybe I'm missing something.

Re: Why "symbol's value" error about a list?

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Tue, 6 Feb 2018 06:27:33 -0500
> Cc: Eli Zaretskii <eliz@gnu.org>, Drew Adams <drew.adams@oracle.com>, 
> 	Emacs developers <emacs-devel@gnu.org>
> 
> On Sun, Feb 4, 2018 at 8:06 PM, Richard Stallman <rms@gnu.org> wrote:
> 
> >   > For example, suppose you have a Lisp program that produces the
> >   > following error message when compiled/executed:
> >
> >   >   Symbol's value as variable is void: 'аbbrevs-changed
> >
> > Does that error message really happen?  If so, how can I reproduce it?
> 
> I don't think there is a way to get this particular message (with an
> ascii apostrophe and cyrillic a).

The point I wanted to make stands if you remove the apostrophe.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > We want to find a way of identifying such situation and telling the
  > Lisp programmer about that in clear and easily understandable ways.
  > One way, perhaps too radical one, is to reject such "confusable"
  > characters outright.

I think that makes sense for Lisp symbols.  Lisp has had strings for
40 years now, so it isn't customary to use symbols to represent
arbitrary text.

We could have a mode where intern converts all these character codes
to a single canonical set, but the default could be to give an error for all
but the preferred one.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Aaron Ecay]

Hi Noam,

2018ko otsailak 2an, Noam Postavsky-ek idatzi zuen:
> 
> In Emacs 26 and earlier the following is valid lisp code:
> 
> (setq ’bar 42)
> (setq foo ’bar)

I was surprised to learn that this is the case, in light of what is
said in the Elisp reference about symbol names: “A symbol name can
contain any characters whatever. Most symbol names are written with
letters, digits, and the punctuation characters ‘-+=*/’. Such names
require no special punctuation; the characters of the name suffice as
long as the name does not look like a number. (If it does, write a ‘\’
at the beginning of the name to force interpretation as a symbol.) The
characters ‘_~!@$%^&:<>{}?’  are less often used but also require no
special punctuation. Any other characters may be included in a symbol's
name by escaping them with a backslash.”  (info "(elisp) Symbol Type")

Would it be worth considering making the reader enforce this fully
specification, as an alternative to your patch?  That would solve
this problem with curly quotes in symbol names (which also bit me at
one point), as well as the potential problems with other confusable
characters raised by Paul.

(It might still be desirable to add a special user-friendly error message
when the illegal characters are confusable with an ASCII single quote, as
an additional user-friendliness measure.)

Aaron

PS if this approach is not taken, the manual should at least be changed
to match the actual behavior of the reader.

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

> I was surprised to learn that this is the case, in light of what is
> said in the Elisp reference about symbol names:
>
> “A symbol name can
> contain any characters whatever. Most symbol names are written with
> letters, digits, and the punctuation characters ‘-+=*/’. Such names
> require no special punctuation; the characters of the name suffice as
> long as the name does not look like a number. (If it does, write a ‘\’
> at the beginning of the name to force interpretation as a symbol.) The
> characters ‘_~!@$%^&:<>{}?’  are less often used but also require no
> special punctuation. Any other characters may be included in a symbol's
> name by escaping them with a backslash.”  (info "(elisp) Symbol Type")

Thank you very much for that.  I guess I wasn't aware of
that text.  I thought that there were only a very few
chars that needed to be escaped in symbol names - `,',
`(', etc.: only chars that have special syntactic
meaning in Lisp.

I suppose that invalidates my objection, though I wonder
_why_ we would require escaping so many ordinary chars.

And like you I wonder whether that text is accurate.
I wonder whether that is the intended design (why?) or it
is just an inaccurate description of the real behavior.

Trying various chars from confusables.txt, it does not
seem like they require escaping (at least not yet).
That text appears to be wrong.

I'd prefer it if escaping was _not_ required for chars
other than those mentioned in that text, including
chars in confusables.txt.  I think it makes more sense
to require escaping only for characters that have
special Lisp significance, syntactically.

IOW, I prefer the actual behavior to the behavior
described in that text.  I don't think someone using
Hebrew or Arabic or Chinese or Korean letters in a
symbol name should need to escape each one (or any
of them).

But if the design described there has already been
decided on then as best for Emacs then I guess my
argument is moot.  In that case, the implementation
is currently waaaaaay out of whack wrt the design.

And if that's the design to be implemented then I
agree with you that implementing it as described
in that text would at least have an advantage of
consistency.

> Would it be worth considering making the reader enforce this fully
> specification, as an alternative to your patch?  That would solve
> this problem with curly quotes in symbol names (which also bit me at
> one point), as well as the potential problems with other confusable
> characters raised by Paul.
> 
> (It might still be desirable to add a special user-friendly error
> message when the illegal characters are confusable with an ASCII
> single quote, as an additional user-friendliness measure.)
> 
> if this approach is not taken, the manual should at least
> be changed to match the actual behavior of the reader.

That's the approach I'd prefer.  Let chars be used in
symbol names without escaping, except for those with
special Lisp syntax.

But add warnings in contexts where we think someone
might have inadvertently used a confusable in place
of a common character.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1230 bytes --]

Aaron Ecay wrote:
> I was surprised to learn that this is the case, in light of what is
> said in the Elisp reference about symbol names

Good point; thanks. In the spirit of "be strict about what you generate", the 
Emacs printer should escape any character that is not in the list of characters 
documented in the Elisp manual as being safe (i.e., as not requiring escaping). 
This is elementary future-proofing, and is independent of whether we want Emacs 
to warn about or disallow confusable chars in symbols.

Proposed patches against 'master' attached. The first merely simplifes the code 
without changing its effect. The second fixes a bug in the manual, which 
incorrectly states that '?' never needs escaping in symbol names. These two 
patches are routine. (I assume the second one should be applied to emacs26 
instead of to master.)

The third patch changes the Lisp printer to escape characters as suggested above.

The fourth patch changes the Lisp printer to escape '?' only at the start of a 
symbol. This is nicer for programs using Scheme-style naming conventions in 
Emacs Lisp, e.g., 'fooish?' rather than 'fooishp'. I discovered the need for 
this patch when I wrote the second patch.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Simplify-print_object-a-bit.patch --]
[-- Type: text/x-patch; name="0001-Simplify-print_object-a-bit.patch", Size: 2780 bytes --]

From c03b816016f8cc2f15d275e7ad23448366489277 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 3 Feb 2018 20:29:00 -0800
Subject: [PATCH 1/4] Simplify print_object a bit

* src/print.c (print_object): Simplify by using C99
constructs, and by taking advantage of the fact that Lisp
strings are are followed by null bytes.
---
 src/print.c | 40 ++++++++++++++++------------------------
 1 file changed, 16 insertions(+), 24 deletions(-)

diff --git a/src/print.c b/src/print.c
index b3c0f6f..d3eb49d 100644
--- a/src/print.c
+++ b/src/print.c
@@ -1916,38 +1916,29 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
     case Lisp_Symbol:
       {
 	bool confusing;
-	unsigned char *p = SDATA (SYMBOL_NAME (obj));
-	unsigned char *end = p + SBYTES (SYMBOL_NAME (obj));
-	int c;
-	ptrdiff_t i, i_byte;
-	ptrdiff_t size_byte;
-	Lisp_Object name;
-
-	name = SYMBOL_NAME (obj);
-
-	if (p != end && (*p == '-' || *p == '+')) p++;
-	if (p == end)
-	  confusing = 0;
+	Lisp_Object name = SYMBOL_NAME (obj);
+	ptrdiff_t size_byte = SBYTES (name);
+	unsigned char *p = SDATA (name);
+	unsigned char *end = p + size_byte;
+
 	/* If symbol name begins with a digit, and ends with a digit,
 	   and contains nothing but digits and `e', it could be treated
 	   as a number.  So set CONFUSING.
 
-	   Symbols that contain periods could also be taken as numbers,
-	   but periods are always escaped, so we don't have to worry
-	   about them here.  */
-	else if (*p >= '0' && *p <= '9'
-		 && end[-1] >= '0' && end[-1] <= '9')
+	   Symbols that contain '.' or '#' could also be taken as
+	   numbers, but these are always escaped so don't worry about
+	   them here.  */
+	if (c_isdigit (p[*p == '-' || *p == '+']) && c_isdigit (end[-1]))
 	  {
-	    while (p != end && ((*p >= '0' && *p <= '9')
-				/* Needed for \2e10.  */
-				|| *p == 'e' || *p == 'E'))
+	    /* Check for 'e' too; needed for \2e10.  */
+	    do
 	      p++;
+	    while (c_isdigit (*p) || *p == 'e' || *p == 'E');
+
 	    confusing = (end == p);
 	  }
 	else
-	  confusing = 0;
-
-	size_byte = SBYTES (name);
+	  confusing = false;
 
 	if (! NILP (Vprint_gensym)
             && !SYMBOL_INTERNED_IN_INITIAL_OBARRAY_P (obj))
@@ -1958,10 +1949,11 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
 	    break;
 	  }
 
-	for (i = 0, i_byte = 0; i_byte < size_byte;)
+	for (ptrdiff_t i = 0, i_byte = 0; i_byte < size_byte;)
 	  {
 	    /* Here, we must convert each multi-byte form to the
 	       corresponding character code before handing it to PRINTCHAR.  */
+	    int c;
 	    FETCH_STRING_CHAR_ADVANCE (c, name, i, i_byte);
 	    maybe_quit ();
 
-- 
2.7.4


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Say-needs-escaping-at-start-of-symbol.patch --]
[-- Type: text/x-patch; name="0002-Say-needs-escaping-at-start-of-symbol.patch", Size: 1210 bytes --]

From 4b945a3fcbf6ff2bde4595dd8b8f472d1b3d17af Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 3 Feb 2018 20:30:21 -0800
Subject: [PATCH 2/4] Say ? needs escaping at start of symbol.

* doc/lispref/objects.texi: ? is also special.
---
 doc/lispref/objects.texi | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/doc/lispref/objects.texi b/doc/lispref/objects.texi
index af74062..f0420e6 100644
--- a/doc/lispref/objects.texi
+++ b/doc/lispref/objects.texi
@@ -557,7 +557,8 @@ Symbol Type
 of the name suffice as long as the name does not look like a number.
 (If it does, write a @samp{\} at the beginning of the name to force
 interpretation as a symbol.)  The characters @samp{_~!@@$%^&:<>@{@}?} are
-less often used but also require no special punctuation.  Any other
+less often used but also require no special punctuation, except that
+@samp{\} must precede @samp{?} at the start of a symbol.  Any other
 characters may be included in a symbol's name by escaping them with a
 backslash.  In contrast to its use in strings, however, a backslash in
 the name of a symbol simply quotes the single character that follows the
-- 
2.7.4


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-prin1-etc.-now-escape-more-chars-in-symbols.patch --]
[-- Type: text/x-patch; name="0003-prin1-etc.-now-escape-more-chars-in-symbols.patch", Size: 3147 bytes --]

From 2add3a1595f709bb071e2b775970038470b2fab2 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 3 Feb 2018 20:30:48 -0800
Subject: [PATCH 3/4] prin1 etc. now escape more chars in symbols

Inspired by email from Aaron Ecay in:
https://lists.gnu.org/r/emacs-devel/2018-02/msg00125.html
* etc/NEWS: Mention this.
* src/print.c (print_object): Escape any character that is not
documented to not require escaping.
---
 etc/NEWS    |  7 +++++++
 src/print.c | 37 +++++++++++++++++++++++++++++++------
 2 files changed, 38 insertions(+), 6 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index afd0fba..2a46002 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -87,6 +87,13 @@ regular expression was previously invalid, but is now accepted:
 
    x\{32768\}
 
+** 'print' and related functions now escape more chars in symbols.
+They now escape any symbol character that is outside the documented
+set of characters that do not need escaping.  For example, (print
+(intern "n\u0456l")) now outputs "n\іl" instead of "nіl", as a hint to
+the reader that the "і" is not the usual U+0069 LATIN SMALL LETTER I,
+but is instead U+0456 CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I.
+
 \f
 * Editing Changes in Emacs 27.1
 
diff --git a/src/print.c b/src/print.c
index d3eb49d..7eca36a 100644
--- a/src/print.c
+++ b/src/print.c
@@ -1959,12 +1959,37 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
 
 	    if (escapeflag)
 	      {
-		if (c == '\"' || c == '\\' || c == '\''
-		    || c == ';' || c == '#' || c == '(' || c == ')'
-		    || c == ',' || c == '.' || c == '`'
-		    || c == '[' || c == ']' || c == '?' || c <= 040
-                    || confusing
-		    || (i == 1 && confusable_symbol_character_p (c)))
+		switch (c)
+		  {
+		    /* The Emacs Lisp manual lists these characters as
+		       not requiring escaping in symbols.  Although some
+		       other characters might also work, play it safe
+		       and escape all but these characters.  */
+		  case '!': case '$': case '%': case '&':
+		  case '*': case '-': case '+': case '/':
+		  case '0': case '1': case '2': case '3': case '4':
+		  case '5': case '6': case '7': case '8': case '9':
+		  case ':': case '<': case '=': case '>': case '@':
+		  case 'A': case 'B': case 'C': case 'D': case 'E': case 'F':
+		  case 'G': case 'H': case 'I': case 'J': case 'K': case 'L':
+		  case 'M': case 'N': case 'O': case 'P': case 'Q': case 'R':
+		  case 'S': case 'T': case 'U': case 'V': case 'W': case 'X':
+		  case 'Y': case 'Z':
+		  case '^': case '_':
+		  case 'a': case 'b': case 'c': case 'd': case 'e': case 'f':
+		  case 'g': case 'h': case 'i': case 'j': case 'k': case 'l':
+		  case 'm': case 'n': case 'o': case 'p': case 'q': case 'r':
+		  case 's': case 't': case 'u': case 'v': case 'w': case 'x':
+		  case 'y': case 'z':
+		  case '{': case '}': case '~':
+		    break;
+
+		  default:
+		    confusing = true;
+		    break;
+		  }
+
+		if (confusing)
 		  {
 		    printchar ('\\', printcharfun);
 		    confusing = false;
-- 
2.7.4


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-Escape-only-at-start-of-symbol.patch --]
[-- Type: text/x-patch; name="0004-Escape-only-at-start-of-symbol.patch", Size: 1913 bytes --]

From 4289ea136de4876b5dfc20d83b5a2556d1b5d8e6 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 3 Feb 2018 20:39:48 -0800
Subject: [PATCH 4/4] Escape ? only at start of symbol

* src/print.c (print_object): Do it.
---
 etc/NEWS    | 4 ++++
 src/print.c | 4 ++--
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 2a46002..c435136 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -94,6 +94,10 @@ set of characters that do not need escaping.  For example, (print
 the reader that the "і" is not the usual U+0069 LATIN SMALL LETTER I,
 but is instead U+0456 CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I.
 
+** 'print' etc. no longer escape non-initial '?' in symbols.
+For example, the symbol 'list?' is now printed as-is.  Initial '?'
+is still escaped, e.g., (print (intern "?x")) still outputs "\?x".
+
 \f
 * Editing Changes in Emacs 27.1
 
diff --git a/src/print.c b/src/print.c
index 7eca36a..dfd6c50 100644
--- a/src/print.c
+++ b/src/print.c
@@ -1938,7 +1938,7 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
 	    confusing = (end == p);
 	  }
 	else
-	  confusing = false;
+	  confusing = *p == '?';
 
 	if (! NILP (Vprint_gensym)
             && !SYMBOL_INTERNED_IN_INITIAL_OBARRAY_P (obj))
@@ -1969,7 +1969,7 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
 		  case '*': case '-': case '+': case '/':
 		  case '0': case '1': case '2': case '3': case '4':
 		  case '5': case '6': case '7': case '8': case '9':
-		  case ':': case '<': case '=': case '>': case '@':
+		  case ':': case '<': case '=': case '>': case '?': case '@':
 		  case 'A': case 'B': case 'C': case 'D': case 'E': case 'F':
 		  case 'G': case 'H': case 'I': case 'J': case 'K': case 'L':
 		  case 'M': case 'N': case 'O': case 'P': case 'Q': case 'R':
-- 
2.7.4

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Andreas Schwab]

On Feb 03 2018, Paul Eggert <eggert@cs.ucla.edu> wrote:

> Good point; thanks. In the spirit of "be strict about what you generate",
> the Emacs printer should escape any character that is not in the list of
> characters documented in the Elisp manual as being safe (i.e., as not
> requiring escaping).

No!

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Noam Postavsky]

On Sat, Feb 3, 2018 at 11:51 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> Aaron Ecay wrote:
>>
>> I was surprised to learn that this is the case, in light of what is
>> said in the Elisp reference about symbol names

      Most symbol names are written with letters, digits, and the
    punctuation characters `-+=*/'.  Such names require no special
    punctuation...

> Good point; thanks. In the spirit of "be strict about what you generate",
> the Emacs printer should escape any character that is not in the list of
> characters documented in the Elisp manual as being safe (i.e., as not
> requiring escaping). This is elementary future-proofing, and is independent
> of whether we want Emacs to warn about or disallow confusable chars in
> symbols.

My impression is that manual passage was written with only ASCII
characters in mind.  But since Emacs has allowed Unicode characters in
symbol names for a long time now, I don't think we should all of a
sudden declare "letters" to mean just [a-zA-Z].

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Sun, 4 Feb 2018 10:04:26 -0500
> Cc: Aaron Ecay <aaronecay@gmail.com>, Drew Adams <drew.adams@oracle.com>,
> 	Emacs developers <emacs-devel@gnu.org>
> 
> I don't think we should all of a sudden declare "letters" to mean
> just [a-zA-Z].

We don't: [:alpha:] nowadays matches much more than just [a-zA-Z].

So indeed, restricting Lisp symbols that way sounds too harsh, almost
arbitrary.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Eli Zaretskii wrote:
> restricting Lisp symbols that way sounds too harsh

OK, I'll omit that patch.

We still have a problem with Emacs Lisp code containing confusable characters 
that can make the code exceedingly hard to review. These characters are 
currently not caught or checked by anything. We really should do better.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am So., 4. Feb. 2018 um 20:36 Uhr:

> Eli Zaretskii wrote:
> > restricting Lisp symbols that way sounds too harsh
>
> OK, I'll omit that patch.
>
> We still have a problem with Emacs Lisp code containing confusable
> characters
> that can make the code exceedingly hard to review. These characters are
> currently not caught or checked by anything. We really should do better.
>
>
The following should be unintrusive and not too hard: Let the reader push
all confusable symbols (i.e. symbols that contain at least one unescaped
character from the Unicode confusables list that maps to a sequence of
ASCII characters) onto an internal dynamic variable. The byte compiler can
then emit warnings if that variable becomes non-nil.

[-- Attachment #2: Type: text/html, Size: 1071 bytes --]

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> Cc: aaronecay@gmail.com, drew.adams@oracle.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 4 Feb 2018 11:36:05 -0800
> 
> We still have a problem with Emacs Lisp code containing confusable characters 
> that can make the code exceedingly hard to review. These characters are 
> currently not caught or checked by anything. We really should do better.

I agree that we need a solid solution to that.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> Date: Sun, 04 Feb 2018 22:10:25 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: aaronecay@gmail.com, emacs-devel@gnu.org, drew.adams@oracle.com,
> 	npostavs@users.sourceforge.net
> 
> > Cc: aaronecay@gmail.com, drew.adams@oracle.com, emacs-devel@gnu.org
> > From: Paul Eggert <eggert@cs.ucla.edu>
> > Date: Sun, 4 Feb 2018 11:36:05 -0800
> > 
> > We still have a problem with Emacs Lisp code containing confusable characters 
> > that can make the code exceedingly hard to review. These characters are 
> > currently not caught or checked by anything. We really should do better.
> 
> I agree that we need a solid solution to that.

How about if we start by warning about any Lisp symbol whose name uses
characters from more than one script for non-punctuation characters?
puny.el has a function that solves a similar problem, which could be
used as a starting point.

We could make this an opt-in feature if the warnings are deemed to be
a potential annoyance.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Eli Zaretskii wrote:
> How about if we start by warning about any Lisp symbol whose name uses
> characters from more than one script for non-punctuation characters?

This problem can occur even in one-character symbols. It might be better to 
establish a default script for the file, and warn about any characters from a 
different script.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Clément Pit-Claudel]

On 2018-02-04 15:48, Paul Eggert wrote:
> Eli Zaretskii wrote:
>> How about if we start by warning about any Lisp symbol whose name uses
>> characters from more than one script for non-punctuation characters?
> 
> This problem can occur even in one-character symbols. It might be better to establish a default script for the file, and warn about any characters from a different script.

We could also default to warning about any characters in the confusables list and not in ascii.  And we'd make it easy to turn the check off using a file-local variable.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Noam Postavsky]

On Fri, 2 Feb 2018 at 17:24, Noam Postavsky
<npostavs@users.sourceforge.net> wrote:
>
> In Emacs 26 and earlier the following is valid lisp code:
>
> (setq ’bar 42)
> (setq foo ’bar)
>
> In the current master branch, this will signal (invalid-read-syntax
> "strange quote" "’").

I've posted a patch which removes the error in this case, and instead
just adds to the error message if evaluating an expression with a
fancy quote leads to an error, see Bug#32939
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32939>.

Archive link to previous discussion:
https://lists.gnu.org/archive/html/emacs-devel/2018-02/msg00093.html

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

I'm afraid this patch is heading in the wrong direction, as we should be 
more vigilant about confusables, not less.

Consider this example, abstracted from the auth-source-secrets-create 
source code:

     (if (eq r 'secret)
         (let ((data data))
           (lambda () data))
       data)

The intent of the (let ((data data)) ...) code is to create a thunk 
which, when evaluated, yields the current value of 'data' (not the value 
of 'data' when the thunk is called), and that is what any human reading 
the code will see. However, that is not what the code actually does. In 
the (let ((data data)) ...), the space between the two instances of 
'data' is really an EN SPACE (U+2002) so the 'let' is declaring an 
identifier 'data data' whose name contains an EN SPACE and whose value 
is nil, an identifier that is never used; so the thunk yields the later 
value of 'data', not the earlier one.

Because humans cannot reliably review source code containing characters 
that are easily confusable with the ASCII symbols that are a basic part 
of Elisp syntax, we should not be relaxing the reader to encourage 
developers to use these characters in their identifiers. On the 
contrary, we should be discouraging their use even more than we do now.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 4 Oct 2018 18:01:26 -0700
> Cc: Drew Adams <drew.adams@oracle.com>
> 
> I'm afraid this patch is heading in the wrong direction, as we should be 
> more vigilant about confusables, not less.
> 
> Consider this example, abstracted from the auth-source-secrets-create 
> source code:
> 
>      (if (eq r 'secret)
>          (let ((data data))
>            (lambda () data))
>        data)

Is this example relevant to the proposed changes?  The latter only
change what we do for quote-like symbols that are not interpreted as
quotes by the Lisp reader.  You, OTOH, are raising a different
problem, one for which AFAIK we currently have no solution.

The general issue of "confusable" characters, both in Lisp code and in
user interaction, is an issue that still awaits a proper solution in
Emacs.  (Many moons ago, I was seduced to write a couple of primitives
to allow detection of confusable text that played tricks with bidi
reordering, but AFAICT those primitives are still not used, which is a
pity, IMO.)  I'd encourage people to work on this.

However, the much more narrow issue brought up by this bug report is
specifically about quote characters.  It is related to changes in our
messages, which now by default produce non-ASCII quotes, something
that made this particular problem more probable than it was before.  I
think as long as we don't disallow such characters in Lisp symbols,
the proposed treatment, via evaluation-time warning, is a reasonable
solution, slightly better than the somewhat confusing error message we
present now.

We could also augment that by displaying the confusable characters in
a distinct face, something we already do for some of them.

IOW, I disagree with "discourage" part of your opinion: there's
nothing wrong with using such characters as long as we don't formally
cease to support them.  And the commonly accepted mechanism of
pointing out potentially wrong constructs is by visual cues and
warning messages, not by erroring out.  Compare how we treat something
like this in C programs:

  if (a = b) { do something; }

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

On 10/5/18 1:43 AM, Eli Zaretskii wrote:
> the commonly accepted mechanism of
> pointing out potentially wrong constructs is by visual cues and
> warning messages

If we decide that Elisp source code must be able to abuse confusable 
characters, then of course we should allow such abuse and support it as 
best we can, including selective highlighting and whatnot to try to warn 
readers of the abuse. Such support won't work outside Emacs, but people 
using non-Emacs programs to look at Elisp code will simply be out of luck.

However, that would be heading in the wrong direction, because we 
shouldn't assume that Elisp code is reviewed only via Emacs. I regularly 
use Savannah's web interface to look at Elisp source code diffs, for 
example, and there's lots of other ways I and other developers use 
non-Emacs programs to look at Elisp source. Because reading source code 
is an essential property of free software, and because it would set a 
bad precedent if we said or implied that one really should use only 
Emacs to read Elisp code, we can't sufficiently address the problem 
merely by highlighting characters when Emacs is viewing them in a 
certain way and saying or implying that people should use only Emacs to 
review Elisp code.

I'm not arguing that Elisp should prohibit symbols from containing 
confusing characters, only that these characters should be easily 
recognizable in plain-text source code, without requiring Emacs itself 
(configured a certain way) to view the source. For example, if we 
required a backslash before every confusable character in a symbol, that 
would go a long way toward addressing the problem.

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

> I'm not arguing that Elisp should prohibit symbols from containing
> confusing characters, only that these characters should be easily
> recognizable in plain-text source code, without requiring Emacs itself
> (configured a certain way) to view the source. For example, if we
> required a backslash before every confusable character in a symbol, that
> would go a long way toward addressing the problem.

The right approach is to let Lisp tell you about its syntax.
Lisp should raise an error only for, well, an actual Lisp error.

If Emacs wants to highlight something that it can guess
(accurately) might be a typo (e.g. a copy+paste gotcha)
then fine. And even that highlighting should be optional
(which it is, if from font-lock).

There are more and more such copy+paste gotchas, for
at least a couple reasons: (1) Emacs has now moved to
using curly quotes more gratuitously, and (2) users copy
code from both Emacs and other sources, and some such
code uses curly quotes (even sometimes mistakenly in
place of apostrophes in Lisp), no-break space chars, and
other confusables.

And maybe also (3) users can sometimes type a curly
quote, no-break space, etc. more easily now, in some
editors, maybe even sometimes by just hitting an
ordinary keyboard key, such as ' or space bar.

We have to live with this now, like it or not. That's not a
reason to tell Lisp to treat a curly quote as an apostrophe,
and it's not a reason to tell it to raise an error when a
curly quote is used in a place where an apostrophe could
be used.

Similarly, it's not a reason for Lisp to guess that you really
meant SPC instead of no-break space. And so on.

This is a judgment call, but we should _let Lisp judge_
about syntax errors, based on, well, its own syntax. If you
use (let (foo  foo)...), where there is a no-break space
between foo and foo, so be it. That's a single symbol,
`foo foo'.

(My mail client doesn't even let me paste a no-break
space char there, it seems, so you'll have to pretend.
That's the kind of second-guessing behavior we should
be avoiding, FWIW.)

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Alan Mackenzie]

Hello, Drew.

Just a quick point.

On Sat, Oct 06, 2018 at 00:20:27 +0000, Drew Adams wrote:

[ .... ]

> This is a judgment call, but we should _let Lisp judge_
> about syntax errors, based on, well, its own syntax. If you
> use (let (foo  foo)...), where there is a no-break space
> between foo and foo, so be it. That's a single symbol,
> `foo foo'.

Do we even allow the syntax (let ((foo))...)?  If we do, then why?
There's (let (foo)...) and (let ((foo nil))...) for binding a symbol to
nil.

We made (setq foo) invalid some while ago.  Why not similarly make (let
((foo))...) invalid?  That would solve at least part of this problem,
is easy to do. and is almost certainly harmless.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Stefan Monnier]

> We made (setq foo) invalid some while ago.  Why not similarly make (let
> ((foo))...) invalid?

I assume you mean that the byte-compiler should signal a warning?
If so, I'm fully in favor.

Signaling an actual error would be problematic at this stage, because
it'd break too much code.


        Stefan

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

> > This is a judgment call, but we should _let Lisp judge_
> > about syntax errors, based on, well, its own syntax. If you
> > use (let (foo  foo)...), where there is a no-break space
> > between foo and foo, so be it. That's a single symbol,
> > `foo foo'.
> 
> Do we even allow the syntax (let ((foo))...)?  If we do, then why?
> There's (let (foo)...) and (let ((foo nil))...) for binding a symbol to
> nil.

Yes, sorry. I wasn't paying attention to the parens in that
example.

My point was only that use of `foo foo' (with a no-break
space between the two foo's) as a mistake/typo for an
intended `foo foo' (with a normal space) should not be
signaled by Lisp as an error. But the no-break space could
be highlighted as sometimes helpful info. `foo foo' (with
no-break space) is just a symbol, for Lisp - not a syntax
error.

E.g. (changing the example):

(let (foo foo)...) binds symbol `foo foo' (with a no-break
space) to nil. It doesn't bind symbol `foo' to the current
value of symbol `foo'.

So, e.g., if symbol `foo' happens to be unbound then
even evaluation of that binding won't raise an error
(e.g. unbound variable `foo').

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Garreau, Alexandre]

On 2018-10-06 at 14:57, Drew Adams wrote:
> My point was only that use of `foo foo' (with a no-break
> space between the two foo's) as a mistake/typo for an
> intended `foo foo' (with a normal space) should not be
> signaled by Lisp as an error. But the no-break space could
> be highlighted as sometimes helpful info. `foo foo' (with
> no-break space) is just a symbol, for Lisp - not a syntax
> error.

Unbreakable space is already colored as some sort of colored underscore.
The problem is there are a bunch of other kind of spaces, though I
personally only use unbreakable, non-justifying and/or “fine” (dunno if
it match an en space) space in French, though I do it commonly in
strings with emacs (which only correctly highlight the normal
unbreakable space, but not the others).  I’d like to see these
highlighted (perhaps, or even preferably, a different way) as well, or
should it be something customizable according user preferences (or
language?).

> E.g. (changing the example):
>
> (let (foo foo)...) binds symbol `foo foo' (with a no-break
> space) to nil. It doesn't bind symbol `foo' to the current
> value of symbol `foo'.

I would have expected it to bind twice foo to nil (or to signal an error
or a warning), yet it seems you used a normal space, unbreakable-space
already highlights in emacs, so I’d noticed it.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Alan Mackenzie wrote:
> Do we even allow the syntax (let ((foo))...)?

Although that's a reasonable question it doesn't address the main problem, as 
there are lots of other opportunities for confusion like this one. For example, 
if I see '(foo ․ bar) in Elisp source code, I'll naturally think it yields a 
cons of two symbols. It doesn't: it yields a list of three symbols, because that 
dot is not a FULL STOP (U+002E); it's a ONE DOT LEADER (U+2024).

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Drew Adams wrote:
> The right approach is to let Lisp tell you about its syntax... we should _let Lisp judge_

This seems to be assuming that there is something out there called "Lisp" that 
tells us what to do. That's entirely backwards. Lisp is our servant, not our 
master. Lisp syntax should be whatever the best syntax we can come up with, to 
help us and our users do our work, and this should be the way we think and write 
about it.

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

> > The right approach is to let Lisp tell you about its syntax... we should _let
> Lisp judge_
> 
> This seems to be assuming that there is something out there called "Lisp" that
> tells us what to do. That's entirely backwards. Lisp is our servant, not our
> master. Lisp syntax should be whatever the best syntax we can come up with,
> to help us and our users do our work, and this should be the way we think and
> write about it.

Such characters have symbol syntax in Lisp (Elisp, for instance).
That's the Lisp syntax in question. Lisp doesn't require you to
escape them in symbols. Hasn't done so before and shouldn't
do so now (IMHO).

Yes, we have liberty to change Lisp, including Lisp syntax, in
any number of ways. That doesn't mean that we should.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We can make any change in Emacs Lisp syntax that we decide to make,
but incoherence with the spirit of Lisp will lead to trouble.

The spirit of Lisp is not something that can be precisely defined,
but experienced Lispers will mostly agree about what it says.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> Cc: npostavs@users.sourceforge.net, emacs-devel@gnu.org, drew.adams@oracle.com
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 5 Oct 2018 16:02:09 -0700
> 
> However, that would be heading in the wrong direction, because we 
> shouldn't assume that Elisp code is reviewed only via Emacs. I regularly 
> use Savannah's web interface to look at Elisp source code diffs, for 
> example, and there's lots of other ways I and other developers use 
> non-Emacs programs to look at Elisp source. Because reading source code 
> is an essential property of free software, and because it would set a 
> bad precedent if we said or implied that one really should use only 
> Emacs to read Elisp code, we can't sufficiently address the problem 
> merely by highlighting characters when Emacs is viewing them in a 
> certain way and saying or implying that people should use only Emacs to 
> review Elisp code.
> 
> I'm not arguing that Elisp should prohibit symbols from containing 
> confusing characters, only that these characters should be easily 
> recognizable in plain-text source code, without requiring Emacs itself 
> (configured a certain way) to view the source. For example, if we 
> required a backslash before every confusable character in a symbol, that 
> would go a long way toward addressing the problem.

I agree that viewing ELisp code outside of Emacs is a valid use case.
But I don't think a backslash before these non-ASCII quotes will
significantly lower the confusion potential when those characters are
used in the source.

Basically, there's a contradiction here between our desire not to
confuse relatively inexperienced users of ELisp and help them avoid
problems which might be hard to figure out, and our desire not to
annoy experienced users.  Personally, I think that using faces strikes
a good balance between these contradictory motives.  I don't see how
we can be harsh to uses of these characters without actually
prohibiting their use in symbols.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Eli Zaretskii wrote:
> I agree that viewing ELisp code outside of Emacs is a valid use case.
> But I don't think a backslash before these non-ASCII quotes will
> significantly lower the confusion potential when those characters are
> used in the source.

I don't follow. If someone writes '(let ((foo\ bar)) baz)' then a human reader 
is put immediately and obviously on notice that there's something odd about that 
code. We already require a backslash for that ordinary space (U+0020); why not 
also require it for EN SPACE (U+2002)? That will significantly lower confusion here.

The point is not to distinguish 'foo\ bar' (with ordinary space) from 'foo\ bar' 
(with en space); the point is to distinguish both from the 'foo bar' (two 
identifiers) that a reader would ordinarily expect here, because that's the main 
way a malicious hacker could confuse even experienced reviewers.
> Basically, there's a contradiction here between our desire not to
> confuse relatively inexperienced users of ELisp and help them avoid
> problems which might be hard to figure out, and our desire not to
> annoy experienced users.

That's not the point I was making. I'm an experienced Elisp user, and I am 
*extremely annoyed* (to put it mildly) that malicious users can put one over on 
us by using characters that look like spaces, or parentheses, or whatever, 
characters that are not what they look like. This has nothing to do with 
confusing inexperienced users. I *really want* Elisp to be relatively immune to 
this problem, at least for programs that I help maintain. And I don't want the 
immunity to work only when I'm using Emacs on a nice display: I often read code 
with Emacs highlighting unavailable or turned off, or without using Emacs at all.

At the very least there should be an option whereby the Emacs source code itself 
is routinely verified to be free of confusable characters in identifiers, to 
help prevent malicious code from sneaking into Emacs itself. Even if we give 
users the ability to let others shoot them, we should at least improve our own 
defenses.

> I don't see how
> we can be harsh to uses of these characters without actually
> prohibiting their use in symbols.

I already gave one proposal for doing just that: require that characters 
confusable with ASCII be escaped. Initially we can merely warn about any 
unescaped confusables; as long as the warning is prominent enough that should be 
OK for starters. This proposal does not prohibit their use in symbols, as one 
can simply escape the characters.

There are other ways to skin this cat as well. We should be heading in this 
direction, not removing the (admittedly inadequate) protection we already have.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: emacs-devel@gnu.org, Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Sat, 6 Oct 2018 08:51:18 -0700
> 
> Eli Zaretskii wrote:
> > I agree that viewing ELisp code outside of Emacs is a valid use case.
> > But I don't think a backslash before these non-ASCII quotes will
> > significantly lower the confusion potential when those characters are
> > used in the source.
> 
> I don't follow. If someone writes '(let ((foo\ bar)) baz)' then a human reader 
> is put immediately and obviously on notice that there's something odd about that 
> code. We already require a backslash for that ordinary space (U+0020); why not 
> also require it for EN SPACE (U+2002)? That will significantly lower confusion here.

How will it lower the confusion, when the same is required for a
space?

And once again, these examples are not relevant to the issue at hand,
which is only about quotes.

> > I don't see how
> > we can be harsh to uses of these characters without actually
> > prohibiting their use in symbols.
> 
> I already gave one proposal for doing just that: require that characters 
> confusable with ASCII be escaped.

That's an annoyance, IMO.  This is why this bug report exists, right?

And again, please don't bring up the more general issue with any other
confusable character, as those require a more general solution about
which we don't yet have a clear idea.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Eli Zaretskii wrote:

> these examples are not relevant to the issue at hand,
> which is only about quotes

Quotes are part of the same problem. For example, here's some code in Gnus:

    (ignore-errors (gnus-get-function method 'open-server))

Change that APOSTROPHE (U+0027) to RIGHT SINGLE QUOTATION MARK (U+2019) and the 
code will look the same but do something quite different, with no diagnostic. 
This sort of code is reasonably common and can easily be security-relevant. If 
Emacs stops diagnosing this abuse of confusable characters, we're opening 
ourselves up more to malicious code.

> How will it lower the confusion, when the same is required for a space?

Let me rephrase my point, with apostrophe rather than space.

The point is not to distinguish ´open-server (with U+00B4 ACUTE ACCENT) from 
՚open-server (with U+055A ARMENIAN APOSTROPHE); the point is to distinguish both 
of these from the 'open-server (apostrophe followed by symbol) that a reader 
would ordinarily expect here. We need to give an obvious way for human readers 
to see that something odd is going on. Readers can then use C-x = (or whatever) 
to find out exactly what the oddness is.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, npostavs@users.sourceforge.net
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 6 Oct 2018 11:03:25 -0700
> 
> Eli Zaretskii wrote:
> 
> > these examples are not relevant to the issue at hand,
> > which is only about quotes
> 
> Quotes are part of the same problem.

Yes, but a much smaller part.  And solving the problem with quotes
doesn't require to solve the more general one.

>     (ignore-errors (gnus-get-function method 'open-server))
> 
> Change that APOSTROPHE (U+0027) to RIGHT SINGLE QUOTATION MARK (U+2019) and the 
> code will look the same but do something quite different, with no diagnostic. 

I understand.  I'm just saying that adding a backslash between the
U+2019 quote will not significantly improve the situation, because
Emacs Lisp uses backslashes in many other situation, like ?\", and
therefore the mere fact that there is a backslash doesn't necessarily
alert the human reader to the existence of an unusual character.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Eli Zaretskii wrote:
> Emacs Lisp uses backslashes in many other situation, like ?\", and
> therefore the mere fact that there is a backslash doesn't necessarily
> alert the human reader to the existence of an unusual character.

Yes, the solution I proposed addresses only symbols, not the more-general 
problem of confusable characters in strings and characters. Although symbols are 
a more significant issue since they occur more often, it would be nice to 
address strings and character constants too. We can do that by adding support 
for a new string escape \cX for the confusable character X (e.g., ?\c՚ would 
mean U+055A ARMENIAN APOSTROPHE), and by diagnosing the use of unescaped 
confusable characters in strings and character constants.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Eli Zaretskii wrote:
>>      (ignore-errors (gnus-get-function method 'open-server))
>>
>> Change that APOSTROPHE (U+0027) to RIGHT SINGLE QUOTATION MARK (U+2019) and the
>> code will look the same but do something quite different, with no diagnostic.

> ... adding a backslash between the
> U+2019 quote will not significantly improve the situation, because
> Emacs Lisp uses backslashes in many other situation, like ?\", and
> therefore the mere fact that there is a backslash doesn't necessarily
> alert the human reader to the existence of an unusual character.

True, a backslash within a string or character is not a sufficient alert. 
However, a backslash within a symbol is. For example, although it's relatively 
common to see strings like this in Elisp source code:

    "color=\"blue\""

it's extremely uncommon to see symbols like this:

    color=\"blue\"

and so if one sees such a symbol (possibly with some other character in place of 
the " marks, possibly not) one will easily know that something odd is going on.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Garreau, Alexandre]

On 2018-10-06 at 21:29, Eli Zaretskii wrote:
> I understand.  I'm just saying that adding a backslash between the
> U+2019 quote will not significantly improve the situation, because
> Emacs Lisp uses backslashes in many other situation, like ?\", and
> therefore the mere fact that there is a backslash doesn't necessarily
> alert the human reader to the existence of an unusual character.

I think what is wanted here is not to alert of unusual character but
alert of a non-syntaxically-relevant (for the reader) character, such as
?\', which quotes (or be it ?\", or ?\(, or ?\), etc.), which looks like
a character that will have syntactic consequences (like quoting), while
being a “normal” (part of the symbol) character.  Like not alerting two
symbols may look alike (the ՚open-server/´open-server case), but one
thing which is a symbol, unquoted (like \'open-server, which is evident
it’s a symbol because of the backslash, since afaik no character
preceded by a backslash can do something with syntax, apart being part
of a symbol (thus the behavior is clear)), and the other which *not* be
a unquoted symbol at all (like 'open-server, which just returns the
symbol).

I guess the same issue indeed arise with ?¸, or anything alike, that
won’t unquote the next symbol.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Garreau, Alexandre]

On 2018-10-05 at 16:02, Paul Eggert wrote:
> However, that would be heading in the wrong direction, because we
> shouldn't assume that Elisp code is reviewed only via Emacs. I
> regularly use Savannah's web interface

In a world where unicode is increasingly present and confusion about its
characters increasingly problematic (typosquatting, etc.) wouldn’t it be
reasonable to expect unicode-related semantic functions to be provided
in most frameworks, systems and languages to allow better handling of
such problems, thus making that problem the interface’s one?

Maybe if ever this problem occurs more and more in domain names,
internet addresses, and such, interfaces such as web ones, or other
editors, will inevitably need to support features to avoid confusion the
same way emacs currently could?

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,  drew.adams@oracle.com,  npostavs@users.sourceforge.net
> Date: Sat, 06 Oct 2018 13:22:14 +0200
> 
> In a world where unicode is increasingly present and confusion about its
> characters increasingly problematic (typosquatting, etc.) wouldn’t it be
> reasonable to expect unicode-related semantic functions to be provided
> in most frameworks, systems and languages to allow better handling of
> such problems, thus making that problem the interface’s one?

I don't think I understand what this means in practice; please
elaborate.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Garreau, Alexandre]

Le 06/10/2018 à 14h50, Eli Zaretskii a écrit :
>> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>> drew.adams@oracle.com, npostavs@users.sourceforge.net
>> Date: Sat, 06 Oct 2018 13:22:14 +0200
>> 
>> In a world where unicode is increasingly present and confusion about its
>> characters increasingly problematic (typosquatting, etc.) wouldn’t it be
>> reasonable to expect unicode-related semantic functions to be provided
>> in most frameworks, systems and languages to allow better handling of
>> such problems, thus making that problem the interface’s one?
>
> I don't think I understand what this means in practice; please
> elaborate.

afaik there are also problems in other contents than source code about
undistinguishable unicode character, such as the latin ?o and the
cyrillic ?о (the first example of unicode-powered typosquatting I ever
heard), the different spaces (sometimes not distinguishable in monospace
font), or, to stay on monospacing problems: I have great pain in writing
correct french text as I must always check in something not-emacs about
which one between ?– and ?— is the medium and the long dash (I normally
recall through their position on my keyboard but as they’re aside I
often forget), not to recall the different hacks about bidirectionality
you highlighted earlier.  I also heard about emails confusing
semantic-based bayesian anti-spam by putting not-spammy words in mails
that, because of some unicode tricks, wouldn’t be displayed to user.

This problems aren’t local to source code, nor to emacs (as many people
use something else than emacs to read mails, websites, news, and reading
domain names), and afaik there are canonicalizations and semantic
unicode categories functions to help knowing what is punctuation, what
is combining, what is displayed and takes how much space, and maybe, but
I’m unsure, which characters are to be difficult or even impossible to
distinguish (or some canonicalizations function to get two differently
encoded (related to combining characters (such as the difference between
"é" and "é" (made of ?e then ?́ (it’s fun to see how this last one is
strangely displayed and finely evaluated by emacs)))) strings comparable
the same, or two characters-different but looking-alike strings
comparable the same too).

I guess this issue is even going to be less a problem in free softwares
where theorically the writers should be well-intentioned and shouldn’t
try to trick the readers on what the software do (and/or it should at
least be reviewed with capable tools and/or knowledge), compared to
cases where this is going to be abusable and profitable, such as
typosquating ("google.com" and "gооgle.com" are not the same (it’s
interesting to notice too how emacs forward/backward-word detects and
use the language-switching to stop at the "оо", I’m astounished by these
capabilities I have to thank you guy for a such great piece of
software!) but google could aford (and took care) to buy both while not
everyone could do as well (and nobody yet reserved "amazоn.com"), and
people might crack, steal or blackmail using something like that).

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Cc: eggert@cs.ucla.edu,  emacs-devel@gnu.org,  drew.adams@oracle.com,  npostavs@users.sourceforge.net
> Date: Sat, 06 Oct 2018 14:10:17 +0200
> 
> afaik there are also problems in other contents than source code about
> undistinguishable unicode character, such as the latin ?o and the
> cyrillic ?о (the first example of unicode-powered typosquatting I ever
> heard), the different spaces (sometimes not distinguishable in monospace
> font), or, to stay on monospacing problems: I have great pain in writing
> correct french text as I must always check in something not-emacs about
> which one between ?– and ?— is the medium and the long dash (I normally
> recall through their position on my keyboard but as they’re aside I
> often forget), not to recall the different hacks about bidirectionality
> you highlighted earlier.  I also heard about emails confusing
> semantic-based bayesian anti-spam by putting not-spammy words in mails
> that, because of some unicode tricks, wouldn’t be displayed to user.

This is the more general problem I mentioned up-thread:

  http://lists.gnu.org/archive/html/emacs-devel/2018-10/msg00052.html

I agree that we should improve Emacs in that area, but I think it
would be wrong to hold off fixing the issue with quotes because the
more general problem is solved.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Noam Postavsky]

On Sat, 6 Oct 2018 at 10:00, Eli Zaretskii <eliz@gnu.org> wrote:
>
> > afaik there are also problems in other contents than source code about
> > undistinguishable unicode character, such as the latin ?o and the
> > cyrillic ?о (the first example of unicode-powered typosquatting I ever
> > heard), the different spaces (sometimes not distinguishable in monospace
[...]
>
> This is the more general problem I mentioned up-thread:
>
>   http://lists.gnu.org/archive/html/emacs-devel/2018-10/msg00052.html
>
> I agree that we should improve Emacs in that area, but I think it
> would be wrong to hold off fixing the issue with quotes because the
> more general problem is solved.

Right, I do see the quotes thing as an obvious place to start, and due
to how difficult the more general issue is, starting with enhancement
of existing warnings and existing errors make sense. It addresses the
immediate problem of confusiong from curly quotes, and we can add more
confusable characters later without fear of breaking anything.

Unicode security-issues workarounds elsewhere [Was: Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?]

[Garreau, Alexandre]

Le 06/10/2018 à 14h50, Eli Zaretskii a écrit :
>> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>> drew.adams@oracle.com, npostavs@users.sourceforge.net
>> Date: Sat, 06 Oct 2018 13:22:14 +0200
>> 
>> In a world where unicode is increasingly present and confusion about its
>> characters increasingly problematic (typosquatting, etc.) wouldn’t it be
>> reasonable to expect unicode-related semantic functions to be provided
>> in most frameworks, systems and languages to allow better handling of
>> such problems, thus making that problem the interface’s one?
>
> I don't think I understand what this means in practice; please
> elaborate.

The point I wanted to make is since as I highlighted this problem is of
greater importance in other interfaces than source codes, especially
browsers and web sites, typically, as these gets to be the most used
interfaces for everything nowadays.  So I guess these unicode
anti-confusion functions and more high-level functions based on these
already are or will become present in browsers and in languages such as
perl and php to end up in high-level functions in frameworks made in
perl or php, for instance, so that at the end “other interfaces than
emacs” such as web-browsers or websites may end supporting features such
as coloring differently mixed-script or unusual spaces, etc.

The other option being “ban unicode as much as possible” or “disallow
mixed-script”, and “ban all unicode punctuation characters (or all
non-letters (or non-alphanumeric?) characters, or something like that)
unless they’re inside ascii”.

I believe with increased support of unicode most languages, frameworks
and software should end with features allowing to allow these without
creating too much problems (at least not that much a lot more than in
emacs).

Re: Unicode security-issues workarounds elsewhere [Was: Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?]

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Date: Sat, 06 Oct 2018 15:15:34 +0200
> Cc: npostavs@users.sourceforge.net, eggert@cs.ucla.edu, drew.adams@oracle.com,
> 	emacs-devel@gnu.org
> 
> The other option being “ban unicode as much as possible” or “disallow
> mixed-script”, and “ban all unicode punctuation characters (or all
> non-letters (or non-alphanumeric?) characters, or something like that)
> unless they’re inside ascii”.

I don't think this is feasible in Emacs, we cannot limit non-ASCII
punctuation to ASCII text.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Garreau, Alexandre wrote:
> ouldn’t it be
> reasonable to expect unicode-related semantic functions to be provided
> in most frameworks, systems and languages to allow better handling of
> such problems

What a wonderful world that would be! But it's not the world we live in. Even 
Emacs misdisplays many of these characters now. And other tools that I routinely 
use (Firefox, Thunderbird, Gnome terminal) are even worse. We can't reasonably 
assume that confusable characters will be displayed nicely everywhere, not even 
five or ten years from now.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Stefan Monnier]

> in. Even Emacs misdisplays many of these characters now. And other tools

FWIW, we have the `uni-confusables` in GNU ELPA for that purpose.
Maybe we should integrate more tightly.


        Stefan

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Noam Postavsky]

On Fri, 5 Oct 2018 at 19:02, Paul Eggert <eggert@cs.ucla.edu> wrote:
>
> On 10/5/18 1:43 AM, Eli Zaretskii wrote:
> > the commonly accepted mechanism of
> > pointing out potentially wrong constructs is by visual cues and
> > warning messages
>
> If we decide that Elisp source code must be able to abuse confusable
> characters, then of course we should allow such abuse and support it as
> best we can, including selective highlighting and whatnot to try to warn
> readers of the abuse. Such support won't work outside Emacs, but people
> using non-Emacs programs to look at Elisp code will simply be out of luck.

The problem is that deciding which characters are confusable and hence
require backslash escaping is based on a shifting mess of heuristics.
So I don't think it's workable to signal a hard error for this. Both
in terms of false positives which could mean possibly breaking code,
and false negatives which means we would be giving a false sense of
security. That's why I proposed adding highlighting and enhancing
existing error messages instead. Of course adding warnings would also
make sense.

By the way, your EN SPACE example already gives a compile warning:

Warning: Unused lexical variable ‘data data’

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Noam Postavsky wrote:
> deciding which characters are confusable and hence
> require backslash escaping is based on a shifting mess of heuristics.

No more than the "shifting mess of heuristics" inevitable in any choice of 
syntax. Quite possibly the confusables list from the Unicode consortium will do. 
The list won't shift much once it's established.

We can start merely by warning about confusable characters and seeing how often 
those warnings are triggered in real (as opposed to malicious or 
purposely-tricky) code. If the warnings are quite rare, in a later Emacs version 
we can change the manual from "confusable characters should be escaped" to 
"confusable characters must be escaped".

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs developers <emacs-devel@gnu.org>,
>  Drew Adams <drew.adams@oracle.com>
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 9 Oct 2018 08:30:22 -0700
> 
> Noam Postavsky wrote:
> > deciding which characters are confusable and hence
> > require backslash escaping is based on a shifting mess of heuristics.
> 
> No more than the "shifting mess of heuristics" inevitable in any choice of 
> syntax. Quite possibly the confusables list from the Unicode consortium will do. 
> The list won't shift much once it's established.
> 
> We can start merely by warning about confusable characters and seeing how often 
> those warnings are triggered in real (as opposed to malicious or 
> purposely-tricky) code. If the warnings are quite rare, in a later Emacs version 
> we can change the manual from "confusable characters should be escaped" to 
> "confusable characters must be escaped".

Confusable characters are confusable only when they are surrounded by
ASCII characters or by characters that look like ASCII.  By
themselves, at least many of them, are entirely legitimate.  For
example, I see no reason to warn about a symbol named "сталин", even
though the characters с and а will be considered confusables if the
symbol would be named something like "саn".

So I think we cannot go by characters here, we need to examine the
context.

That is why I think we shouldn't link this particular issue, of quote
characters, with the more general problem: the latter is much more
complicated to solve correctly.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert]

Eli Zaretskii wrote:
> I see no reason to warn about a symbol named "сталин", even
> though the characters с and а will be considered confusables if the
> symbol would be named something like "саn".

Sure, that's fine. We can limit symbol warnings to the symbols containing 
non-ASCII chars all of which are confusable with ASCII. This will warn about 
"саn" (with Cyrillic "с" and "а") but not about "сталин" (with Cyrillic "a").

The point of the guideline is not to warn about every possible confusable 
character; it's to defend against malicious code.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Andreas Schwab]

On Okt 09 2018, Paul Eggert <eggert@cs.ucla.edu> wrote:

> Sure, that's fine. We can limit symbol warnings to the symbols containing
> non-ASCII chars all of which are confusable with ASCII. This will warn
> about "саn" (with Cyrillic "с" and "а") but not about "сталин" (with
> Cyrillic "a").

I'm pretty sure you can find many Russian words that are written with
only Latin-alike letters.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Aaron Ecay]

2018ko urriak 9an, Andreas Schwab-ek idatzi zuen:
> 
> On Okt 09 2018, Paul Eggert <eggert@cs.ucla.edu> wrote:
> 
>> Sure, that's fine. We can limit symbol warnings to the symbols containing
>> non-ASCII chars all of which are confusable with ASCII. This will warn
>> about "саn" (with Cyrillic "с" and "а") but not about "сталин" (with
>> Cyrillic "a").
> 
> I'm pretty sure you can find many Russian words that are written with
> only Latin-alike letters.

Should this be a warning?

(let ((с cyrillic-ess)) ...)

What about this?

(let ((c latin-c)
      (с cyrillic-ess))
  ...)

IMO the answer to both questions is yes (because Latin letters are
used for elisp special forms like “let,” so they should be inherently
privileged) – but I only use Latin letters in programs I write, so I
probably donʼt have the perspective to know how annoying such warnings
could be to regular users of other scripts.

However, since warnings are only (potentially) annoying rather than
changing the behavior of programs, it makes sense to be aggressive with
them, in order to gauge how disruptive it would be to actually change
the way text is interpreted as code.

PS An issue that seems related is that it is presently possible to bind
the symbols ö (one character, U+00F6 LATIN SMALL LETTER O WITH DIAERESIS)
and ö (two characters, U+006F LATIN SMALL LETTER O followed by U+0308
COMBINING DIAERESIS) to different values.  This seems like the kind of
thing that should be (at least) warned about and (probably) disallowed.

-- 
Aaron Ecay

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Garreau, Alexandre]

On 2018-10-10 at 10:39, Aaron Ecay wrote:
> PS An issue that seems related is that it is presently possible to bind
> the symbols ö (one character, U+00F6 LATIN SMALL LETTER O WITH DIAERESIS)
> and ö (two characters, U+006F LATIN SMALL LETTER O followed by U+0308
> COMBINING DIAERESIS) to different values.  This seems like the kind of
> thing that should be (at least) warned about and (probably) disallowed.

Oh yes… I confirm, this is the case here too (but my version is only
25.1.1, so maybe it changed), it really shouldn’t be that way at all
(tried with é and é (btw these two display quite strangely slightly
differently…))…

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> From: "Garreau\, Alexandre" <galex-713@galex-713.eu>
> Date: Wed, 10 Oct 2018 13:18:09 +0200
> Cc: Andreas Schwab <schwab@linux-m68k.org>, emacs-devel@gnu.org
> 
> (tried with é and é (btw these two display quite strangely slightly
> differently…))…

It depends on your fonts.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Tue, 09 Oct 2018 21:18:51 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, npostavs@users.sourceforge.net,
> 	drew.adams@oracle.com, emacs-devel@gnu.org
> 
> I'm pretty sure you can find many Russian words that are written with
> only Latin-alike letters.

I wrote the following toy program:

  (let ((confusing '(?а ?е ?о ?р ?с ?у ?х))
	(buf (get-buffer-create "*confusing*")))
    (while (not (eobp))
      (let* ((word (buffer-substring (line-beginning-position)
				     (line-end-position)))
	     (chars (append word nil)))
	(if (null (seq-difference chars confusing))
	    (with-current-buffer buf
	      (insert word ?\n))))
      (forward-line))))

and ran it on a list of 174800 words from a Russian dictionary.  The
result was 60 words that used only Latin-alike letters.  So, not too
many, but not just a few, either.

Of course, there are many more non-word combinations of the above
letters that might look like Latin words.

Also note that "confusability" sometimes depends on letter-case.  For
example, the lower-case "вор" doesn't look like a Latin word, but the
upper-case "ВОР" does.

RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams]

It sounds like the contexts where a char might be confused
with another are varied and depend on things that can even
include user attention and intention.

If we want to help users be aware of character-confusion
possibilities then I think whatever we offer them in this
regard needs to be (1) optional and (2) configurable
(granularity, specifying contexts/uses/conditions, etc.).

I think we can offer to help by highlighting characters (or
their surrounding contexts, e.g., when a char is tiny or
otherwise unobtrusive or invisible).

I think we should avoid raising errors, but that could be
an option that some users might want to choose in some
contexts. We could perhaps offer a range of help
responses, from a range of highlighting possibilities to
outright error-raising.

We can have code that tries to be clever, but that should
only be used if asked for by a user. We should not try to
second-guess text or users by default.

The last thing we should want is to bother users by
default, or systematically, warning them left and right
about possibilities of confusion. Such warnings or
notifications or highlights need to be opt-in, IMHO.

Above all, Emacs, and especially Emacs Lisp, should
continue to be an environment where you can do what
you want without obstruction or unnecessary
hand-holding or helicopter-parenting.

(Note that I qualified that with "unnecessary". If there
is some real, strong, unambiguous danger that we can
identify then of course we need to offer protection up
front. That help would not be "unnecessary".)

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Yuri Khan]

On Wed, Oct 10, 2018 at 10:20 PM Eli Zaretskii <eliz@gnu.org> wrote:

>   (let ((confusing '(?а ?е ?о ?р ?с ?у ?х))

> Also note that "confusability" sometimes depends on letter-case.  For
> example, the lower-case "вор" doesn't look like a Latin word, but the
> upper-case "ВОР" does.

Yes. In uppercase: ?А ?В ?Е ?К ?М ?Н ?О ?Р ?С ?Т ?Х. (Coincidentally,
Russian car license plates use confusable letters exclusively, so that
people who are not fluent in the Cyrillic script could still report
violations.)

Also, confusability depends on font style. In lowercase italic, these
pairs are also confusable: д/g, з/z, и/u, п/n, т/m, ч/r.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Juri Linkov]

>>   (let ((confusing '(?а ?е ?о ?р ?с ?у ?х))
>
>> Also note that "confusability" sometimes depends on letter-case.  For
>> example, the lower-case "вор" doesn't look like a Latin word, but the
>> upper-case "ВОР" does.
>
> Yes. In uppercase: ?А ?В ?Е ?К ?М ?Н ?О ?Р ?С ?Т ?Х. (Coincidentally,
> Russian car license plates use confusable letters exclusively, so that
> people who are not fluent in the Cyrillic script could still report
> violations.)

There are programs composed completely of confusable characters like
http://compuhumour.narod.ru/listing/prog_tormoz.html

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Sure, that's fine. We can limit symbol warnings to the symbols containing 
  > non-ASCII chars all of which are confusable with ASCII. This will warn about 
  > "саn" (with Cyrillic "с" and "а") but not about "сталин" (with Cyrillic "a").

I agree, but we need to extend this protection to things other than
program code.

  > The point of the guideline is not to warn about every possible confusable 
  > character; it's to defend against malicious code.

Confusables are dangerous in host names, too.

The same principle could apply: if the host name contains, as well as
the confusables, some non-confusable non-ASCII characters from the
same Unicode page, there is no need to warn about it.

Perhaps there are other cases, too.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The problem is that deciding which characters are confusable and hence
  > require backslash escaping is based on a shifting mess of heuristics.
  > So I don't think it's workable to signal a hard error for this. Both
  > in terms of false positives which could mean possibly breaking code,
  > and false negatives which means we would be giving a false sense of
  > security.

In principle, that\s a valid point.  But can't we assembe a fixed list
of characters that are confusable with the usual fonts. and base the
warning on that list?

We could conceivably have a feature that would check any fontset for
confusable characters, and warn the user if it has confusable pairs
that are not in the usual list of confusable pairs.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eggert@cs.ucla.edu, eliz@gnu.org, drew.adams@oracle.com,
> 	emacs-devel@gnu.org
> Date: Tue, 09 Oct 2018 23:57:01 -0400
> 
> We could conceivably have a feature that would check any fontset for
> confusable characters

I'm not an expert on fonts, but I don't think this is reliable enough.
Are you saying that a font might use the same glyph for similarly
looking characters from different scripts?  If that is true, then yes,
we could detect that.  But the fact that we have such a font doesn't
yet mean there is a problem worth warning the user, since these
characters need to be _used_ in a certain context to cause confusion.

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'm not an expert on fonts, but I don't think this is reliable enough.
  > Are you saying that a font might use the same glyph for similarly
  > looking characters from different scripts?  If that is true, then yes,
  > we could detect that.  But the fact that we have such a font doesn't
  > yet mean there is a problem worth warning the user, since these
  > characters need to be _used_ in a certain context to cause confusion.

Which characters are confusable is one question,
and which contexts they can cause confusion in is another question.

We need to distinguish those, to factor the problem.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

eval-last-sexp / C-x C-e, and punctuation like `?’' [Was: Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?)]

[Garreau, Alexandre]

On 2018-10-04 at 20:03, Noam Postavsky wrote:
> On Fri, 2 Feb 2018 at 17:24, Noam Postavsky
> <npostavs@users.sourceforge.net> wrote:
>>
>> In Emacs 26 and earlier the following is valid lisp code:
>>
>> (setq ’bar 42)
>> (setq foo ’bar)

I just noticed: in emacs 25, if evaluating `’bar' with `eval-last-sexp'
/ C-x C-e, this gives an error as it ignores the ?’ and eval only `bar',
the same way, if point is placed after the ?’, it tries to eval “setq”…

Maybe I do not know enough of elisp, but why that? are there other
punctuation characters triggering this behavior?  meanwhile, are they
all okay for the reader to put in symbols unescaped (except ? , ?\",
?\(, ?\), ?,, ?`, and maybe some others from ascii I forgot)?

Why does eval-last-sexps treat this differently than the reader?

Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Garreau, Alexandre]

On 2018-10-04 at 20:03, Noam Postavsky wrote:
> On Fri, 2 Feb 2018 at 17:24, Noam Postavsky
> <npostavs@users.sourceforge.net> wrote:
>>
>> In Emacs 26 and earlier the following is valid lisp code:
>>
>> (setq ’bar 42)
>> (setq foo ’bar)
>>
>> In the current master branch, this will signal (invalid-read-syntax
>> "strange quote" "’").

Btw, aren’t there any ways of, at least locally, extending/redefining
such reader behavior such as the one of “'”, “,”/“,@”, “`”, “.”, “:”,
etc.?

For instance to experiment having such fancy and strange quotes in
source code: people really wanting to use it *might* want to use it as
such, instead of symbol component, which, inside ascii, often (with a
lot of exceptions such as in “!”, “?”, “:” (though it can have special
meaning) or “.” (though this one doesn’t work alone) other
non-human-text “punctuation” (also named “special characters”) such as
in “%&*+/<>=@^_|”) cannot be part of a symbol without escaping (for
instance: “"#'(),;\[]`” (though this is tiny, it is not that simple)).