RE: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Drew Adams <drew.adams@oracle.com>]

> Those bug reports complained about obscure error messages that are
> unhelpful when a Lisp programmer tries to figure out the root cause.
> I'm saying that we should find an alternative way of making clear,
> helpful error messages in those special cases where characters which
> display similarly might make the error message confusing if it just
> cites the symbol's name.

OK.  Except I would say warnings, not error messages, at
least in most cases.

But even if we have an error message, that's not a call
to change the syntax of Lisp.  User errors happen.  We
should just want to help users avoid making such errors.

> For example, suppose you have a Lisp program that produces
> the following error message when compiled/executed:
> 
>   Symbol's value as variable is void: 'аbbrevs-changed
> 
> You then type "C-h v abbrevs-changed RET" and get the expected result,
> meaning that the variable is known to Emacs.  How quickly will you be
> able to spot the cause of the error message?

Some people will wonder for a while.  Others, perhaps
already bitten by this gotcha, will notice the quote
mark there right away.

One thing that would help, I think, and which should
be done in general, would be to put the offending
thingie between `...':

 Symbol's value as variable is void: `'аbbrevs-changed'

That makes it more obvious that the symbol name
includes that fancy quote char.

Still, all of this is pilot error, where "pilot" can
include the user who wrote the code but more likely
means a user who copy+pasted it.

> The change that got reverted from the emacs-26 branch was about a
> similar case, but for a character that's much more important for Lisp
> than 'a': it's about the character used to quote symbol names.  But
> the essence is the same: due to how characters are displayed, some
> characters can be confused for others.
> 
> We want to find a way of identifying such situation and telling the
> Lisp programmer about that in clear and easily understandable ways.
> One way, perhaps too radical one, is to reject such "confusable"
> characters outright.  We could decide that we don't want such a
> radical solution, but that doesn't mean we should give up on the
> attempt to find some other solution for the problem.  Neither does it
> mean we should proclaim people who installed the change as enemies of
> the society.

Agreed.  As I've said, I'm in favor of providing
friendly warnings/reminders that point out that
such a character is present.

I think that should be enough.

There are lots of potential confusables, and lots
of different use contexts.  But if we start with
just one or two such chars and one or two common
and clear contexts where a warning might help, that
would be good.  We can always add more such warnings
as cases come up (get reported or otherwise become
obvious).

It would be an overreaction, IMO, to jump to
changing the existing Lisp syntax to raise errors
when someone uses such a character in, say, a symbol
name.  We should not require such chars to be
escaped in a symbol name.  Such chars have no special
meaning for Lisp (unlike `.', `,' `'', ``', `(', `)',
`[', `]', `"', `<', `>', `#' `;', and perhaps some more).

> > Bug #23425, on the other hand, is a gigantic
> > stream-of-consciousness about anything and
> > everything [...]
> > [...]
> > How is it helpful to throw all of #23425 into
> > this Lisp syntax-change question, as if the
> > present issue puts into question everything
> > ever discussed about curly quotes?
> 
> I could turn the table and ask you how is it helpful
> to dump on us all your random thoughts about this,
> instead of simply saying you didn't understand the
> relevance and asking for more explanations.  Which I
> just provided.

Whoa!  I don't see a connection between the current
issue and the many things discussed in #23425.  And
I don't think I dumped any random thoughts on anyone.

> I hope now the issue is clear enough.

No idea what your point is there.

If there is some part of bug #23425 that you think
is relevant here, and you think it will be UNfixed by
reverting the Lisp-syntax change made for bug #30217,
please tell us what that part is.

I don't see anything in #23425 that needs the change
in Lisp syntax made for #30217.  And I don't see that
Lisp change being necessary to fix #30217 either.
It wasn't requested by the bug filer, AFAIK.  Same
for the other bugs you mentioned.  The filers just
asked for warnings, AFAICT.

> > And the original error message from bug #23425
> > is _more_ meaningful and helpful, not less,
> > than the new one after the "fix".
>
> I think you are so eager to make your point that you are willing to
> claim that black is white and vice versa.  Any objective person would
> agree that the new error message is more directly pointing to the root
> cause, which is the syntax of specifying a quoted symbol name using a
> "strange quote".  If we are good in writing and indexing our ELisp
> manual, then I'd expect to find there an index entry for "strange
> quote", which will land me where this issue is explained.  Case
> closed.

We can perhaps agree to disagree about that.
But of course if you say the case is closed then
it's closed.

> Once again, I can agree that this measure might be too harsh, but I
> would still like to see clear diagnostics of such typos, and like
> Paul, I thing we should take our inspiration from the Unicode
> Standard's notion of "confusables".

I've agreed about that from the beginning.  It can
be helpful to warn users about possible confusion
when they use confusables.  And I agree that clear
diagnostics are needed - that was one of my points.

That's different from changing the syntax of Lisp.

> Ideas and proposals for patches along those lines
> are welcome.

Ditto.

> Ignoring the problem, or trying to convince us
> that it doesn't exist, is not.

I recognize the problems of confusable characters.
Not all such possible confusions are equally likely,
in practice.

Recognizing contexts where something might well be
a typo, and warrants a helpful reminder/warning, is
what's needed - case by case.

What's not needed, IMO (and probably the only place
where I differ from you on this, even if you don't
want to recognize it) is a change in Lisp syntax,
making it a read error not to escape such a character.

> > Either doing nothing or trying to warn about such
> > gotchas is right.  Changing Lisp syntax here is
> > not right.
> 
> Doing nothing would be ignoring the problem.

Yes.  It's maybe not the best help for users, but
it would be one way to handle those few reports of
confusion.  We get a lot more questions due to
other confusions wrt Lisp than we do such questions
due to confusing one char for another.

I didn't, and don't, say that doing nothing is the
best approach.  I said it's one way to deal with
such reports.  Unlike changing Lisp syntax, it at
least doesn't introduce new problems.

> That changing Lisp syntax is not right is your
> opinion: legitimate, but clearly not shared by at
> least some.

That's why we're having this discussion.

I have yet to hear a reason why it is right to
change Lisp syntax for this - why a simple warning
is not sufficient and we need to also make Lisp
raise an error.

> > Lisp doesn't have a bug here.
> 
> That's a strawman, and you know it.  We are talking
> about diagnostics for bugs in Lisp programs.

I have no objection to diagnostics.  Add warnings
for byte-compilation, loading, whatever.

Make sure the warnings are clear.  Say, for instance
that a curly quote was used in sexp `...'.  Don't
just say that invalid syntax was read (somewhere).
Clearly pointing out the confusable char in the
possibly confused sexp should go a long way to
making things clear.

My objection is to making such chars be escaped to
prevent Lisp from raising an error.  I don't put
`a’b' in the same class as, say, `a,b'.

`,' is special in Lisp, and (setq a,b 42) should
(and does) raise an error.  `’' is not special in
Lisp, and (setq a’b 42) should not raise an error (IMO).
Likewise, (setq ,b 42) (yes) and (setq ’b 42) (no).

If you want to argue for this syntax change, why
not address some of my arguments against it?  Where
will you draw the line, for instance?  There are
_lots_ of possible confusables.

I'd say start with only the few that have actually
been reported (is there only one reported?), trying
to come up with reasonable warnings in particular
contexts of use.  That would be a good start.

We might even have a user option that lists the
confusables to check/warn for, with whatever
default value people here think is best (it might
be only `’', to start with - or both left and
right curly quotes).

Are you thinking instead (since both you and Paul
mentioned the Unicode list of confusables) of
starting with _all_ characters in that list?

http://www.unicode.org/Public/security/8.0.0/confusables.txt

I won't argue about which chars should be warned
about, though I might be interested to see what
contexts we warn for and what the messages will be.

My objection is not about detecting this or that
use of this or that character and warning/reminding
users about it.

My objection is to making Lisp require escaping of
such characters.  That's all.  I think I've made
that as clear as I possibly can.

But you seem to want to paint my objection as
being against helping users know about accidental
use of confusables, e.g., `’' instead of `''.  Why?