Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Alan Mackenzie]

[ Message posted to emacs-devel@gnu.org, with Reply-To: set. ]

Hello, Paul.

On Fri, Jun 02, 2017 at 14:47:42 -0700, Paul Eggert wrote:
> On 06/02/2017 02:02 PM, Alan Mackenzie wrote:
> > How about transferring this discussion to emacs-devel, so that more
> > people would be able to see it.

> We had that discussion there already, and I doubt whether it’d be 
> worthwhile to repeat it. Of course you're free to take it there again if 
> you like.

The previous discussion was abruptly cut short, without reaching a
conclusion, with a "we're too close to a release to change anything
now".  This came with the implicit understanding the discussion could be
countinued after the release.

> > The default behaviour has been changed already, and that change has
> > caused problems and will cause further problems.

> I expect that most of the practical problems have been shaken out 
> already with the changes published in Emacs 25.1 last year, so that most 
> of this cost has already been paid.

They have not.  We have moved from a state where everybody knew what
`message' did (in Emacs 24), to one with wild special characters which
only apply sometimes, and necessitate crazy prolix formulations to work
around unwanted translations of quote characters.

`message' is currently broken.

Introducing quote translation has introduced a burden on all users of
`message'.  Seeing how poisonous and insidious this translation can be,
it makes sense to shift this burden over to the use cases where the
programmers need quote translation, and hence will be aware of it.

> > I propose restoring the default to what it was in Emacs 24.  For
> > simplicity's sake, only % should be special, and for translating quotes
> > we could introduce %' and %`, such that quote translation happens only
> > when a %' or %` is present.

> That kind of approach was proposed way back when. It doesn’t address 
> quotes in doc strings, though.

Of course it doesn't.  Doc strings have their own peculiar quoting
mechanisms which are disjoint from those of `message'.

> More generally, a problem with this sort of approach is that although
> it simplifies ‘message’ (obviously), this is at the price of
> complicating everything else.

What is the "everything else" that gets thus complicated?  I don't see
anything else getting more complicated.

> I estimate we’d need to change 10,000 lines of Emacs source code to
> make that change to ‘message’. (This is a just a quick estimate based
> on ‘grep '".*`.*"'’.) The resulting source code would typically be a
> bit harder to read than it is now.

I think this estimate is way over the mark.  There are around 17,000
occurrances of "message" in our Lisp sources, and probably a few in our
C sources.  Only (some of) those containing the quote characters in the
format string would need amendment.  These will comprise a tiny portion
of these ~17,000, and can be found easily enough with a script.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Paul Eggert]

Alan Mackenzie wrote:

> We have moved from a state where everybody knew what
> `message' did (in Emacs 24), to one with wild special characters which
> only apply sometimes, and necessitate crazy prolix formulations to work
> around unwanted translations of quote characters.

This exaggerates somewhat. We moved from Emacs 24 where only % is special, to 
Emacs 25 where %, ` and ' are special. Although some people don't know that ` 
and ' are special, that's also true for %. And although it can be annoying to 
write (message "%s" STR) to avoid unwanted translation of STR, that annoyance 
was already present for %.

> it makes sense to shift this burden over to the use cases where the
> programmers need quote translation, and hence will be aware of it.
When text-quoting-style specifies translation, most instances of ` and ' in 
Emacs messages are better off translated. So it also makes sense to translate by 
default in this situation, with a way to avoid translation in the rare cases 
where translation isn't wanted. The question is about which approach makes more 
sense, not whether one approach is sensible and the other nonsense.

>> although it simplifies ‘message’ (obviously), this is at the price of
>> complicating everything else.
> 
> What is the "everything else" that gets thus complicated?

I was referring to the hassle of going through hundreds or thousands of message 
strings or calls, deciding which instances of ` and ' should be replaced with %` 
and %', and replacing the instances accordingly. It's also possible that at 
times we'll need two format strings instead of one, complicating the code.

> There are around 17,000
> occurrances of "message" in our Lisp sources, and probably a few in our
> C sources.  Only (some of) those containing the quote characters in the
> format string would need amendment.  These will comprise a tiny portion
> of these ~17,000

How many lines do you think will be in that "tiny portion"? No matter how you 
count them, it'll be quite a few changes.

> and can be found easily enough with a script

I'm afraid not, because in many cases the string is not a simple literal 
constant argument to the message function. For starters, there's also the error 
function; that's another 14,000 text matches in the Elisp source -- many of them 
false alarms of course, but not all of them.

I'm not saying this sort of change is impossible. It's just that it'd be quite a 
bit of work, work that someone would need to volunteer to do. Is this really the 
best use of our limited resources?

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Alan Mackenzie]

Hello, Paul.

On Sun, Jun 04, 2017 at 14:01:42 -0700, Paul Eggert wrote:
> Alan Mackenzie wrote:

> > We have moved from a state where everybody knew what
> > `message' did (in Emacs 24), to one with wild special characters which
> > only apply sometimes, and necessitate crazy prolix formulations to work
> > around unwanted translations of quote characters.

> This exaggerates somewhat. We moved from Emacs 24 where only % is special, to 
> Emacs 25 where %, ` and ' are special.

Yes.  We moved from regularity (where %x, for varying x, and nothing
else was special) to a ragbag (where there are 3 special characters,
with the two "new" ones being syntactically totally different from %).

> Although some people don't know that ` and ' are special, that's also
> true for %.

No.  _Anybody_ who's used `message' knows that %s is how you print out
an arbitrary sexp.  Anybody who's used printf in C knows this, too.  It
is very easy not to know that ` and ' are special, and horribly easy to
get caught out by it, as happened to me.

> And although it can be annoying to write (message "%s" STR) to avoid
> unwanted translation of STR, that annoyance was already present for %.

It is not merely annoying, it is hideously irregular.  Having to write
(message "%s" (format "...." arg1 arg2 ....)) screams out "we didn't
think this through properly".  A call to message should only need one
format string.  The change I am proposing would achieve this.

This was never the case for %.  It is and always was trivially easy to
cause a literal % sign to be output by message, and there was never
danger of confusion in this.

We also have `format' and `format-message' which handle format strings
inconsistently.  (Yes, I know, `format-message' was introduced
deliberately to create this inconsistency, because `format' was no
longer able to cope on its own.)

> > it makes sense to shift this burden over to the use cases where the
> > programmers need quote translation, and hence will be aware of it.

> When text-quoting-style specifies translation, most instances of ` and ' in 
> Emacs messages are better off translated. So it also makes sense to translate by 
> default in this situation, with a way to avoid translation in the rare cases 
> where translation isn't wanted.

I disagree with this, of course.  Translating behind people's backs is
not a friendly thing to do.  Translation should only be done where it is
specifically specified.

> The question is about which approach makes more sense, not whether one
> approach is sensible and the other nonsense.

OK.

> >> although it simplifies ‘message’ (obviously), this is at the price of
> >> complicating everything else.

> > What is the "everything else" that gets thus complicated?

> I was referring to the hassle of going through hundreds or thousands of message 
> strings or calls, deciding which instances of ` and ' should be replaced with %` 
> and %', and replacing the instances accordingly.

Yes.  There are quite a lot, but not an unmanageable number.

> It's also possible that at times we'll need two format strings instead
> of one, complicating the code.

We need two strings instead of one at the moment, with (message "%s"
(format "..." .....)).  With %` and %' we'd only need one string in each
message invocation.  This is simplification.

Can you give an example of something which might need two strings?

> > There are around 17,000 occurrances of "message" in our Lisp
> > sources, and probably a few in our C sources.  Only (some of) those
> > containing the quote characters in the format string would need
> > amendment.  These will comprise a tiny portion of these ~17,000

> How many lines do you think will be in that "tiny portion"? No matter how you 
> count them, it'll be quite a few changes.

By searching for 

    "(\\(message\\|error\\)\\s +\\([^\"]\\|\"\\(\\\\.\\|[^\"\\]\\)*[`']\\)"

, i.e. an invocation of message or error followed by either something
which isn't a literal string, or a literal string containing ` or ', I
get 2745 matches in our Lisp sources.  There'll be a smaller number also
in our C sources.  I would have to enhance that regexp to recognise
comments, and maybe a few other things, but 2745 is a good first
approximation.

A very great number of these are "(error ..." handlers in condition-case
forms.  A great number of those remaining could be simply and
mechanically translated, for example "don't", "can't", "couldn't", etc.,
and a lot of "`%s'"s and "`%S'"s.

I estimate there will be a few hundred forms remaining which need
decision making to adapt them.  For example, where message is used in
macros, and the format string is a macro parameter.

> > and can be found easily enough with a script

> I'm afraid not, because in many cases the string is not a simple literal 
> constant argument to the message function. For starters, there's also the error 
> function; that's another 14,000 text matches in the Elisp source -- many of them 
> false alarms of course, but not all of them.

See above.

> I'm not saying this sort of change is impossible. It's just that it'd be quite a 
> bit of work, work that someone would need to volunteer to do. Is this really the 
> best use of our limited resources?

Clearly, that someone would have to be me.  The consequences of
surreptitious unwanted translation are so severe that I think this would
indeed be a good use of resources.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Paul Eggert]

On 06/05/2017 09:27 AM, Alan Mackenzie wrote:

> Can you give an example of something which might need two strings?

Suppose the same string is used for both messages and documentation. 
You're proposing %` in the former where ` is in the latter, so we would 
need distinct strings.People cut and paste between messages and doc 
strings, and it's possible there's even code that does this (I haven't 
checked). Using different quoting notations for the two kinds of strings 
will cause confusion and problems.

> 2745 is a good first approximation.

This is being optimistic. It's not just the message and error functions; 
there are others, ranging from general functions like user-error to 
more-specialized functions like icalendar--dmsg.These cannot be found 
merely by a simple text search for function names. A simple albeit 
tedious and somewhat error-prone approach would be to examine every ` 
and ' in every string and character constant (in either C or Elisp) and 
fix those intended for the message function or any of its callers. This 
is the sort of thing that I did when converting for Emacs 25.

> The consequences of surreptitious unwanted translation are so severe

It's not surreptitious: it's documented. And the consequences are not 
severe: they are a minor change to strings intended for human 
consumption, where humans can easily recover from unwanted translation, 
and where humans who don't want translation can easily turn it off.

As I understand it, you're proposing making hundreds or thousands of 
changes like this:

          (error "Can't find `%s' in %s" thing where)))
  =>      (error "Can%'t find %`%s%' in %s" thing where)))

These changes will adversely affect code readability, and this will have 
a maintenance cost in the long run -- not just to whoever makes the 
hundreds or thousands of changes, but to everybody who has to read the 
code afterwards.It's not at all clear that this cost is greater than the 
benefit of fixing some of the minor problems that you're objecting to.

> _Anybody_ who's used `message' knows

Perhaps I should introduce you to my students someday. I teach them 
elements of Emacs and I assure you that some of them do not know. 
Although I readily concede that more people know about % than ` or ', 
this doesn't alter the fact that it causes problems to support special 
characters of any sort in the message function. Elisp code needed to use 
(message "%s" STR) even before the change you're objecting to, and it'd 
still need (message "%s" STR) even if the change were reverted.

> It is not merely annoying, it is hideously irregular. Having to write
> (message "%s" (format "...." arg1 arg2 ....)) screams out "we didn't
> think this through properly".

Yes, and that's just as true for % as it is for ` and '. In hindsight, 
the message function should have had a better API. Hindsight is 
wonderful....

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Andreas Schwab]

On Jun 05 2017, Paul Eggert <eggert@cs.ucla.edu> wrote:

> Suppose the same string is used for both messages and
> documentation. You're proposing %` in the former where ` is in the latter,
> so we would need distinct strings.People cut and paste between messages
> and doc strings, and it's possible there's even code that does this (I
> haven't checked). Using different quoting notations for the two kinds of
> strings will cause confusion and problems.

That's a bad argument.  A doc string already has completely different
special characters.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Alan Mackenzie]

Hello, Paul.

On Mon, Jun 05, 2017 at 11:19:06 -0700, Paul Eggert wrote:
> On 06/05/2017 09:27 AM, Alan Mackenzie wrote:

[ .... ]

> > 2745 is a good first approximation.

> This is being optimistic. It's not just the message and error functions; 
> there are others, ranging from general functions like user-error to 
> more-specialized functions like icalendar--dmsg.These cannot be found 
> merely by a simple text search for function names.

Maybe not, but 2745 remains a good first approximation.  Other functions
which call message will be found in their turn, and the number of calls
of things like icalendar--dmsg will be small, both individually and in
total.  There are around 275 calls to message which have a non-literal
format argument.

> A simple albeit tedious and somewhat error-prone approach would be to
> examine every ` and ' in every string and character constant (in
> either C or Elisp) and fix those intended for the message function or
> any of its callers. This is the sort of thing that I did when
> converting for Emacs 25.

> > The consequences of surreptitious unwanted translation are so severe

> It's not surreptitious: it's documented.

And this documentation is useless for preventing the problems.  Somebody
using message to output Lisp will use ' just as I did - and suffer the
same horrendous problems trying to make sense of non-sensical error
messages saying 'foo does not exist, when it plainly does.  Do you,
perhaps, have another strategem for preventing this problem?

As I said, the burden of dealing with this translation should fall on
those who are actually and consciously using it.  Unless, of course,
you're in favour of hackers remaining unaware of the stealthy
alterations being made to their expressed intentions.  Such a burden, on
those wanting the translation, is small.  

> And the consequences are not severe: they are a minor change to
> strings intended for human consumption, where humans can easily
> recover from unwanted translation, and where humans who don't want
> translation can easily turn it off.

When I got caught by this bug there was no "easy" way to turn off the
translation.  The process of turning it off involved a long period of
puzzlement over the non-sensical error messages referred to above, and
an even longer period of being very angry at having my work messed up by
Emacs.  How do you propose to prevent such puzzlement and anger in the
future, if not by %` and %'?  And I was somebody who had been familiar
with discussions on emacs-devel over such matters.  Such things might
not seem severe to you.  I am sure they are to many people.

> As I understand it, you're proposing making hundreds or thousands of 
> changes like this:

>           (error "Can't find `%s' in %s" thing where)))
>   =>      (error "Can%'t find %`%s%' in %s" thing where)))

Yes.

> These changes will adversely affect code readability, and this will have 
> a maintenance cost in the long run

No.  The addition of quote translation is what has affected readability.
It is currently implicit and vague, a bit like the lack of connection
between written English and spoken English.  I am proposing restoring it
to an explicit form, where the sole special character is %, as it had
been for ever before Emacs 25.

I don't think you've considered the maintenance cost of this
impliciticity, this vagueness, the fact that people will be using ` and
' in format strings without understanding what they mean.

> -- not just to whoever makes the hundreds or thousands of changes, but
> to everybody who has to read the code afterwards.

Those people will be able to understand the code much more easily with
explicit %` and %' rather than the stealthy behind-the-back translation
which happens at the moment.

> It's not at all clear that this cost is greater than the benefit of
> fixing some of the minor problems that you're objecting to.

It's easy to minimise and dismiss other people's problems.  I don't see
non-sensical error messages as anything but a major problem.

One of the problems of the Emacs 25 approach is that the distinction
between ` and ', and the curly quotes has become vague, fuzzy, and
confused.  I do not want this confusion to carry on into Emacs 26.

> > _Anybody_ who's used `message' knows

> Perhaps I should introduce you to my students someday. I teach them 
> elements of Emacs and I assure you that some of them do not know.

The ones who have never used `message', you mean.  You're not seriously
telling me that any of your students who've written a message call with
a "%s" in the format string remain unaware of the role of %, are you?

> Although I readily concede that more people know about % than ` or ', 
> this doesn't alter the fact that it causes problems to support special 
> characters of any sort in the message function. Elisp code needed to use 
> (message "%s" STR) even before the change you're objecting to,

Did it?  When and why?

> and it'd still need (message "%s" STR) even if the change were
> reverted.

When and why?

> > It is not merely annoying, it is hideously irregular. Having to write
> > (message "%s" (format "...." arg1 arg2 ....)) screams out "we didn't
> > think this through properly".

> Yes, and that's just as true for % as it is for ` and '.

Not at all.  In message, % causes no problems.  ` and ' do.

> In hindsight, the message function should have had a better API.
> Hindsight is wonderful....

If you think a better API could have been conceived, outline it.  I
don't.  And even if it could, that's no excuse to worsen the workings of
message.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Clément Pit-Claudel]

On 2017-06-05 16:37, Alan Mackenzie wrote:
>  How do you propose to prevent such puzzlement and anger in the
> future, if not by %` and %'?

Note that this introduces a backwards compatibility issue, if not done carefully:

  Debugger entered--Lisp error: (error "Not enough arguments for format string")
    message("%`woops%'")
    eval((message "%`woops%'") nil)

Did something similar happen with \=' ?

Clément.

bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Alan Mackenzie]

Hello, Clément.

On Mon, Jun 05, 2017 at 17:05:43 -0400, Clément Pit-Claudel wrote:
> On 2017-06-05 16:37, Alan Mackenzie wrote:
> >  How do you propose to prevent such puzzlement and anger in the
> > future, if not by %` and %'?

> Note that this introduces a backwards compatibility issue, if not done carefully:

>   Debugger entered--Lisp error: (error "Not enough arguments for format string")
>     message("%`woops%'")
>     eval((message "%`woops%'") nil)

I think it would be done carefully!  Clearly, %` and %' would be format
elements for which the count of necessary arguments would not be
incremented.  Or, more concisely, we wouldn't count them.

[ .... ]

> Clément.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Clément Pit-Claudel]

On 2017-06-07 12:50, Alan Mackenzie wrote:
> Hello, Clément.
> 
> On Mon, Jun 05, 2017 at 17:05:43 -0400, Clément Pit-Claudel wrote:
>> On 2017-06-05 16:37, Alan Mackenzie wrote:
>>>  How do you propose to prevent such puzzlement and anger in the
>>> future, if not by %` and %'?
> 
>> Note that this introduces a backwards compatibility issue, if not done carefully:
> 
>>   Debugger entered--Lisp error: (error "Not enough arguments for format string")
>>     message("%`woops%'")
>>     eval((message "%`woops%'") nil)
> 
> I think it would be done carefully!  Clearly, %` and %' would be format
> elements for which the count of necessary arguments would not be
> incremented.  Or, more concisely, we wouldn't count them.

Sorry, that's not what I was talking about.  The problem I was worrying about is that these new format constructs would raise exceptions in older versions of Emacs.

bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Paul Eggert]

 >> Elisp code needed to use
 >> (message "%s" STR) even before the change you're objecting to,
 >
 > Did it?  When and why?

Yes, because one can’t pass arbitrary strings to the message function 
and expect them to be displayed as-is. This has been true for years. For 
example, lisp/ChangeLog.8 says this:

   1999-02-23  Richard M. Stallman  <rms@gnu.org>

     * subr.el (with-temp-message): Use %s so % in old msg won't fool us.

This use of (message "%s" ...) is still in master, and would still be 
needed even if we went back to Emacs 24-style message formatting.

 > Somebody using message to output Lisp will use ' just as I did - and 
suffer the
 > same horrendous problems

Sure, like the “horrendous” problems with %. For example, in Emacs 24 
your example, when used with data involving %:

   (setq c-parse-state-state '((a '%%)))
   (message
    (concat "(setq "
     (mapconcat
      (lambda (arg)
        (format "%s %s%s" (car arg)
            (if (atom (cdr arg)) "" "'")
            (if (markerp (cdr arg))
            (format "(copy-marker %s)" (marker-position (cdr arg)))
          (cdr arg))))
      c-parse-state-state "  ")
     ")"))

returns "(setq a '((quote %)))", which is obviously wrong and which 
results in silent data corruption.

 > Do you, perhaps, have another strategem for preventing this problem?

Sure: don’t pass arbitrary strings to the message function.

 > How do you propose to prevent such puzzlement and anger in the future

Not by this:

 >>           (error "Can't find `%s' in %s" thing where)))
 >>   =>      (error "Can%'t find %`%s%' in %s" thing where)))

For Emacs code this would likely be a cure worse than the disease, by 
causing more puzzlement and anger than it would prevent. It would make 
formats significantly harder to read. And as Clément mentioned, it would 
introduce compatibility problems of its own.

There is a better way if the primary goal is to avoid quote translation:

            (error "Can't find `%s' in %s" thing where)))
    =>      (error "Can’t find ‘%s’ in %s" thing where)))

Compared to %` and %', this is simpler, easier to read, and more 
compatible with current and older Emacs versions. A downside, though, is 
that it would involve changing hundreds or thousands of strings in the 
Emacs source (just as %` and %' would).

 > You're not seriously
 > telling me that any of your students who've written a message call with
 > a "%s" in the format string remain unaware of the role of %, are you?

Sure, they learn about % after the message function doesn’t work the way 
they naively expected. In that respect, % is like ` and '.

 > There are around 275 calls to message which have a non-literal
 > format argument.

Each one stands for possibly many other calls, and we don’t know how 
many of these other calls might cause a problem.

 > The consequences of surreptitious unwanted translation ...
 >> It's not surreptitious: it's documented.
 > And this documentation is useless for preventing the problems.

True, documentation by itself does not prevent programming problems. 
However, this doesn’t change the fact that quote translation is 
documented. It is not “surreptitious” or “implicit” or “vague” or 
“stealthy” or “fuzzy”.

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Andreas Schwab]

On Jun 05 2017, Paul Eggert <eggert@cs.ucla.edu> wrote:

>>> Elisp code needed to use
>>> (message "%s" STR) even before the change you're objecting to,
>>
>> Did it?  When and why?
>
> Yes, because one can’t pass arbitrary strings to the message function and
> expect them to be displayed as-is.

Another bad argument, because we are talking about format strings, which
you cannot be replaced by "%s".

>>>           (error "Can't find `%s' in %s" thing where)))
>>>   =>      (error "Can%'t find %`%s%' in %s" thing where)))

A better way would be a new modifier to place quotes around the
argument.  That's what GCC is using now.  Also, contractions like don't
should be avoided anyway.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Clément Pit-Claudel]

On 2017-06-06 04:21, Andreas Schwab wrote:
> Also, contractions like don't
> should be avoided anyway.

That doesn't matter much in the long run, I think: we intend to support languages in which apostrophes in the middle of words can't be avoided.

Clément.

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Yuri Khan]

On Tue, Jun 6, 2017 at 7:24 PM, Clément Pit-Claudel
<cpitclaudel@gmail.com> wrote:
> On 2017-06-06 04:21, Andreas Schwab wrote:
>> Also, contractions like don't
>> should be avoided anyway.
>
> That doesn't matter much in the long run, I think: we intend to support languages in which apostrophes in the middle of words can't be avoided.

In that long run, I have more news for you: there are languages which
have a letter that looks almost, but not exactly, like the apostrophe
punctuation mark, but is encoded differently. (E.g. Ukrainian, U+02BC
Modifier Letter Apostrophe, as opposed to U+2019 Right Single
Quotation Mark).

Good luck substituting the correct character in each case given only '
U+0027 Apostrophe.

bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Clément Pit-Claudel]

On 2017-06-06 09:55, Yuri Khan wrote:
> On Tue, Jun 6, 2017 at 7:24 PM, Clément Pit-Claudel
> <cpitclaudel@gmail.com> wrote:
>> On 2017-06-06 04:21, Andreas Schwab wrote:
>>> Also, contractions like don't
>>> should be avoided anyway.
>>
>> That doesn't matter much in the long run, I think: we intend to support languages in which apostrophes in the middle of words can't be avoided.
> 
> In that long run, I have more news for you: there are languages which
> have a letter that looks almost, but not exactly, like the apostrophe
> punctuation mark, but is encoded differently. (E.g. Ukrainian, U+02BC
> Modifier Letter Apostrophe, as opposed to U+2019 Right Single
> Quotation Mark).
> 
> Good luck substituting the correct character in each case given only '
> U+0027 Apostrophe.

Do you mean that in these languages the casual way is to write ', though the formal way would be ʼ?

Clément.

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Yuri Khan]

On Wed, Jun 7, 2017 at 1:21 AM, Clément Pit-Claudel
<cpitclaudel@gmail.com> wrote:

>> In that long run, I have more news for you: there are languages which
>> have a letter that looks almost, but not exactly, like the apostrophe
>> punctuation mark, but is encoded differently. (E.g. Ukrainian, U+02BC
>> Modifier Letter Apostrophe, as opposed to U+2019 Right Single
>> Quotation Mark).
>
> Do you mean that in these languages the casual way is to write ', though the formal way would be ʼ?

Hm, I haven’t thought about that. That depends.

The default XKB Ukrainian layout maps the <TLDE> key to U+0027 and
Shift+<TLDE> to U+02BC. I have no knowledge whether X11/GNU/Linux
users actually bother to press Shift.

The Ukrainian layout on Windows (Vista and above) has only U+0027.
There exist third-party typographic layouts but I assume they are only
used by people who are (1) aware of the difference, and (2) care.

So, yes, I expect that the majority of Ukrainian computer users do in
fact enter U+0027 when the correct character is U+02BC.

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Clément Pit-Claudel]

On 2017-06-06 14:59, Yuri Khan wrote:
> So, yes, I expect that the majority of Ukrainian computer users do in
> fact enter U+0027 when the correct character is U+02BC.

Makes sense, thanks for clarifying.
It might be possible to enforce that translation files use the proper character (people writing translations for Emacs will be able to rely on an appropriate Emacs input method).  Can you confirm that this would sidestep the issue that you mentioned earlier?  We'd never have ' where ʼ is meant, and so automatic `'-translation wouldn't cause unexpected issues.

(The language that prompted my initial email was unsurprisingly French, which uses ' between consecutive characters (it'd be a pain to have to type "l%'arrivée d%'un email", and I don't know anyone who types in the proper "l’arrivée d’un email", except when their text editor auto-corrects).

Clément.

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 6 Jun 2017 15:39:57 -0400
> Cc: Andreas Schwab <schwab@suse.de>, Paul Eggert <eggert@cs.ucla.edu>,
> 	23425@debbugs.gnu.org, Alan Mackenzie <acm@muc.de>,
> 	Emacs developers <emacs-devel@gnu.org>
> 
> (The language that prompted my initial email was unsurprisingly French, which uses ' between consecutive characters (it'd be a pain to have to type "l%'arrivée d%'un email", and I don't know anyone who types in the proper "l’arrivée d’un email", except when their text editor auto-corrects).

You should know that there are people who claim that using u+2019 in
"l’arrivée" is incorrect, and that one should write "lʼarrivée"
instead.  The rationale is that u+2019 is a punctuation character, so
it doesn't belong in the middle of a word.  There was a long
discussion about that on the Unicode list a couple of months ago.

bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Paul Eggert]

On 06/06/2017 01:21 AM, Andreas Schwab wrote:
> we are talking about format strings, which you cannot be replaced by "%s".

The original bug report is about the ‘message’ function, not about 
formats in general. For that function, "%s" is the longstanding way to 
output arbitrary strings. For formats in general, one can use the 
‘format’ function (which does not translate quotes) or the 
‘format-message’ function (which does); this suffices in practice.

> contractions like don't should be avoided anyway

As a practical matter Emacs messages often have contractions and I am 
not aware of a GNU style guideline that would discourage their use. 
Plus, some uses of apostrophes in messages are for possessives, not 
contractions. Surely we should not ban contractions and possessives 
merely because our API has glitches with apostrophes.

bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Andreas Schwab]

On Jun 06 2017, Paul Eggert <eggert@cs.ucla.edu> wrote:

> The original bug report is about the ‘message’ function, not about formats
> in general. For that function, "%s" is the longstanding way to output
> arbitrary strings. For formats in general, one can use the ‘format’
> function (which does not translate quotes) or the ‘format-message’
> function (which does); this suffices in practice.

That doesn't make sense, since the first argument of message is a format
string.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

RE: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Drew Adams]

Could you guys please possibly pick only one mailing list,
bugs or emacs-devel?

bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Alan Mackenzie]

Hello, Paul.

On Mon, Jun 05, 2017 at 17:14:37 -0700, Paul Eggert wrote:
>  >> Elisp code needed to use
>  >> (message "%s" STR) even before the change you're objecting to,

>  > Did it?  When and why?

> Yes, because one can’t pass arbitrary strings to the message function 
> and expect them to be displayed as-is.

This has absolutely nothing to do with the current bug, and I'm sorry if
I've discussed the point as though it did.  I'm not suggesting arbitrary
strings should be messaged without a "%s" format string.

What I am arguing against is being required to use _two_ format strings
in one message invocation, as (message "%s" (format "..." ...)).  This
just screams out "not thought through" and is unacceptable.

>  > Somebody using message to output Lisp will use ' just as I did - and 
>  > suffer the same horrendous problems

> Sure, like the “horrendous” problems with %.

That's pure sophistry.  _Nobody_, excepting the least bright of your
students (I'm assuming here that it's not the teaching which is at
fault) will have the slightest difficulty with % in message, because it
stands out visually.  Every format construct (up to Emacs 24) started
off with %, and that simplicity and symmetry has now been destroyed.  I
propose to restore it.

By contrast, the current ` and ' are disguised format constructs which
look like plain characters.  They should become %` and %' to make them
explicit and compatible with all the other format constructs.

> For example, in Emacs 24 your example, when used with data involving
> %:

[ .... ]

That example from CC Mode, which gave rise to this bug report, did not
have and could not have had % signs.  If the reverse had been the case,
they would have been carefully and correctly coded before even trying
the thing out, because I, like everybody else here, know that %
introduces format constructs.

>  > Do you, perhaps, have another strategem for preventing this problem?

> Sure: don’t pass arbitrary strings to the message function.

In other words, just ignore the problem, and have it hit as many people
as it will, without giving them any help.  That's not very nice of you.

>  > How do you propose to prevent such puzzlement and anger in the future

> Not by this:

>  >>           (error "Can't find `%s' in %s" thing where)))
>  >>   =>      (error "Can%'t find %`%s%' in %s" thing where)))

> For Emacs code this would likely be a cure worse than the disease,
> ....

How so?  It would make the format constructs explicit, giving maximum
control to the hacker.  Or do you want to make all the curvy quote stuff
as sly and stealthy as possible, so that as few hackers as possible will
be aware of it?

> .... by causing more puzzlement and anger than it would prevent.

How could it cause puzzlement?  It's mnemonic and consistent with the
other format constructs.

> It would make formats significantly harder to read.

But only slightly.  You seem to be saying that the formats need to be
dumbed down for hackers to understand them.  I'm sure you're not right,
here.

> And as Clément mentioned, it would introduce compatibility problems of
> its own.

Rubbish!  What Clément pointed out is that the part of styled_format
which counts format constructs and argumnts would need to be modified.
This would be trivial.

> There is a better way if the primary goal is to avoid quote translation:

>             (error "Can't find `%s' in %s" thing where)))
>     =>      (error "Can’t find ‘%s’ in %s" thing where)))

> Compared to %` and %', this is simpler, easier to read, and more 
> compatible with current and older Emacs versions.

Except those characters don't appear on non-Finnish keyboards, hence are
difficult to type, and they don't display nicely on all supported
environments.  These things make that suggestion a non-starter for
current purposes.

> A downside, though, is that it would involve changing hundreds or
> thousands of strings in the Emacs source (just as %` and %' would).

There's already a volunteer to do this work, so that's not really a
downside at all.

>  > You're not seriously
>  > telling me that any of your students who've written a message call with
>  > a "%s" in the format string remain unaware of the role of %, are you?

> Sure, they learn about % after the message function doesn’t work the way 
> they naively expected. In that respect, % is like ` and '.

Only in the same sense that Emacs is like Microsoft's notepad, in that
they're both text editors.

Only the rawest of beginners will be confused by a % in a format string.
That is not the level of experience we expect of our target users.

>  > There are around 275 calls to message which have a non-literal
>  > format argument.

> Each one stands for possibly many other calls, and we don’t know how 
> many of these other calls might cause a problem.

No, but the number will be small enough for each instance to be dealt
with individually.

>  > The consequences of surreptitious unwanted translation ...
>  >> It's not surreptitious: it's documented.

It may be documented, but it's still surreptitious.  It seems intended
by its writer to be as hidden as possible, so as to make those using it
as unaware as possible of the consequences of its use.  If you don't
like %` and %', perhaps you could suggest some other way of preventing
the nature of ` and ' in format strings from being so obscure.

>  > And this documentation is useless for preventing the problems.

> True, documentation by itself does not prevent programming problems. 
> However, this doesn’t change the fact that quote translation is 
> documented. It is not “surreptitious” or “implicit” or “vague” or 
> “stealthy” or “fuzzy”.

It is most definitely implicit, in that there is no indication in the
format string that quote translation happens, it is stealthy, in that it
goes out of its way to hide its actions.  It is vague, in that a hacker
who sort of knows message will be permanently unsure which characters in
a format string are not literal.

This documentation, no matter how good it might be made, will be
insufficient to prevent hackers falling into the trap which the code
sets.  This part of Emacs is badly thought out for this reason: it's an
accident waiting to happen.  I am proposing to fix it, and have
volunteered to do the work.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Clément Pit-Claudel]

On 2017-06-07 15:13, Alan Mackenzie wrote:
> Rubbish!  What Clément pointed out is that the part of styled_format
> which counts format constructs and argumnts would need to be modified.
> This would be trivial.

Uh? What I pointed out was that new code wouldn't work with old Emacsen.

Maybe I misunderstood your proposal?  As far as I can tell, the change you propose will force me to write every `message' call like this in my packages:
 
(if (< emacs-version 26)
    (message "Hello, `world'!")
  (message "Hello, %`world%'!"))

Am I missing something?

Clément.

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Dmitry Gutov]

On 6/7/17 10:27 PM, Clément Pit-Claudel wrote:

> Uh? What I pointed out was that new code wouldn't work with old Emacsen.

That's a valid concern, and the only tangible downside of introducing %` 
and %' that I'm aware of.

> Maybe I misunderstood your proposal?  As far as I can tell, the change you propose will force me to write every `message' call like this in my packages:
>   
> (if (< emacs-version 26)
>      (message "Hello, `world'!")
>    (message "Hello, %`world%'!"))
> 
> Am I missing something?

Or wait until you only need to support Emacs 26 and newer. Not a huge 
loss, IMO. That time will come.

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Alan Mackenzie]

Hello, Clément.

On Wed, Jun 07, 2017 at 15:27:02 -0400, Clément Pit-Claudel wrote:
> On 2017-06-07 15:13, Alan Mackenzie wrote:
> > Rubbish!  What Clément pointed out is that the part of styled_format
> > which counts format constructs and argumnts would need to be modified.
> > This would be trivial.

> Uh? What I pointed out was that new code wouldn't work with old Emacsen.

> Maybe I misunderstood your proposal?  As far as I can tell, the change you propose will force me to write every `message' call like this in my packages:
 
> (if (< emacs-version 26)
>     (message "Hello, `world'!")
>   (message "Hello, %`world%'!"))

> Am I missing something?

Apologies, I misunderstood you completely.

I also missed the point you are making.  I'll have to think about it.

Thanks!

> Clément.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Alan Mackenzie]

Hello again, Clément.

On Wed, Jun 07, 2017 at 15:27:02 -0400, Clément Pit-Claudel wrote:
> On 2017-06-07 15:13, Alan Mackenzie wrote:
> > Rubbish!  What Clément pointed out is that the part of styled_format
> > which counts format constructs and argumnts would need to be modified.
> > This would be trivial.

> Uh? What I pointed out was that new code wouldn't work with old Emacsen.

> Maybe I misunderstood your proposal?  As far as I can tell, the change you propose will force me to write every `message' call like this in my packages:
 
> (if (< emacs-version 26)
>     (message "Hello, `world'!")
>   (message "Hello, %`world%'!"))

That would clearly be unacceptable.

> Am I missing something?

No, I don't think so.  But it is probably feasible to put advice on
`message' in older Emacsen.  This advice would strip the % from %` and
%'.

It should be borne in mind that the handling of ` and ' by `message' in
Emacs-25 is in any case incompatible with earlier versions of Emacs.

> Clément.

-- 
Alan Mackenzie (Nuremberg, Germany).

RE: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Drew Adams]

(Removed the bug list from CC.  The discussion does not now
seem to be only about a specific bug.  But if it is, then it
belongs only on the bug list and not also on emacs-devel.) 

> > the change you propose will force me to write every `message'
> > call like this in my packages:
> 
> > (if (< emacs-version 26)
> >     (message "Hello, `world'!")
> >   (message "Hello, %`world%'!"))
> 
> That would clearly be unacceptable.

Truly unacceptable.  There are probably thousands of `message'
calls like the first one.  (My code alone might have thousands
of them. ;-))

There should be no reason to have to do such a thing.

It is equally unacceptable, IMO, if Emacs does not respect the
chars ` and ' as they are, but stupidly substitutes curly quotes
for them.  This is true for `message' but for so much more than
just `message'.

Someone should really consider putting a stop to this and reverse
course.  This problem exists since Emacs 25.1 (Emacs 24.5 is the
last vestige of sanity in this regard).

It REALLY does not make any sense that you write
(message "Hello, `world'!") and you get Hello, ‘world’!.

I have a hard time believing that this virus got past the
maintainers.  (But this is not the first time I've had a hard
time believing that some changes have actually been made.)

> No, I don't think so.  But it is probably feasible to put advice on
> `message' in older Emacsen.  This advice would strip the % from %`
> and %'.
> 
> It should be borne in mind that the handling of ` and ' by `message' in
> Emacs-25 is in any case incompatible with earlier versions of Emacs.

That's the origin of the currently discussed problem.  And it is
a real problem in its own right.

`message' with quotes [was: bug#23425: `message' wrongly corrupts ' to curly quote]

[Drew Adams]

`text-quoting-style' controls this (at least in Emacs 25).
So at least you can bind that to `grave' around any code
that uses `message', to get back the sane, pre-Emacs 25
behavior.

Given that, is there any reason for Alan's proposed new
format escapes to to tell Emacs: "Hands off my quotes!"?

Of course, you should not HAVE to do that.  That should
be the _default_ behavior: no crazy quote translation.

`electric-quote-mode' is DISabled by default.  We got one
thing right!

Why is it that the crazy quote translation for help and
`message' happens by default, and you have to jump through
a nearly secret hoop to get back the sane behavior?

That makes no sense.  It's an incompatible change that
affects all older code as well.

The default for `text-quoting-style' should be the
misleadingly named `grave' behavior, i.e., preserve the
long-established behavior of just leaving ` and ' alone.

Instead, it is a value that gives you curly quotes when
available.  That behavior should not be the default (and
it really should have a value such as `prefer-curly', not
`nil').

Then, anyone wanting to be adventurous or what s?he
considers "modern" could always bind it to `prefer-curly'
or whatever, to get today's misguided default behavior,
or to `curve' to always get translation-to-curly.

This one we got backwards.

It's not too late to change the default behavior of
`text-quoting-style'.  (But it is probably too late
to change the meaning of `nil' for it.)  Just make
the initial value `grave', not `nil'.

It's also a mistake to (in effect) hide this variable
from ordinary users, instead of making it easy for them
to express their preferences in this regard.  This is
what the Elisp manual says about it (it is not in the
Emacs manual at all):

  This variable can be used by experts on platforms that
  have problems with curved quotes.  As it is not intended
  for casual use, it is not a user option.

That couldn't be more misguided.  Experts?  Only platforms
that have "problems with curved quotes"?  Not for casual
use?  It's a self-styled "style" variable, but it's not for
you or me.  Sheesh.  Why?  What happened to the interests
of individual users?

Re: `message' with quotes [was: bug#23425: `message' wrongly corrupts ' to curly quote]

[Alan Mackenzie]

Hello, Drew.

On Wed, Jun 07, 2017 at 15:05:51 -0700, Drew Adams wrote:
> `text-quoting-style' controls this (at least in Emacs 25).
> So at least you can bind that to `grave' around any code
> that uses `message', to get back the sane, pre-Emacs 25
> behavior.

> Given that, is there any reason for Alan's proposed new
> format escapes to to tell Emacs: "Hands off my quotes!"?

Being accurate, these sequences tell Emacs "please translate to the
user's preferred version of quotes".

> Of course, you should not HAVE to do that.  That should
> be the _default_ behavior: no crazy quote translation.

Indeed.

> `electric-quote-mode' is DISabled by default.  We got one
> thing right!

> Why is it that the crazy quote translation for help and
> `message' happens by default, and you have to jump through
> a nearly secret hoop to get back the sane behavior?

> That makes no sense.  It's an incompatible change that
> affects all older code as well.

> The default for `text-quoting-style' should be the
> misleadingly named `grave' behavior, i.e., preserve the
> long-established behavior of just leaving ` and ' alone.

Yes.  It should also leave literal curly quotes untouched (I believe it
does this).

> Instead, it is a value that gives you curly quotes when
> available.  That behavior should not be the default (and
> it really should have a value such as `prefer-curly', not
> `nil').

> Then, anyone wanting to be adventurous or what s?he
> considers "modern" could always bind it to `prefer-curly'
> or whatever, to get today's misguided default behavior,
> or to `curve' to always get translation-to-curly.

> This one we got backwards.

> It's not too late to change the default behavior of
> `text-quoting-style'.  (But it is probably too late
> to change the meaning of `nil' for it.)  Just make
> the initial value `grave', not `nil'.

> It's also a mistake to (in effect) hide this variable
> from ordinary users, instead of making it easy for them
> to express their preferences in this regard.

This is a thing which reinforces the impression that curly quotes were
foisted surreptitiously on users.  The normal Emacs way is to make such
things configurable in a customisable option.

> This is what the Elisp manual says about it (it is not in the Emacs
> manual at all):

>   This variable can be used by experts on platforms that
>   have problems with curved quotes.  As it is not intended
>   for casual use, it is not a user option.

Yes.  How about _users_ who have problems with curved quotes?

> That couldn't be more misguided.  Experts?

I don't see any need for particular expertise to use this variable.

> Only platforms that have "problems with curved quotes"?  Not for casual
> use?  It's a self-styled "style" variable, but it's not for you or me.
> Sheesh.  Why?  What happened to the interests of individual users?

Indeed.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Alan Mackenzie]

[ Posted to emacs-devel@gnu.org.  Reply-To: set there too. ]

On Fri, Jun 02, 2017 at 20:13:22 -0400, Glenn Morris wrote:

> > We had that discussion there already, and I doubt whether it'd be
> > worthwhile to repeat it. Of course you're free to take it there again
> > if you like.

The discussion was abruptly suspended with "we're too close to a
release", without any conclusion having been reached.

`message' and friends are in a thoroughly unsatisfactory state, with
wild ad-hoc rules, and obstruse formulations needed as workarounds.

> I think this is a situation where the maintainer(s) should simply
> announce a decision, so that we can stop going over the same ground
> again and again. (Personally I think a year and two releases means the
> decision is already made.)

The decision was made in the near infinite number of releases prior to
Emacs 25 to have a rational and clear system of special characters
beginning with %.  Quote translation, which was introduced into Emacs
without a consensus amongst developers has destroyed that rationality
and simplicity.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Alan Mackenzie]

[ Posted to emacs-devel@gnu.org, Reply-To: set there, too. ]

Hello, Dmitry.

On Sat, Jun 03, 2017 at 00:15:49 +0300, Dmitry Gutov wrote:
> On 6/3/17 12:02 AM, Alan Mackenzie wrote:

> > I propose restoring the default to what it was in Emacs 24.  For
> > simplicity's sake, only % should be special, and for translating quotes
> > we could introduce %' and %`, such that quote translation happens only
> > when a %' or %` is present.

> I recall proposing something to that effect back in one of the original 
> discussions. E.g. GCC does that with its message function.

Can you see any disadvantage with such a change?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Dmitry Gutov]

On 6/4/17 12:10 AM, Alan Mackenzie wrote:

>> I recall proposing something to that effect back in one of the original
>> discussions. E.g. GCC does that with its message function.
> 
> Can you see any disadvantage with such a change?

IIRC, it was waved away with complaints of verbosity.

Re: bug#23425: master branch: `message' wrongly corrupts ' to curly quote.

[Alan Mackenzie]

Hello, Glenn.

On Thu, Jun 01, 2017 at 12:56:05 -0400, Glenn Morris wrote:

> So in your view what needs to happen for this bug to be closed?
> As was explained a year ago, change your initial example to (message
> "%s" ...) and your issue goes away.

No it does not.  The issue is that this is a trap waiting to spring and
bite hackers.  Also the workaround you identify is ugly and irregular -
it would involve writing two format strings (one of them being the "%s")
to do one `message' invocation.

I have identified a regular and rational way to repair the damage,
namely to introduce the new %-constructs %` and %', which would invoke
quote translation, and would remain compatible with `message' as it was
before Emacs-25.

-- 
Alan Mackenzie (Nuremberg, Germany).