A simple solution to "Upcoming loss of usability ..."

A simple solution to "Upcoming loss of usability ..."

[Oleh Krehel]

Hi all,

It seems that even though many conservative people disagree, the
"experiment" isn't going away.

So I'm proposing the following change that could satisfy both Paul and
the conservatives (myself included):

(font-lock-add-keywords
 'emacs-lisp-mode
 '(("\\(`\\)\\([a-zA-Z-0-9]+\\)\\('\\)"
    (0
     (compose-region
      (match-beginning 1)
      (match-end 1)
      "‘"))
    (0
     (compose-region
      (match-beginning 3)
      (match-end 3)
      "’")))))

The above code can also be made part of `prettify-symbols-mode', and
some customization can be added to turn things on or off. The above code
is only a rough draft, it may not be optimal, which the experts could
quickly fix. But it works for all Emacs versions installed on my system.

The advantages of the above approach:

1. Paul's students get to see the pretty quotes.

2. The Emacs sources need not be changed, no new syntax definitions need
to be introduced.

3. The pretty quotes are easy to input for Paul's students: they're
right there on the standard issue keyboard, available with zero modifier
keys: even the Shift modifier isn't necessary.

4. Zero problems in the terminal and with copy pasting: that part of
`prettify-symbols-mode' could be just disabled for terminals that don't
support Unicode. In addition, I can actually see the Unicode quotes fine
in GNOME Terminal. They look garbled on a TTY, but again: the minor mode
can be enabled/disabled conditionally.

5. The quotes stay what they are: markup, as they were intended to be.
The markup can be rendered in many different ways, including as Unicode
quotes, if the user prefers.

regards,
Oleh

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/25/2015 05:59 PM, Oleh Krehel wrote:

> It seems that even though many conservative people disagree, the
> "experiment" isn't going away.
>
> So I'm proposing the following change that could satisfy both Paul and
> the conservatives (myself included):

I'm not sure it'll satisfy Paul, because he seems intent on introducing 
actual curly quotes in the Elisp source code. Myself, I'd very much 
prefer this approach.

> (font-lock-add-keywords
>   'emacs-lisp-mode
>   '(("\\(`\\)\\([a-zA-Z-0-9]+\\)\\('\\)"
>      (0
>       (compose-region
>        (match-beginning 1)
>        (match-end 1)
>        "‘"))
>      (0
>       (compose-region
>        (match-beginning 3)
>        (match-end 3)
>        "’")))))

Otherwise, this looks pretty close to the proposed font-lock-only 
solution in 
http://lists.gnu.org/archive/html/emacs-devel/2015-06/msg00547.html

And yes, we can add similar font-lock rules in emacs-lisp-mode buffers, 
but they'll have to handle escaped quotes a bit differently (do not hide 
the escaping chars).

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Oleh Krehel wrote:
> (font-lock-add-keywords
>   'emacs-lisp-mode
>   '(("\\(`\\)\\([a-zA-Z-0-9]+\\)\\('\\)"

The proposed approach would mishandle many cases where the things being quoted 
are not typical Lisp identifiers.  E.g.:

"Press ‘h’ for complete help; press ‘?’ repeatedly for a summary"
"Make ‘funcall/apply’ form to map SOURCE-ARGLIST to TARGET-ARGLIST...."
"... Example: ‘(ad-map-arglists '(a &rest args) '(w x y z))’ will return ..."

Also, the proposed approach won't easily generalize to diagnostics, which often 
quote non-identifiers like ‘%s’.  There's also a UI problem: ot would cause 
action-at-a-distance, because typing an apostrophe in one place in the buffer 
would visually alter a part of the line many characters away. 
(Action-at-a-distance is not a fatal objection, but it is better to avoid it 
when possible.)

Most of the advantages you mention for the proposed approach are also advantages 
of the approach in master.  With the current approach, the Emacs sources don't 
need to be changed, quotes are just as easy to input (in Electric Quote mode), 
terminal and copy-pasting work, and quotes are markup.

The main advantage of the proposed approach over the current master is that the 
source code often can still contain grave accent and apostrophe unmodified, even 
though people reading and editing the source code will see curved quotes.  To my 
mind this is more a recipe for confusion than anything else -- at least, I 
wouldn't want to inflict it on Emacs newcomers.

Re: A simple solution to "Upcoming loss of usability ..."

[Oleh Krehel]

Paul Eggert <eggert@cs.ucla.edu> writes:

> With the current approach, the Emacs sources don't need to be changed

My sole objection was that they were changed. And I thought that was the
whole point of it: that you want the markup to be its own visual
representation.

> "Make ‘funcall/apply’ form to map SOURCE-ARGLIST to TARGET-ARGLIST...."

This is already in the Emacs sources, so they were indeed changed.

> "Press ‘h’ for complete help; press ‘?’ repeatedly for a summary"

There should be a separate markup for quoting key bindings in my
opinion. For example, Markdown has <kbd>h</kbd>, and I use ~h~ in my
org-mode documents intended for HTML export.

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Oleh Krehel wrote:
> Paul Eggert<eggert@cs.ucla.edu>  writes:
>
>> >With the current approach, the Emacs sources don't need to be changed
> My sole objection was that they were changed. And I thought that was the
> whole point of it: that you want the markup to be its own visual
> representation.

Yes, I would like the markup to head that way.  But the Emacs source docstrings 
have largely not needed to change, and there's no need to change them right now.

>> >"Make ‘funcall/apply’ form to map SOURCE-ARGLIST to TARGET-ARGLIST...."
> This is already in the Emacs sources, so they were indeed changed.

I gave that example because I knew about it already and had already changed it 
for other reasons.  Most such examples haven't changed recently.  Here are some:

\(1) a local variables list or the `-*-' line specifies a major mode, or
If MARK is nil, then the default character `?r' is used.
the working buffer, `*1' the value in the process buffer.
same as in `newsticker--parse-atom-1.0'.
auto-newline minor mode is activated (as evidenced by a `/a' or `/ah'
Call `reftex-citation' with a format selector `?p'.

The current highlighting heuristics work reasonably well with these docstrings, 
because we want to highlight only quoted symbols, and we don't want to highlight 
the ‘-*-’ or the ‘?r' or etc.  But if we want to use font-lock to alter how 
quotes are displayed, these heuristics won't work well, as they'll fail to alter 
many quotes that they should alter.

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/25/2015 11:48 PM, Paul Eggert wrote:

> Yes, I would like the markup to head that way.  But the Emacs source
> docstrings have largely not needed to change, and there's no need to
> change them right now.

That's sneaky wording. You obviously want them to change.

The majority of people opposing don't want the curly quotes in 
docstrings, nor do they want to have two standards at the same time (now 
and then effectively forever).

Of course, the question of whether to use curly quotes in the docstrings 
or render them later is orthogonal enough to the choice of using 
font-lock or substitute-command-keys, but I think I gave a few arguments 
in favor of the former.

> But if we want to
> use font-lock to alter how quotes are displayed, these heuristics won't
> work well, as they'll fail to alter many quotes that they should alter.

Why do you think so? We can alter any quotes we want using font-lock 
too. No real need to alter them in pairs either, if we want to disregard 
Richard's advice, and we're fine with using escaping more often.

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:

>> But if we want to
>> use font-lock to alter how quotes are displayed, these heuristics won't
>> work well, as they'll fail to alter many quotes that they should alter.
>
> Why do you think so? We can alter any quotes we want using font-lock too.

Whatever heuristics we use are likely to be wrong for many real-world uses in 
Elisp code.  Many uses of grave accent in Elisp strings, characters, and 
comments are intended to be grave accent, not left quote.  Similarly for 
apostrophe and right quote.  It's unlikely that any easy-to-explain heuristic 
based on font-lock will distinguish among the uses reliably.

 >> Yes, I would like the markup to head that way.  But the Emacs source
 >> docstrings have largely not needed to change, and there's no need to
 >> change them right now.
 >
 > That's sneaky wording. You obviously want them to change.

There was no intent to be sneaky.  I don't see how one can interpret my remarks 
in any way other than saying that I would like the markup to change eventually.

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/26/2015 01:15 AM, Paul Eggert wrote:

> Whatever heuristics we use are likely to be wrong for many real-world
> uses in Elisp code.

font-lock rules don't have to be about heuristics. The conversion can be 
as straight as we want it to me.

> Many uses of grave accent in Elisp strings,
> characters, and comments are intended to be grave accent, not left
> quote.  Similarly for apostrophe and right quote.  It's unlikely that
> any easy-to-explain heuristic based on font-lock will distinguish among
> the uses reliably.

It has been established already that we need an escaping syntax, too.

> There was no intent to be sneaky.  I don't see how one can interpret my
> remarks in any way other than saying that I would like the markup to
> change eventually.

In the context of people arguing against the change, stating that "the 
Emacs source docstrings have largely not needed to change, and there's 
no need to change them right now" is, at best, irrelevant.

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:
>> Whatever heuristics we use are likely to be wrong for many real-world
>> uses in Elisp code.
>
> font-lock rules don't have to be about heuristics. The conversion can be as
> straight as we want it to me.

No straightforward conversion will work, I'm afraid.

>> Many uses of grave accent in Elisp strings,
>> characters, and comments are intended to be grave accent, not left
>> quote.  Similarly for apostrophe and right quote.  It's unlikely that
>> any easy-to-explain heuristic based on font-lock will distinguish among
>> the uses reliably.
>
> It has been established already that we need an escaping syntax, too.

Sorry, I'm not following.  This would be a new escape syntax for Elisp strings? 
  How would it work?  For example, would \` and \' have different meaning in 
strings than they do in character literals?  When exactly would these escapes be 
transformed to user-preferred quote characters?  Although I have already 
explored solutions along these lines and none of them worked well, perhaps you 
have ideas that will improve them.

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/26/2015 01:41 AM, Paul Eggert wrote:

> No straightforward conversion will work, I'm afraid.

The logic can follow along the lines of the new parts in 
substitute-command-keys.

>> It has been established already that we need an escaping syntax, too.
>
> Sorry, I'm not following.  This would be a new escape syntax for Elisp
> strings? How would it work?

In short, via re-search-foward, followed by compose-region.

> For example, would \` and \' have
> different meaning in strings than they do in character literals?

No. That would be nice, but more or less impossible to implement.

I think we could use "\\", though. Or else, something like "\\=", 
although that exact sequence is already taken (substitute-command-keys 
would eat it).

> When
> exactly would these escapes be transformed to user-preferred quote
> characters?

During buffer fontification.

Re: A simple solution to "Upcoming loss of usability ..."

[raman]

1+. Changing the source code as proposed is a slippery slope and we'll
end up over time (shudder shudder) with who knows, ay be snippets of
HTML markup and other gunk.


-- 

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/25/2015 07:36 PM, Paul Eggert wrote:

> The proposed approach would mishandle many cases where the things being
> quoted are not typical Lisp identifiers.  E.g.:
>
> "Press ‘h’ for complete help; press ‘?’ repeatedly for a summary"
> "Make ‘funcall/apply’ form to map SOURCE-ARGLIST to TARGET-ARGLIST...."
> "... Example: ‘(ad-map-arglists '(a &rest args) '(w x y z))’ will return
> ..."

We already have a different regexp that will handle those. It's not an 
insurmountable problem either way.

> Also, the proposed approach won't easily generalize to diagnostics,
> which often quote non-identifiers like ‘%s’.

Same here.

> There's also a UI problem:
> ot would cause action-at-a-distance, because typing an apostrophe in one
> place in the buffer would visually alter a part of the line many
> characters away.

Just like typing a double-quote will make Emacs consider the rest of the 
buffer a string literal? Not that big a problem.

> Most of the advantages you mention for the proposed approach are also
> advantages of the approach in master.  With the current approach, the
> Emacs sources don't need to be changed,

Because you already changed them?

You'd still have to propagate those changes (which a sizable number of 
people expressed dislike for), to all third-party Elisp code out there.

> The main advantage of the proposed approach over the current master is
> that the source code often can still contain grave accent and apostrophe
> unmodified, even though people reading and editing the source code will
> see curved quotes.  To my mind this is more a recipe for confusion than
> anything else -- at least, I wouldn't want to inflict it on Emacs
> newcomers.

We can keep it disabled by default, if you like. Unlike the 
substitute-command-keys approach, font-lock is flexible that way.

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:
>> "Press ‘h’ for complete help; press ‘?’ repeatedly for a summary"
>> "Make ‘funcall/apply’ form to map SOURCE-ARGLIST to TARGET-ARGLIST...."
>> "... Example: ‘(ad-map-arglists '(a &rest args) '(w x y z))’ will return
>> ..."
>
> We already have a different regexp that will handle those. It's not an
> insurmountable problem either way.

Perhaps some complex regular expression would work most of the time.  We'd have 
to see.  However, its complexity will make its behavior hard to document, and it 
will likely lead to counterintuitive display of source code on occasion.  It's a 
minor thing to get a color wrong; it's a bigger deal to display the wrong 
character.  The resulting problems are likely not to be worth the aggravation.

>> There's also a UI problem:
>> ot would cause action-at-a-distance, because typing an apostrophe in one
>> place in the buffer would visually alter a part of the line many
>> characters away.
>
> Just like typing a double-quote will make Emacs consider the rest of the buffer
> a string literal? Not that big a problem.

As I said, action-at-a-distance is not a fatal objection.  But yes, that 
behavior of double-quote is objectionable, and I wish that Emacs didn't do it. 
I hope we don't need to add more glitches like it.

> You'd still have to propagate those changes (which a sizable number of people
> expressed dislike for), to all third-party Elisp code out there.

Third-party code doesn't need to change.  It'll still work reasonably well.

>> The main advantage of the proposed approach over the current master is
>> that the source code often can still contain grave accent and apostrophe
>> unmodified, even though people reading and editing the source code will
>> see curved quotes.  To my mind this is more a recipe for confusion than
>> anything else -- at least, I wouldn't want to inflict it on Emacs
>> newcomers.
>
> We can keep it disabled by default, if you like. Unlike the
> substitute-command-keys approach, font-lock is flexible that way.

Sorry, I don't understand this remark.  Both approaches allow for defaults, and 
for behavior to be disabled by default, so neither has an advantage in that respect.

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/26/2015 01:17 AM, Paul Eggert wrote:

> Perhaps some complex regular expression would work most of the time.
> We'd have to see.  However, its complexity will make its behavior hard
> to document, and it will likely lead to counterintuitive display of
> source code on occasion.  It's a minor thing to get a color wrong; it's
> a bigger deal to display the wrong character.  The resulting problems
> are likely not to be worth the aggravation.

This is just hand-waving. I gave you a patch, feel free to criticize it, 
ask to support different cases, etc.

You can display a wrong character with sustitute-command-keys just as 
well. Like in its own docstring currently ("Each ‘ and ’ are replaced 
by...").

> As I said, action-at-a-distance is not a fatal objection.  But yes, that
> behavior of double-quote is objectionable, and I wish that Emacs didn't
> do it.

I don't see a way how Emacs would be able not do it.

> Third-party code doesn't need to change.  It'll still work reasonably well.

Then we'll be in the "double standard" area.

>> We can keep it disabled by default, if you like. Unlike the
>> substitute-command-keys approach, font-lock is flexible that way.
>
> Sorry, I don't understand this remark.  Both approaches allow for
> defaults, and for behavior to be disabled by default, so neither has an
> advantage in that respect.

With substitute-command-keys, you can't control the way the characters 
look in the source buffer (which Oleh's patch was for). Only how they 
look in the Help buffer.

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:

> This is just hand-waving. I gave you a patch, feel free to criticize it, ask to
> support different cases, etc.

I think the most recent patch you sent is in 
<http://lists.gnu.org/archive/html/emacs-devel/2015-06/msg00547.html>.  I just 
now tried it against the current master (commit 
99ad90dcb1beccde926d9b6475a393c6f8743f5c), and didn't notice any difference in 
display.  I suppose you intended the patch to be combined with reverting some 
recent changes to the master, but I don't know which changes those would be.

Also, as I understand it this part of the thread was about display of Elisp 
source code, whereas that patch seems to about display of *Help* buffers.

> You can display a wrong character with sustitute-command-keys just as well.

Yes, of course.  But in practice the effect of substitute-command-keys is 
limited to strings displayed in *Help* buffers.  This makes its misbehavior less 
likely (and when it happens, less of a problem) than a font-lock approach 
designed to display arbitrary Elisp source code.

> Like in its own docstring currently ("Each ‘ and ’ are replaced by...").

I don't see any wrong character there.  That part of the documentation is about 
curved single quotes, and that's what I see.

> With substitute-command-keys, you can't control the way the characters look
 > in the source buffer (which Oleh's patch was for)

OK, in that case I agree that the font-lock approach should give more control on 
how source code is displayed, whereas substitute-command-keys does not affect 
source-code display.  I still don't see how font-lock would work with 
source-code strings containing quotes, though; I expect it will mishandle 
display in a significant number of practical cases.  (Again, this appears to be 
a different topic than the abovementioned patch, so it's not clear what's being 
proposed here.)

>> No straightforward conversion will work, I'm afraid.
>
> The logic can follow along the lines of the new parts in substitute-command-keys.

That would be plausible if we were talking about *Help* buffer contents, 
although there's still the matter of dealing with user string contents (which 
the current master handles but the abovementioned patch seemingly would 
mishandle).  But this part of the thread was about arbitrary Elisp source-code 
strings, and the substitute-command-keys heuristics won't work there.

> I think we could use "\\", though. Or else, something like "\\=", although that
> exact sequence is already taken (substitute-command-keys would eat it).

I'm afraid I'm lost now.  I don't know what you're proposing here.  Your other 
comments suggest you're talking about escapes that can appear in any Elisp 
source-code string, but I don't know what the escapes look like or what they 
would do.

> font-lock runs in the help buffers, as well as info (if I'm not mistaken), so that approach can be used in those too.

This makes it sounds like you want one font-lock solution for help buffers, 
another font-lock solution for info buffers, and a third font-lock solution for 
Elisp source-code files.  Is that the idea?  But I'm having trouble seeing what 
the three would have in common or how their combination would be documented. 
For example, for a typical user what would font-lock do for info files that is 
not already being done?

>> OK, that can be the format-message function already discussed.
>
> Indeed.

I'll look into implementing that then, as this issue seems separable from the 
others.

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/26/2015 05:35 AM, Paul Eggert wrote:

> I think the most recent patch you sent is in
> <http://lists.gnu.org/archive/html/emacs-devel/2015-06/msg00547.html>.
> I just now tried it against the current master (commit
> 99ad90dcb1beccde926d9b6475a393c6f8743f5c), and didn't notice any
> difference in display.  I suppose you intended the patch to be combined
> with reverting some recent changes to the master, but I don't know which
> changes those would be.

Like mentioned in the preceding email, that patch is against f743819.

But if you were looking for changes to revert, those would be the uses 
of curly quotes as markup in the source code, the new 
`substitute-command-keys' calls, as well as the code in 
`substitute-command-keys' that performs quote replacement.

The "differences is display" is the wrong thing to look for, though, 
because the intent is to reproduce the current behavior, more or less.

> Also, as I understand it this part of the thread was about display of
> Elisp source code, whereas that patch seems to about display of *Help*
> buffers.

The patches can be very similar. In the source code, we'll not hide the 
escape sequences; that should be the main difference.

> Yes, of course.  But in practice the effect of substitute-command-keys
> is limited to strings displayed in *Help* buffers.  This makes its
> misbehavior less likely (and when it happens, less of a problem) than a
> font-lock approach designed to display arbitrary Elisp source code.

It will be very easy to limit the conversion to only within strings.

> I don't see any wrong character there.  That part of the documentation
> is about curved single quotes, and that's what I see.

Hmm, okay. Guess I misread it.

> (Again, this appears to be a different topic than the abovementioned
> patch, so it's not clear what's being proposed here.)

Why don't we start with the help-mode patch (maybe even in the thread 
it's been posted to), and then return to the source buffers later.

The issue of curly quotes in Help and Info buffers seems to be the more 
important one.

> That would be plausible if we were talking about *Help* buffer contents,
> although there's still the matter of dealing with user string contents
> (which the current master handles but the abovementioned patch seemingly
> would mishandle).  But this part of the thread was about arbitrary Elisp
> source-code strings, and the substitute-command-keys heuristics won't
> work there.

I think the only heuristic we won't be able to reproduce is the fact 
that one can intentionally pass only some strings to 
`substitute-command-keys', but not the others.

Depending on the number of false positives created this way, we may or 
may not decide to turn the quote translation in the source buffer off by 
default.

Or else, decide we prefer to have false negatives and only translate 
quotes in strings that are in known docstring positions (like within a 
top-level `defun' form, after the arguments).

> I'm afraid I'm lost now.  I don't know what you're proposing here.  Your
> other comments suggest you're talking about escapes that can appear in
> any Elisp source-code string, but I don't know what the escapes look
> like or what they would do.

"`'" will render as "‘’" in the Help buffer and the source buffer (if 
the quote translation is turned on).

"\\`\\'" will render as "`'" in the Help buffer and "\\`\\'" in the 
source buffer.

Looking at it now, these would generate quite a few false positives in 
regexps (if we don't eliminate this possibility in a different way). So 
we could use "\\~", for example ("\\~`" -> "`").

Or else, make the quote characters escape themselves.

Then, "``" would render as "`" in the Help buffer, and "''" as "'".

"```'''" would translate to "`‘'’", though.

> This makes it sounds like you want one font-lock solution for help
> buffers, another font-lock solution for info buffers, and a third
> font-lock solution for Elisp source-code files.  Is that the idea?

They Help and Elisp buffers can share the conversion function (with one 
or two modifier arguments), as well as the search regexp. The font-lock 
rule would have to be set in several places, that is correct.

> But
> I'm having trouble seeing what the three would have in common or how
> their combination would be documented.

What will need to be documented is the quoting syntax, as well as the 
escaping rules.

> For example, for a typical user
> what would font-lock do for info files that is not already being done?

Is Info actually relevant in this discussion? Looking at info.el, it 
seems to deal files where quotes have already been translated 
(Info-mode-font-lock-keywords indicates that), so maybe neither 
substitute-command-keys, nor the new font-lock rules, have to do 
anything about quotes there.

But if we were discussing the texinfo-mode buffers, then handling of 
quotes would be similar to Emacs Lisp source buffers (but necessarily 
different, because of different syntax).

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:
> On 06/26/2015 05:35 AM, Paul Eggert wrote:
>> I just now tried it against the current master (commit
>> 99ad90dcb1beccde926d9b6475a393c6f8743f5c), and didn't notice any
>> difference in display.

> Like mentioned in the preceding email, that patch is against f743819.
>
> But if you were looking for changes to revert, those would be the uses of curly
> quotes as markup in the source code, the new `substitute-command-keys' calls, as
> well as the code in `substitute-command-keys' that performs quote replacement.

In that case I don't understand the patch being against f743819.  f743819 
already has some curved quotes in docstrings, to avoid ambiguities of using 
grave accent and apostrophe to quote.  So that patch against f743819 won't 
determine whether font-lock can address this issue without using curved quotes 
in docstrings.

Even with this in mind, though, the patch mishandles some quotes.  For example, 
the docstring for texinfo-format-verb contains:

   For example, @verb\{|@|\} results in @ and
   @verb\{+@'e?`!`+} results in @'e?`!`.

The patch displays this as:

   For example, @verb{|@|} results in @ and
   @verb{+@'e?`!‘+} results in @’e?`!`.

which is incorrect: those curved quotes should be grave accent and apostrophe.

It'd be impractical to work around this sort of problem entirely with clever 
font-lock regular expressions.  We will need some escape syntax to suppress 
transliteration for exceptional docstrings like the above.  I think you've 
mentioned the need for that sort of thing, but it's not clear what it would look 
like or how it would be implemented with font-lock.

Plus, as we've mentioned, the patch can mishandle user-supplied values that 
contain grave accent and apostrophe.

> It will be very easy to limit the conversion to only within strings.

As I understand it we gave up on transliteration of Elisp source code after you 
wrote the above, so for now I'll not comment on this point (or the other points 
of the message that talk about transliterating source code).

> Is Info actually relevant in this discussion? Looking at info.el, it seems to
> deal files where quotes have already been translated
> (Info-mode-font-lock-keywords indicates that), so maybe neither
> substitute-command-keys, nor the new font-lock rules, have to do anything about
> quotes there.

It's relevant, as it uses and displays the curved quotes that some users find 
objectionable, and did so even in 24.5.  But it's fine with me if we leave Info 
mode alone.  In practice it works reasonably well in 24.5.

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/27/2015 08:28 PM, Paul Eggert wrote:

> In that case I don't understand the patch being against f743819.
> f743819 already has some curved quotes in docstrings, to avoid
> ambiguities of using grave accent and apostrophe to quote.  So that
> patch against f743819 won't determine whether font-lock can address this
> issue without using curved quotes in docstrings.

I just needed a version where substitute-command-keys doesn't replace 
the quotes (so that the font-lock rules do have something to work on). 
The question of ambiguity is a matter of refining the patch.

> Even with this in mind, though, the patch mishandles some quotes.  For
> example, the docstring for texinfo-format-verb contains:
>
>    For example, @verb\{|@|\} results in @ and
>    @verb\{+@'e?`!`+} results in @'e?`!`.
>
> The patch displays this as:
>
>    For example, @verb{|@|} results in @ and
>    @verb{+@'e?`!‘+} results in @’e?`!`.
>
> which is incorrect: those curved quotes should be grave accent and
> apostrophe.

Only because the regexp is constructed this way, and we haven't picked 
an escaping syntax.

> It'd be impractical to work around this sort of problem entirely with
> clever font-lock regular expressions.  We will need some escape syntax
> to suppress transliteration for exceptional docstrings like the above.
> I think you've mentioned the need for that sort of thing, but it's not
> clear what it would look like or

I've described the possible options in the previous message here of this 
subthread (in the second half):

http://lists.gnu.org/archive/html/emacs-devel/2015-06/msg00592.html

So, do you find "\\" acceptable? Should I send a patch with support for it?

 > how it would be implemented with font-lock.

There are several examples in our Lisp code of searching for only 
unescaped instances of some character, using a regexp. But failing that, 
examining characters preceding a match would work almost as well.

> Plus, as we've mentioned, the patch can mishandle user-supplied values
> that contain grave accent and apostrophe.

Nothing surprising there either. I'll try to handle it in the next 
version of the patch.

>> It will be very easy to limit the conversion to only within strings.
>
> As I understand it we gave up on transliteration of Elisp source code
> after you wrote the above, so for now I'll not comment on this point (or
> the other points of the message that talk about transliterating source
> code).

You have also snipped the suggestions about the escaping syntax.

>> Is Info actually relevant in this discussion? Looking at info.el, it
>> seems to
>
> It's relevant, as it uses and displays the curved quotes that some users
> find objectionable, and did so even in 24.5.

I thought the external program performs the conversion. But yes, 
converting those curly quotes to straight ones in font-lock is also 
possible.

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:
> I've described the possible options in the previous message here of this
> subthread (in the second half):
>
> http://lists.gnu.org/archive/html/emacs-devel/2015-06/msg00592.html

I'm afraid I don't understand those options.  I can't make heads or tails of 
what was suggested there.

How about if you pick a plausible option and give a realistic example showing 
(1) what the source code would be, (2) how the source code would be displayed, 
(3) what the *Help* buffer would be, (4) how the *Help* buffer would be 
displayed normally, and (5) how the *Help* buffer would be displayed if the user 
requests ASCII approximations.

You're objecting to a simple approach where (1) thru (4) are typically all the 
same and use curved quotes, and where (5) uses ASCII approximations.  I don't 
know what you're proposing in place of this.

> BTW, do you have an example we can easily refer to when testing the patches?

I don't think any single example will illustrate all the gotchas.  But how about 
if we start with the previously-mentioned example, e.g., type ‘C-h 
texinfo-format-verb’ after loading textmodes/texinfmt?

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/28/2015 12:09 AM, Paul Eggert wrote:

> How about if you pick a plausible option and give a realistic example
> showing (1) what the source code would be, (2) how the source code would
> be displayed, (3) what the *Help* buffer would be, (4) how the *Help*
> buffer would be displayed normally, and (5) how the *Help* buffer would
> be displayed if the user requests ASCII approximations.

Ok, try this, please. "\\" before a quote keeps it untranslated.

However, it order for us to be able to show an arbitrary number of 
backslashes before a quote (both translated or not), the backslashes 
also escape themselves. I've limited this effect to only before a quote, 
so that we don't have to add them in a lot of places (like 
substitute-command-keys docstring).

> I don't think any single example will illustrate all the gotchas.  But
> how about if we start with the previously-mentioned example, e.g., type
> ‘C-h texinfo-format-verb’ after loading textmodes/texinfmt?

It's good to see how escaping works, but I meant an example of a Help 
buffer where we must not translate quotes in the value.

diff --git a/lisp/help-fns.el b/lisp/help-fns.el
index 4982ee5..aaaf02b 100644
--- a/lisp/help-fns.el
+++ b/lisp/help-fns.el
@@ -746,7 +746,8 @@ it is displayed along with the global value."
  		      (setq from (point))
  		      (pp origval)
  		      (if (< (point) (+ from 20))
-			  (delete-region (1- from) from)))))))
+			  (delete-region (1- from) from)))))
+                (put-text-property val-start-pos (point) 'help-value t)))
  	    (terpri)
  	    (when locus
  	      (cond
diff --git a/lisp/help-mode.el b/lisp/help-mode.el
index f99e916..b5bf618 100644
--- a/lisp/help-mode.el
+++ b/lisp/help-mode.el
@@ -97,6 +97,18 @@ The format is (FUNCTION ARGS...).")
    "Hook run by `help-mode'."
    :type 'hook
    :group 'help)
+
+(defcustom help-quote-translation nil
+  "Style to use for single quotes in help.
+The value is a left single quote character of some style.
+Quote ‘like this’ if the value is ?‘ (left single quotation mark).
+Quote \\'like this\\' if the value is ?\\' (apostrophe).
+Quote \\`like this\\' if the value is ?\\` (grave accent).
+The default value is nil, which means quote with left single quotation mark
+if displayable, and with grave accent otherwise."
+  :type 'character
+  :group 'help)
+
  \f
  ;; Button types used by help

@@ -287,9 +299,37 @@ Commands:
  \\{help-mode-map}"
    (set (make-local-variable 'revert-buffer-function)
         'help-mode-revert-buffer)
+  (setq font-lock-defaults '(nil t))
+  (font-lock-add-keywords
+   nil '(("\\(?:\\=\\|[^\\]\\)\\(\\\\*\\)[`']"
+          (0 (let* ((mbeg (match-beginning 1))
+                    (mend (match-end 1))
+                    (escapes (- mend mbeg)))
+               (unless (get-text-property mend 'help-value)
+                 ;; Collapse all escaped backslashes.
+                 (compose-region mbeg (+ mbeg (- escapes (/ escapes 
2))) "")
+                 ;; If there's an even number of backslashes,
+                 ;; translate the quote.
+                 (when (eq (logand escapes 1) 0)
+                   (help--translate-quote mend)))
+               nil)))))
    (set (make-local-variable 'bookmark-make-record-function)
         'help-bookmark-make-record))

+(defun help--translate-quote (beg)
+  (let* ((char (char-after beg))
+         (replacement
+          (cond
+           ((or (and (null help-quote-translation)
+                     (char-displayable-p ?‘))
+                (eq help-quote-translation ?‘))
+            (cdr (assq char '((?` . ?‘) (?' . ?’)))))
+           ((eq help-quote-translation ?')
+            ?')
+           (t char))))
+    (unless (eq char replacement)
+      (compose-region beg (1+ beg) replacement))))
+
  ;;;###autoload
  (defun help-mode-setup ()
    (help-mode)
diff --git a/lisp/textmodes/texinfmt.el b/lisp/textmodes/texinfmt.el
index e7b6835..392d45b 100644
--- a/lisp/textmodes/texinfmt.el
+++ b/lisp/textmodes/texinfmt.el
@@ -2493,8 +2493,8 @@ surrounded by in angle brackets."
  Enclose the verbatim text, including the delimiters, in braces.  Print
  text exactly as written (but not the delimiters) in a fixed-width.

-For example, @verb\{|@|\} results in @ and
-@verb\{+@'e?`!`+} results in @'e?`!`."
+For example, @verb{|@|} results in @ and
+@verb{+@\\'e?\\`!\\`+} results in @\\'e?\\`!\\`."

    (let ((delimiter (buffer-substring-no-properties
  		    (1+ texinfo-command-end) (+ 2 texinfo-command-end))))

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 2623 bytes --]

Dmitry Gutov wrote:
>>
>
> Ok, try this, please. "\\" before a quote keeps it untranslated.
>
> However, it order for us to be able to show an arbitrary number of backslashes
> before a quote (both translated or not), the backslashes also escape themselves.
> I've limited this effect to only before a quote, so that we don't have to add
> them in a lot of places (like substitute-command-keys docstring).

That rule is too complicated.  Please let's keep it simple.  It's simpler to 
explain if the escape sequence always works.  It should be rare to need these 
escapes, so it's OK if they are multicharacter.  How about your other suggestion 
of using backslash-tilde?  This would be \\~ in the source-file string.  This is 
unlikely to occur in docstrings; there are no occurrences in the current Emacs 
sources.

> +Quote \\'like this\\' if the value is ?\\' (apostrophe).
> ...
> +Quote \\`like this\\' if the value is ?\\` (grave accent).

There should be no need to escape the apostrophes in these lines (see the next 
comment).

> +   nil '(("\\(?:\\=\\|[^\\]\\)\\(\\\\*\\)[`']"

First, this translates all apostrophes to quotes.  It should translate only the 
apostrophes that are used as quotes (i.e., those that match grave accents). 
That's what the current master does, and what your earlier prototype did.  We 
shouldn't try to second-guess unmatched apostrophe, any more than we should try 
to second-guess double-quote.

Second, the last bracketed RE should be [`'‘’] if we are to mimic the 
current-master behavior.  (However, please see the next comment.)

> +           ((or (and (null help-quote-translation)
> +                     (char-displayable-p ?‘))
> +                (eq help-quote-translation ?‘))

As I understand it, the need for help-quote-translation has gone away, since the 
primary impetus for it was that one couldn't easily search for quotes, and that 
problem has been fixed in the meantime by Artur's commits.  So this part can be 
simplified and the code can always assume that the above expression evaluates to 
t.  (This is also true of the current master of course.)

> I meant an example of a Help buffer where we must not translate quotes in the value.

Ah, OK, try loading the attached file.  The file name contains accent grave and 
apostrophe, which should not be translated when one types ‘C-h v foo’.

PS.  Can you please attach patches instead of pasting them bodily into your 
email?  I could not apply the patch in your email automatically, as some 
characters were munged somewhere in the process.


[-- Attachment #2: `big deal'.el --]
[-- Type: text/x-emacs-lisp, Size: 97 bytes --]

(defvar foo
  "`xxx-yyy' \\`zzz-www' \\~`aaa-bbb\\~'"
  "Don't ask me what this variable does.")

Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/28/2015 06:20 PM, Paul Eggert wrote:

> That rule is too complicated.  Please let's keep it simple.  It's
> simpler to explain if the escape sequence always works.  It should be
> rare to need these escapes, so it's OK if they are multicharacter.

Well, if you think so.

> How
> about your other suggestion of using backslash-tilde?  This would be \\~
> in the source-file string.  This is unlikely to occur in docstrings;
> there are no occurrences in the current Emacs sources.

I remember Stefan disliking \\=. But okay, let's try this one.

>> +   nil '(("\\(?:\\=\\|[^\\]\\)\\(\\\\*\\)[`']"
>
> First, this translates all apostrophes to quotes.  It should translate
> only the apostrophes that are used as quotes (i.e., those that match
> grave accents). That's what the current master does, and what your
> earlier prototype did.  We shouldn't try to second-guess unmatched
> apostrophe, any more than we should try to second-guess double-quote.

You expressed dislike for "action at a distance" and wanted a simple 
translation, so there it was. Ok, let's try matching.

> Second, the last bracketed RE should be [`'‘’] if we are to mimic the
> current-master behavior.  (However, please see the next comment.)

There's no technical difficulty there, but I don't want to see curly 
quotes used as markup.

And if we also want to translate them to help with terminals that can't 
display curlies, the translation should affect all curlies, not just the 
markup ones (so we can't limit it to paired quotes, or unescaped ones).

> As I understand it, the need for help-quote-translation has gone away,
> since the primary impetus for it was that one couldn't easily search for
> quotes, and that problem has been fixed in the meantime by Artur's
> commits.  So this part can be simplified and the code can always assume
> that the above expression evaluates to t.  (This is also true of the
> current master of course.)

IIUC, there was also a problem with some terminals that can't display 
them, and also users that would prefer not to see them in Help. Maybe we 
can disregard the latter, at least for now.

>> I meant an example of a Help buffer where we must not translate quotes
>> in the value.
>
> Ah, OK, try loading the attached file.  The file name contains accent
> grave and apostrophe, which should not be translated when one types ‘C-h
> v foo’.

Thanks. The last patch seems to successfully deal with it already.

> PS.  Can you please attach patches instead of pasting them bodily into
> your email?  I could not apply the patch in your email automatically, as
> some characters were munged somewhere in the process.

Ok. But I'd rather push to the scratch/quote-escaping branch, if you 
don't mind.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/28/2015 11:27 PM, Dmitry Gutov wrote:

> Ok. But I'd rather push to the scratch/quote-escaping branch, if you
> don't mind.

Check it out.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:
> On 06/28/2015 11:27 PM, Dmitry Gutov wrote:
>
>> Ok. But I'd rather push to the scratch/quote-escaping branch, if you
>> don't mind.
>
> Check it out.

I tried it out.  One thing I noticed right away is odd behavior that results 
from storing grave accent and apostrophe in the buffer and displaying them as 
curved quotes.  For example, run 'emacs -Q -nw', then read the documentation for 
the 'length' function and copy it to a new file /tmp/foo by typing "C-h f length 
RET C-x o C-x h M-w C-x 4 f /tmp/foo RET C-y C-s".  On the screen you'll see a 
buffer 'foo' containing curved quotes.  Now, revisit /tmp/foo by typing "C-x C-v 
RET".  The buffer will contain the same contents as before, except the curved 
quotes are magically transformed to grave accent and apostrophe on the display. 
  It's weird that copied text mysteriously changes its display representation at 
a seemingly unrelated moment.

How about the following idea instead.  Instead of displaying grave accent and 
apostrophe specially, have with-help-window transliterate these characters in 
place before displaying itself as usual.  with-help-window should not 
transliterate characters that are marked as being escaped, or as being user data 
(not clear that we need two kinds of marks here; one should do, no?).

That's the big picture.  Here are a few more-minor remarks.

> +  (font-lock-add-keywords
> +   nil '(("\\(\\\\~\\)\\(?:\\\\~\\|.\\)"

As already mentioned, the new \~ quoting syntax doesn't seem to be needed; we 
can get by with the existing \= quoting syntax.  So let's go that direction.

Also, this regexp string matches either (1) \~\~ or (2) \~ followed by any 
non-newline character.  But if \~ is supposed to escape the next character, the 
regexp string should simply implement that, i.e., the regexp string should be 
"\\\\~\\(.\\|\n\\)".

I assume the extra complication is about escaping backslash itself, e.g., if the 
docstring is \~\ (which would look like "\\~\\" in the source code) this should 
stand for \ in the *Help* buffer.  But the above regexp doesn't do that. If we 
were to keep \~ perhaps we could do the simple regexp match first, and then look 
backwards programatically just before the matching \~ and verifying that either 
(1) it's not preceded by ordinary ~ or (2) it's preceded by ordinary ~ but the ~ 
is not preceded by ordinary \.  By "ordinary" I mean a character that does not 
have the help-value or help-tilde-escaped properties.

But all and all this is sounding way tooo complicated and we should simply use 
the same escape sequences we've always been using.

> +               (unless (get-text-property mbeg 'help-value)

Supposed the matched string is partly help-value, and partly not.  E.g., mbeg 
has help-value but mbeg+1 does not but mbeg+2 does.  Shouldn't this test that 
all the matched characters are not help-value characters?

> +                 ;; If we use "" as the third argument, cursor
> +                 ;; stumbles once when moving over its position.

I don't understand this comment.  Can you explain?  For example, does the 
comment apply to the just the compose-region call, or to the rest of the 'unless'?

> +                                 (buffer-substring-no-properties
> +                                  mend (1+ mend)))

This may go haywire if it returns "\t", because a TAB is special to 
compose-region.  Also, what if the buffer has some properties other than 
help-value that should be preserved?

At this point I stopped reviewing the changes in detail, as really the big 
picture should be addressed first.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 07/01/2015 05:56 AM, Paul Eggert wrote:

> I tried it out.  One thing I noticed right away is odd behavior that
> results from storing grave accent and apostrophe in the buffer and
> displaying them as curved quotes.  For example, run 'emacs -Q -nw', then
> read the documentation for the 'length' function and copy it to a new
> file /tmp/foo by typing "C-h f length RET C-x o C-x h M-w C-x 4 f
> /tmp/foo RET C-y C-s".  On the screen you'll see a buffer 'foo'
> containing curved quotes.  Now, revisit /tmp/foo by typing "C-x C-v
> RET".  The buffer will contain the same contents as before, except the
> curved quotes are magically transformed to grave accent and apostrophe
> on the display.  It's weird that copied text mysteriously changes its
> display representation at a seemingly unrelated moment.

It's the same if you copy some syntax-highlighted text (say, from 
src/doc.c) to /tmp/foo and save it. Kill the buffer, reopen - the 
highlighting is gone! :)

That may seem counter-intuitive, but it's something Emacs users are 
generally familiar with.

> How about the following idea instead.  Instead of displaying grave
> accent and apostrophe specially, have with-help-window transliterate
> these characters in place before displaying itself as usual.

That should work, too. In help-mode-finish, before help-make-xrefs.

> with-help-window should not transliterate characters that are marked as
> being escaped, or as being user data (not clear that we need two kinds
> of marks here; one should do, no?).

They're semantically different. I think we currently apply linkification 
(help-make-xrefs) to the contents of user data, and we might continue 
doing that.

On the other hand, we should probably avoid linkifying symbols inside 
quotes when at least one of the quotes is escaped.

> That's the big picture.  Here are a few more-minor remarks.
>
>> +  (font-lock-add-keywords
>> +   nil '(("\\(\\\\~\\)\\(?:\\\\~\\|.\\)"
>
> As already mentioned, the new \~ quoting syntax doesn't seem to be
> needed; we can get by with the existing \= quoting syntax.  So let's go
> that direction.

For us to go there, could you please make substitute-command-keys add 
the `escaped' property to the escaped characters in its output? And push 
it to scratch/quote-escaping.

> Also, this regexp string matches either (1) \~\~ or (2) \~ followed by
> any non-newline character.  But if \~ is supposed to escape the next
> character, the regexp string should simply implement that, i.e., the
> regexp string should be "\\\\~\\(.\\|\n\\)".

I guess so. Sorry, brain fart.

> I assume the extra complication is about escaping backslash itself,
> e.g., if the docstring is \~\ (which would look like "\\~\\" in the
> source code) this should stand for \ in the *Help* buffer.  But the
> above regexp doesn't do that.

Doesn't it? Works for me. But anyway, let's try to reuse the quoting 
from substitute-command-keys first.

>> +               (unless (get-text-property mbeg 'help-value)
>
> Supposed the matched string is partly help-value, and partly not.  E.g.,
> mbeg has help-value but mbeg+1 does not but mbeg+2 does.  Shouldn't this
> test that all the matched characters are not help-value characters?

Why? I'm assuming the value is separated from the other contents by 
whitespace or newlines. IMHO, that would too defensive. We're throwing 
that code out anyway.

On the other hand, we could encounter a value between ` and ', in 
help--translate-quotes, if the value is shown at the end of the buffer.

Should be taken care of in f9f3aa5 (as well as another nearby bug).

>> +                 ;; If we use "" as the third argument, cursor
>> +                 ;; stumbles once when moving over its position.
>
> I don't understand this comment.  Can you explain?  For example, does
> the comment apply to the just the compose-region call, or to the rest of
> the 'unless'?

Only to the compose-region call. If we pass "" to it as COMPOSITION, the 
result will still be considered a valid position for the cursor, even 
though it has zero width. Anyway, we're going a different route: no 
compose-region calls.

>> +                                 (buffer-substring-no-properties
>> +                                  mend (1+ mend)))
>
> This may go haywire if it returns "\t", because a TAB is special to
> compose-region.  Also, what if the buffer has some properties other than
> help-value that should be preserved?

Err, I don't think the current code deletes any existing properties, it 
only changes how the buffer looks.

On the other hand, if help-mode-finish performs the translations 
(destructively, I'm assuming), then we indeed might lose some properties.

I wouldn't worry too much about that, though. If help-mode doesn't know 
about them, no other code is likely to use them either.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:

> It's the same if you copy some syntax-highlighted text (say, from src/doc.c) to
> /tmp/foo and save it. Kill the buffer, reopen - the highlighting is gone! :)
> That may seem counter-intuitive, but it's something Emacs users are generally
> familiar with.

I've seen highlighting vanish, but I've never seen the characters change. 
Anyway....

>> How about the following idea instead.  Instead of displaying grave
>> accent and apostrophe specially, have with-help-window transliterate
>> these characters in place before displaying itself as usual.
>
> That should work, too. In help-mode-finish, before help-make-xrefs.

OK, let's shoot for that instead.

> For us to go there, could you please make substitute-command-keys add the
> `escaped' property to the escaped characters in its output? And push it to
> scratch/quote-escaping.

OK, I'll look into that.

>>> +               (unless (get-text-property mbeg 'help-value)
>>
>> Supposed the matched string is partly help-value, and partly not.  E.g.,
>> mbeg has help-value but mbeg+1 does not but mbeg+2 does.  Shouldn't this
>> test that all the matched characters are not help-value characters?
>
> Why? I'm assuming the value is separated from the other contents by whitespace
> or newlines.

I don't think that's a safe assumption.  It's common for quoted help-values to 
be jammed into the middle of other text.

>>> +                                 (buffer-substring-no-properties
>>> +                                  mend (1+ mend)))
>>
>> This may go haywire if it returns "\t", because a TAB is special to
>> compose-region.  Also, what if the buffer has some properties other than
>> help-value that should be preserved?
>
> Err, I don't think the current code deletes any existing properties, it only
> changes how the buffer looks.

But buffer-substring-no-properties has no properties in the result, right?  So 
effectively they're removed.  And what about the TAB?

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 07/02/2015 09:57 AM, Paul Eggert wrote:

> I've seen highlighting vanish, but I've never seen the characters
> change. Anyway....

Try that with prettify-symbols-mode.

>> Why? I'm assuming the value is separated from the other contents by
>> whitespace
>> or newlines.
>
> I don't think that's a safe assumption.  It's common for quoted
> help-values to be jammed into the middle of other text.

I'd need an example. But anyway, this question will be moot if we 
delegate escaping to the lower level.

> But buffer-substring-no-properties has no properties in the result,
> right?  So effectively they're removed.

No, they remain on the text. We just use a small piece of it, without 
properties, to change the appearance of the said text (compose-region 
adds an extra property).

> And what about the TAB?

Indeed, it's a problem in that implementation. But anyway, again, we're 
unlikely to end up using it.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

>> For us to go there, could you please make substitute-command-keys add the
>> `escaped' property to the escaped characters in its output? And push it to
>> scratch/quote-escaping.

I've done that, and have discovered a couple of problems.

First, a doc string that contains \[foo] is supposed to generate a key 
description for 'foo', derived by calling (key-description 'foo).  But 
key-description can return a string that contains quote marks, and in general 
some of them should be escaped and others should not be.  It's not clear to me 
whether both possibilities can occur in doc strings, but the situation is worrisome.

Second, and more generally, the overall approach seems error prone.  It asks 
programmers to mark every quote character that should not be transformed.  But 
characters can come from many different sources, and it's hard to keep track of 
each place that can insert user data of some sort.  In contrast, it's easy to 
find grave accents in doc strings, to fix the relative few that aren't already 
transformed automatically, and to not worry about user data.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 07/06/2015 09:12 AM, Paul Eggert wrote:

> I've done that, and have discovered a couple of problems.

Thank you.

> First, a doc string that contains \[foo] is supposed to generate a key
> description for 'foo', derived by calling (key-description 'foo).  But
> key-description can return a string that contains quote marks, and in
> general some of them should be escaped and others should not be.  It's
> not clear to me whether both possibilities can occur in doc strings, but
> the situation is worrisome.

Why not put `escaped' (or rename that `translated', maybe) on those 
generated regions as well?

> Second, and more generally, the overall approach seems error prone.  It
> asks programmers to mark every quote character that should not be
> transformed.  But characters can come from many different sources, and
> it's hard to keep track of each place that can insert user data of some
> sort.

So far I've only seen a few places of complication nearby: key 
translations, keymap translations, and the escaping mechanism itself.

The rest of them will likely have to be handled in 
substitute-command-keys anyway, somehow.

> In contrast, it's easy to find grave accents in doc strings, to
> fix the relative few that aren't already transformed automatically, and
> to not worry about user data.

If we find it hard to expose this data to Lisp, it will make the output 
presentation less flexible, and could indicate an overall problem in our 
docstring generation (we don't know all the possible sources the 
characters can come from? that's not very good).

By the way, when doing translation in Lisp, instead of using 
help-mode-finish, we can instead examine all the functions that generate 
help buffers for Lisp stuff, and translate only the docstrings, when 
they are being inserted.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:
> Why not put `escaped' (or rename that `translated', maybe) on those generated
> regions as well?

Yes, in theory it could be done.  But see below.

>> Second, and more generally, the overall approach seems error prone.  It
>> asks programmers to mark every quote character that should not be
>> transformed.  But characters can come from many different sources, and
>> it's hard to keep track of each place that can insert user data of some
>> sort.
>
> So far I've only seen a few places of complication nearby: key translations,
> keymap translations, and the escaping mechanism itself.

I'm afraid this is because you've only started looking.  Even if we limit 
ourselves to the regions you mention, the relevant code is scattered all over 
the place and it's hard to find exactly where to put 'escaped' or 'translated' 
into those regions.  For example, one place might be the following code in 
keymap.c's single_key_description function:

	  Lisp_Object result;
	  char *buffer = SAFE_ALLOCA (sizeof "<>"
				      + SBYTES (SYMBOL_NAME (key)));
	  esprintf (buffer, "<%s>", SDATA (SYMBOL_NAME (key)));
	  result = build_string (buffer);
	  SAFE_FREE ();
	  return result;

Since KEY can be an arbitrary symbol, quite possibly its characters should be 
marked as 'escaped' or 'translated', which would mean adding this:

   put_text_property (1, 1 + SCHARS (SYMBOL_NAME (key)), Qescaped, Qt, result);

before the return.  Unfortunately it's not at all obvious whether this is needed 
unless one examines all the ways that C-h can put text into the *Help* buffer 
(which I haven't done).  And one must do this potentially every time a string or 
symbol name is looked at.  Plus, there are a lot of possibilities for off-by-one 
errors: for example, that '1 +' might easily be forgotten.  Plus, even then 
there are opportunities for bugs: the above call to 'put_text_property' is not 
correct if the symbol name contains a NUL byte.  Plus, there are other areas 
that will require this sort of complication, e.g., button labels, *Apropos* buffers.

In contrast, it's much easier to look through the code for ` inside a string, 
and mark the exceptional characters that are intended to be quotes and that are 
not automatically translated already.

> (we don't know all the possible sources the characters can
> come from? that's not very good).

No, actually, it's a good thing.  It's nice that programs can put arbitrary text 
into *Help* buffers, and that that we don't need to worry about where the text 
came from: it just works.  That's a simple interface, and simple interfaces are 
good.

> By the way, when doing translation in Lisp, instead of using help-mode-finish,
> we can instead examine all the functions that generate help buffers for Lisp
> stuff, and translate only the docstrings, when they are being inserted.

That's what the master branch is doing now, no?

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 07/06/2015 07:30 PM, Paul Eggert wrote:

> I'm afraid this is because you've only started looking.  Even if we
> limit ourselves to the regions you mention, the relevant code is
> scattered all over the place and it's hard to find exactly where to put
> 'escaped' or 'translated' into those regions.

I meant complications related to `substitute-command-keys'. That 
function should know exactly where to put them, because it expands 
\\[command] and similar syntax.

But if you were looking for consumers of `key-description' in 
help-fns.el, these are concentrated in `help-fns--key-bindings'. And 
it's only used in one place: `describe-function-1'. Putting `help-value' 
on it is trivial.

> For example, one place
> might be the following code in keymap.c's single_key_description function:

That doesn't sound right to me. `key-description' should continue 
returning a plain string.

> Since KEY can be an arbitrary symbol, quite possibly its characters
> should be marked as 'escaped' or 'translated', which would mean adding
> this:

I never was proposing a "dirty" string tagging throughout all functions 
that deal with them.  Only in functions that output to Help buffers (and 
know about it), that deal with `' markup.

>    put_text_property (1, 1 + SCHARS (SYMBOL_NAME (key)), Qescaped, Qt,
> result);
>
> before the return.  Unfortunately it's not at all obvious whether this
> is needed unless one examines all the ways that C-h can put text into
> the *Help* buffer (which I haven't done).  And one must do this
> potentially every time a string or symbol name is looked at.

I'm afraid I don't understand you here.

> Plus,
> there are a lot of possibilities for off-by-one errors: for example,
> that '1 +' might easily be forgotten.

That's one of the reasons to implement as little as possible in C.

> Plus, even then there are
> opportunities for bugs: the above call to 'put_text_property' is not
> correct if the symbol name contains a NUL byte.  Plus, there are other
> areas that will require this sort of complication, e.g., button labels,
> *Apropos* buffers.

Apropos buffers - maybe. But since when do button labels have quotes in 
them?

> In contrast, it's much easier to look through the code for ` inside a
> string, and mark the exceptional characters that are intended to be
> quotes and that are not automatically translated already.

Sorry, I don't understand this either. Mark? Automatically translated?

>> (we don't know all the possible sources the characters can
>> come from? that's not very good).
>
> No, actually, it's a good thing.  It's nice that programs can put
> arbitrary text into *Help* buffers, and that that we don't need to worry
> about where the text came from: it just works.  That's a simple
> interface, and simple interfaces are good.

Well, it's one way of looking at it.

>> we can instead examine all the functions that generate help buffers
>> for Lisp
>> stuff, and translate only the docstrings, when they are being inserted.
>
> That's what the master branch is doing now, no?

That would be closer, but the current master implements more in C, which 
means fewer people who can change or maintain the code. And it 
complicates the API of a low-level function (substitute-command-keys), 
in a way that seems incompatible with radically changing the output later.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:

> That doesn't sound right to me. `key-description' should continue returning a
> plain string.

Perhaps you're right, I haven't looked into key-description carefully enough.  I 
gave it only as an example of the sorts of problems we'd run into.

> since when do button labels have quotes in them?

In customization buffers.

>> In contrast, it's much easier to look through the code for ` inside a
>> string, and mark the exceptional characters that are intended to be
>> quotes and that are not automatically translated already.
>
> Sorry, I don't understand this either. Mark? Automatically translated?

I meant that although normally doc strings are translated and one can quote with 
either grave accent and apostrophe or curved quotation marks and they will be 
translated automatically to the user preferred style, occasionally one wants an 
actual grave accent or curved quotation mark to be output as-is regardless of 
user preference and these exceptional cases need to be marked in the doc strings.

> That would be closer, but the current master implements more in C, which means
> fewer people who can change or maintain the code. And it complicates the API of
> a low-level function (substitute-command-keys), in a way that seems incompatible
> with radically changing the output later.

We can't anticipate radical changes that future maintainers might want to do. 
Worrying about that is likely to lead to overengineering and wasted effort. 
We'll have enough trouble just solving today's problems.

As for whether the implementation should be in C -- some parts have to be used 
by C, since C code generates quoted strings on occasion.  How much should be in 
C vs Lisp is an implementation detail that is no big deal.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 07/07/2015 10:54 AM, Paul Eggert wrote:

> In customization buffers.

Ok. We haven't considered those so far.

So yeah, if there are a lot of places where the docstrings are 
displayed, it's probably better to perform the translation of docstring 
markup in a centralized way.

> I meant that although normally doc strings are translated and one can
> quote with either grave accent and apostrophe or curved quotation marks
> and they will be translated automatically to the user preferred style,
> occasionally one wants an actual grave accent or curved quotation mark
> to be output as-is regardless of user preference and these exceptional
> cases need to be marked in the doc strings.

I sounds like you're describing a use case for an escaping mechanism. 
But what is the point of that? We've already agreed on a need for it.

> We can't anticipate radical changes that future maintainers might want
> to do. Worrying about that is likely to lead to overengineering and
> wasted effort. We'll have enough trouble just solving today's problems.

Concern about the API complication is not over-engineering. We can 
totally worry about it now.

And one alternative use has already been mentioned (at least twice): 
don't use the quotes in the output at all, and instead highlight the 
quoted expressions with color; and possibly a different font-face.

> As for whether the implementation should be in C -- some parts have to
> be used by C, since C code generates quoted strings on occasion.

Not necessarily, C code is perfectly able to use Lisp functions. You'd 
have to look at what the diagnostics formatting will do anyway (would it 
have to support escaping, for instance).

Even so, some duplication might be in order: docstrings and disgnostics 
can do different things in the output; you cannot reliably use colors 
and fonts in the latter.

> How much should be in C vs Lisp is an implementation detail that is no big
> deal.

No big deal for you?

I hope we can agree, at least, that substitute-command-keys shouldn't 
perform the translation itself.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:

> one alternative use has already been mentioned ... don't use
> the quotes in the output at all, and instead highlight the quoted expressions
> with color; and possibly a different font-face.

We can cross that bridge if we ever need to come to it.  Such a use can't be the 
default everywhere, since not every platform supports colors and faces (e.g., 
batch diagnostics).  So we'll need something with quotes anyway.  And we already 
have code that highlights the contents of quotations so that may well be good 
enough.

> C code is perfectly able to use Lisp functions.

I expect that some of the diagnostics are at a low level, and can't assume that 
Lisp functions are callable, and that we'll need to do some of this at the C 
level regardless.  Once it's done there, why bother with redoing it in Lisp?

> I hope we can agree, at least, that substitute-command-keys shouldn't perform
> the translation itself.

Whether substitute-command-keys does it directly, or via calling some other 
function, is an implementation detail and it's not clear to me yet which is the 
right way to go here.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 08/01/2015 04:36 AM, Paul Eggert wrote:

> We can cross that bridge if we ever need to come to it.  Such a use
> can't be the default everywhere, since not every platform supports
> colors and faces (e.g., batch diagnostics).  So we'll need something
> with quotes anyway.  And we already have code that highlights the
> contents of quotations so that may well be good enough.

It doesn't need to be the default, or the default everywhere. But for it 
to be feasible to implement at least in some places, 
substitute-command-keys shouldn't translate the quotes. It should limit 
itself to translating just the constructs it translated before, as well 
as putting `escaped' text property on some characters in the output string.

>> C code is perfectly able to use Lisp functions.
>
> I expect that some of the diagnostics are at a low level, and can't
> assume that Lisp functions are callable, and that we'll need to do some
> of this at the C level regardless.  Once it's done there, why bother
> with redoing it in Lisp?

Highlighting is performed in Lisp. Linkification is performed in Lisp.

For instance, help-make-xrefs scans the Help buffer, the contents of 
which have already passed through substitute-command-keys, for matches 
of help-xref-symbol-regexp. Without knowing which quotes were escaped, 
and which appeared in the docstring as-is, it can make wrong 
cross-references. Take this definition:

(defvar foo nil
   "Referencing `\\=`--pcase-macroexpander' macro.")

The references to the function name is not linkified, and to solve this 
problem better in general, Lisp will need to know which characters were 
escaped and which weren't. Here's a contrived example which can't be 
fixed without that:

(defun ’‘ (a b c)
   "It's called `’‘'."
   (+ 1 2 3))

Of course, we don't have any functions with curly quotes in the name 
now. But weren't some people just recently clamoring for wider use of 
Unicode in Emacs Lisp sources?

> Whether substitute-command-keys does it directly, or via calling some
> other function, is an implementation detail and it's not clear to me yet
> which is the right way to go here.

It shouldn't perform the translation at all. Not directly, nor via 
another function. Its callers can do the translation better, and maybe 
in different ways, depending on the context.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:
> Here's a contrived example which can't be fixed without that:
>
> (defun ’‘ (a b c)
>    "It's called `’‘'."
>    (+ 1 2 3))

Yes, and one can come up with other contrived examples that fail even without 
putting curved quotes into the name.  Elisp function names can contain any 
character: apostrophe, grave accent, newline, parenthesis, space, etc., and I'm 
sure many of these other special characters also cause problems in help buffers.

While it might be worthwhile to fix this, it's an independent issue.  If it is 
fixed, I expect you're right that substitute-command-keys will need to be teased 
apart; this is not just because of curved quotes, but also because its other 
substitutions can generate characters that also need special treatment.  If 
someone wants to take on that task, that's great, so long as it doesn't get in 
the way of ordinary use of *Help* buffers.  In particular, users should be able 
to type ‘C-h f length RET’, see this in a *Help* buffer:

   To get the number of bytes, use ‘string-bytes’.

and if they save and later yank this text into a source-code buffer they should 
get what they see, not a bowdlerized version with ASCII approximations.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 08/02/2015 09:56 AM, Paul Eggert wrote:

> Yes, and one can come up with other contrived examples that fail even
> without putting curved quotes into the name.  Elisp function names can
> contain any character: apostrophe, grave accent, newline, parenthesis,
> space, etc., and I'm sure many of these other special characters also
> cause problems in help buffers.

But we have an escaping syntax! Yet, Elisp doesn't know what's escaped 
and what isn't, so even this won't help:

(defun ’‘ (a b c)
  "It's called `\\=’\\=‘'."
   (+ 1 2 3))

> While it might be worthwhile to fix this, it's an independent issue.  If
> it is fixed, I expect you're right that substitute-command-keys will
> need to be teased apart;

It's also a matter of readability. Take the latest related commit that 
you pushed. Sometime later, someone will come along and wonder: why do 
we need to substitute command keys in widget options? Who would put keys 
in there?

> this is not just because of curved quotes, but
> also because its other substitutions can generate characters that also
> need special treatment.

You might want to give an example. But in general, those characters also 
could have `escaped' put on them. Or `substituted', for instance.

> If someone wants to take on that task, that's
> great, so long as it doesn't get in the way of ordinary use of *Help*
> buffers.

As long as that task involves rearranging C code, you're excluding a 
significant portion of Emacs developers from contributing, myself included.

> In particular, users should be able to type ‘C-h f length
> RET’, see this in a *Help* buffer:
>
>    To get the number of bytes, use ‘string-bytes’.
>
> and if they save and later yank this text into a source-code buffer they
> should get what they see, not a bowdlerized version with ASCII
> approximations.

First, it's an arbitrary condition that you've set yourself, and we've 
discussed it already. If you asked all Emacs users, I suspect the 
majority won't care, and the rest won't reach a unanimous decision. 
Personally, I'd prefer that ‘string-bytes’, after copying and pasting to 
a "normal" buffer, turned into the original markup, which, in this case, 
is `string-bytes'. But that seems non-trivial to implement.

Second, as long as font-lock isn't used for translation (and we've 
pretty much agreed that it won't), why wouldn't a Lisp solution work 
like you describe?

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:
> we have an escaping syntax! Yet, Elisp doesn't know what's escaped and what
> isn't, so even this won't help:
>
> (defun ’‘ (a b c)
>   "It's called `\\=’\\=‘'."
>    (+ 1 2 3))

Again, this problem is independent of quoting style, and has long existed in 
Emacs evem with the older quoting style.  For example:

(defun \'\` (a b c)
  "It's called `\\='\\=`'."
   (+ 1 2 3))

has even worse problems than the ’‘ example does (and these problems also exist 
in older Emacs).

> It's also a matter of readability. Take the latest related commit that you
> pushed. Sometime later, someone will come along and wonder: why do we need to
> substitute command keys in widget options? Who would put keys in there?

That can be addressed by renaming the function to 'substitute-doc-string'.

>> this is not just because of curved quotes, but
>> also because its other substitutions can generate characters that also
>> need special treatment.
>
> You might want to give an example.

(defun foo (a b c)
   "It's invoked by `\\[next-error]'."
     (+ 1 2 3))

> But in general, those characters also could
> have `escaped' put on them. Or `substituted', for instance.

Yes, and substituting \[...] has problems and solutions that are quite similar 
to substituting ` and '.  So even if it makes sense to tease this functionality 
apart in some cases, it also makes sense to have a single function that does 
both subsitutions for convenience, as typically programs will not want to do one 
without doing the other.

> As long as that task involves rearranging C code, you're excluding a significant
> portion of Emacs developers from contributing, myself included.

It's a simple-enough matter to add an argument to substitute-doc-string 
specifying which kinds of substitutions are wanted.  I can volunteer to do that 
if it would be helpful (though I confess I don't see the use case).  Or if you 
prefer you can rewrite substitute-doc-string in Lisp -- as long as it doesn't 
affect performance significantly that should be merely an implementation detail.

> as long as font-lock isn't used for translation (and we've pretty much
> agreed that it won't), why wouldn't a Lisp solution work like you describe?

I mentioned the possibility because it's still not clear to me what a Lisp 
solution would be, if it's not something involving font-lock.  If the Lisp 
solution merely translates the existing C code to Lisp, then of course your 
point is correct.

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 08/02/2015 06:13 PM, Paul Eggert wrote:

> Again, this problem is independent of quoting style, and has long
> existed in Emacs evem with the older quoting style.  For example:

We didn't have an escaping syntax for quotes before. You've increased 
the complexity of our handling of quoting. Might as well fix some 
pre-existing problems. Right?

Look at it from the user's point of view: quotes can be escaped, it's 
documented in the substitute-command-keys docstring. And yet, it doesn't 
help in this kind of situations, where it seemingly should.

> (defun foo (a b c)
>    "It's invoked by `\\[next-error]'."
>      (+ 1 2 3))

This works now, and would continue to work, as per my suggestion.

> Yes, and substituting \[...] has problems and solutions that are quite
> similar to substituting ` and '.  So even if it makes sense to tease
> this functionality apart in some cases, it also makes sense to have a
> single function that does both subsitutions for convenience, as
> typically programs will not want to do one without doing the other.

When both substitutions are required, calling the second function on the 
result of the first function will be just as easy. Splitting the logic 
between two functions will make sure the first one doesn't lose information.

> It's a simple-enough matter to add an argument to substitute-doc-string
> specifying which kinds of substitutions are wanted.  I can volunteer to
> do that if it would be helpful (though I confess I don't see the use
> case).

It's better to keep the quote translation logic in one place (in a 
different function).

> Or if you prefer you can rewrite substitute-doc-string in Lisp
> -- as long as it doesn't affect performance significantly that should be
> merely an implementation detail.

Rewriting it also requires some C chops, if only to read the original. I 
don't know whether it'll lose performance: it might, handing certain 
keymaps might be intensive.

Anyway, even if the resulting function is in Lisp, it won't help as long 
as it's just one function, and it still loses information about which 
characters were originally escaped.

> I mentioned the possibility because it's still not clear to me what a
> Lisp solution would be, if it's not something involving font-lock.  If
> the Lisp solution merely translates the existing C code to Lisp, then of
> course your point is correct.

The Lisp solution would replace actual characters with different 
characters. In the text, not apply the transformation via text 
properties (although that's also an option).

Re: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Przemysław Wojnowski]

Hi.

Is this still thread about a change from the type of quote that is on
each keyboard (`') to the other one (that is not)?

IMHO it would be fantastic exercise to do a retrospective after
implementing the feature. My two questions for the retrospective:

1. What was purpose of the change?
To replace `' with the other one (sorry, don't have it on my keyboard)?
What for?

2. *Was it worth it*?
Considering number of people involved and time spent by those
writing and reading emails (even I'm wasting my time now on writing
this email).

Don't get me wrong, I'm not saying that you shouldn't do what you are
doing. Everyone can do with their time what they want, but maybe it
would be good to create a Backlog for Emacs and prioritize (at least
a bit) to see what's really important and _is_ worth the effort.

Cheers,
Przemysław

RE: Escaping quotes in docstrings, Was: A simple solution to "Upcoming loss of usability ..."

[Drew Adams]

> 1. What was purpose of the change?
>    To replace `' with the other one (sorry, don't have it on my 
>    keyboard)? What for?
> 
> 2. *Was it worth it*?
>    Considering number of people involved and time spent by those
>    writing and reading emails (even I'm wasting my time now on 
>    writing this email).
> 
> Don't get me wrong, I'm not saying that you shouldn't do what
> you are doing.

Provided it is optional for users to suffer from it.
Let them opt in, if they really want this behavior.

At a minimum, let them easily opt out, pretty-please.

I don't even see that possibility currently.  How to opt out
doesn't seem to be documented, at least, and lots of source
code seems to hard-code your changes.

We were told over and over that this would be just a healthy
"experiment" and that people shouldn't react prematurely.

Well, it's been several months now.  Are you prepared to
back this misguided feature out?  

Or were those who decried such a gratuitous & invasive
development correct, in supposing that backing-out would
become next to impossible and this gross hack job would be
here to stay, whatever the post-mortem judgment might be?

> Everyone can do with their time what they want, but maybe it
> would be good to create a Backlog for Emacs and prioritize
> (at least a bit) to see what's really important and _is_
> worth the effort.

+1

So far, my impression is that there have been a shiPload of
deep changes that try to work around all kinds of problems
that have been introduced - and to work around the problems
that those workarounds have introduced...

I haven't seen such an invasive change in a long time (never?).
Makes Rube Goldberg machines look Occam-elegant.  And for
what?  Just to fit the esthetic, cosmetic concerns of two
or three developers?

Please tell users now _how to opt out completely_, at a
minimum.  If letting users restore the longstanding behavior
means getting rid of hard-coded hacks, so be it - please do it.

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/27/2015 08:28 PM, Paul Eggert wrote:

> Plus, as we've mentioned, the patch can mishandle user-supplied values
> that contain grave accent and apostrophe.

BTW, do you have an example we can easily refer to when testing the patches?

Re: A simple solution to "Upcoming loss of usability ..."

[João Távora]

On Thu, Jun 25, 2015 at 11:17 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:

> As I said, action-at-a-distance is not a fatal objection.  But yes, that
> behavior of double-quote is objectionable, and I wish that Emacs didn't do
> it. I hope we don't need to add more glitches like it.

Just my two cents, I largely avoid this problem using electric-pair-mode, which
I designed with this use case in mind. It works in strings and comments as well.
Unsure if it supports your single-quote example, but if not, it should/could.

João

Re: A simple solution to "Upcoming loss of usability ..."

[Oleh Krehel]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Third-party code doesn't need to change.  It'll still work reasonably
> well.

By "reasonably well" you mean that 3rd party will use one notation while
the Emacs sources will use another.

I thought the whole purpose of the change was to make things "more
clear" for new users. I think that the change makes it less clear.

Especially if you consider that a new user is more likely to contribute
to 3rd party Elisp projects before he starts to contribute to Emacs
sources (or even starts to look at them). I know this because that was
the case for me.

I hope most people will agree that having two notations for one thing is
bad for everyone: both novices and experts. There's one popular
programming language with the motto:

> There should be one-- and preferably only one --obvious way to do it.

Look at what happened to it when they tried to mix things up: Python 3
was released in 2008. And I'm still using 2.7.6. Granted, I'm not a
Python expert, but my impression is that many people still use 2.7.

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Oleh Krehel wrote:
> I hope most people will agree that having two notations for one thing is
> bad for everyone: both novices and experts.

Of course.  The problem is that Emacs Lisp currently lacks notation and 
mechanism for doing the right thing with respect to user preference in quoting 
style.  It will be a hassle to modify the old ASCII-only notation and 
implementation to address this problem while remaining ASCII-only.  (I've 
started that job, but it's by no means done.)

In contrast, it's quite simple to use curved quotes to denote quotes.

There's no question that we'll continue to support the old ASCII-only style 
indefinitely, even though it's relatively clumsy and confusing.  The only 
question is whether we'll also support a simpler style in which quotes normally 
stand for themselves, with the idea of moving to the simpler in the long run if 
it works out.  That is, the question is whether the long-term gain is worth the 
extra short-term pain.  (No matter what we do, we'll have some short-term pain; 
this is inevitable.)

This is a judgment call, and the conservative approach is of course to add some 
more escape sequence complication to our ASCII-only approach, and/or to use 
gimmicks like font-lock mode to display characters other than what's actually in 
the buffer.  But really, it's much simpler to use quotes to denote quotes, some 
of us prefer doing it the simpler way, and we should give the simpler 
alternative a try.

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/26/2015 05:54 PM, Paul Eggert wrote:

> In contrast, it's quite simple to use curved quotes to denote quotes.

People voiced their concerns about discomforts when typing those 
already. It's only "simple" in a very narrow sense.

> This is a judgment call, and the conservative approach is of course to
> add some more escape sequence complication to our ASCII-only approach,
> and/or to use gimmicks like font-lock mode to display characters other

font-lock mode is no gimmick.

But you're conflating its usage with the ASCII-only approach again. 
They're orthogonal: you can just as well use ASCII quotes and translate 
them using substitute-command-keys. Or vice versa.

> than what's actually in the buffer.  But really, it's much simpler to
> use quotes to denote quotes, some of us prefer doing it the simpler way,
> and we should give the simpler alternative a try.

I really wish you gave it a try in a feature branch, and then asked for 
a vote before merging it into master.

Re: A simple solution to "Upcoming loss of usability ..."

[Alan Mackenzie]

Hello, Paul.

On Thu, Jun 25, 2015 at 09:36:53AM -0700, Paul Eggert wrote:
> Oleh Krehel wrote:
> > (font-lock-add-keywords
> >   'emacs-lisp-mode
> >   '(("\\(`\\)\\([a-zA-Z-0-9]+\\)\\('\\)"

> The proposed approach would mishandle many cases where the things being quoted 
> are not typical Lisp identifiers.  E.g.:

> "Press ‘h’ for complete help; press ‘?’ repeatedly for a summary"
> "Make ‘funcall/apply’ form to map SOURCE-ARGLIST to TARGET-ARGLIST...."
> "... Example: ‘(ad-map-arglists '(a &rest args) '(w x y z))’ will return ..."

I think there will be several of these cases rather than many.

> Also, the proposed approach won't easily generalize to diagnostics, which often 
> quote non-identifiers like ‘%s’.  There's also a UI problem: it would cause 
> action-at-a-distance, because typing an apostrophe in one place in the buffer 
> would visually alter a part of the line many characters away. 
> (Action-at-a-distance is not a fatal objection, but it is better to avoid it 
> when possible.)

It's not a problem at all.  Font lock does it all the time.  For
example, just type in "save-excursion" a letter at a time.  Only on
typing the "n" does the whole symbol get fontified.

> Most of the advantages you mention for the proposed approach are also advantages 
> of the approach in master.  With the current approach, the Emacs sources don't 
> need to be changed, ....

they've already been massively changed.  

> .... quotes are just as easy to input (in Electric Quote mode), ....

This is an unpleasant workaround.  It violates "what you type is what
you get".

> .... terminal and copy-pasting work, and quotes are markup.

Which is true, for certain values of "terminal" and "copy-pasting".

> The main advantage of the proposed approach over the current master is that the 
> source code often can still contain grave accent and apostrophe unmodified, even 
> though people reading and editing the source code will see curved quotes.  To my 
> mind this is more a recipe for confusion than anything else -- at least, I 
> wouldn't want to inflict it on Emacs newcomers.

The main advantage is that non-working characters, the curly quotes,
would not take a central role in Emacs Lisp source code, together with
all the workarounds of questionable taste that they necessitate.  It is
solely these non-working characters which I take exception to, and I
think the same is true of several others objecting to these changes.

How about considering these other approaches, in which non-working
characters would not be proliferated through the strings in our source
code?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Alan Mackenzie wrote:
> The main advantage is that non-working characters, the curly quotes,
> would not take a central role in Emacs Lisp source code

Even with the font-lock approach, curved quotes will commonly appear on the 
screen and users and developers will have to deal with them, regardless of 
whether most of the quotes are the result of on-the-fly transformations.  So 
this is not a real advantage over the current approach.

I have been seriously considering font-lock approaches but so far they appear to 
be more confusing and less reliable than the alternatives.  For example, how 
would font-lock alter the quotes that Emacs outputs in batch diagnostics?

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/26/2015 01:34 AM, Paul Eggert wrote:

> Even with the font-lock approach, curved quotes will commonly appear on
> the screen and users and developers will have to deal with them,

Not if they aren't in source files, and if the user decided to turn off 
their rendering.

> I have been seriously considering font-lock approaches but so far they
> appear to be more confusing and less reliable than the alternatives.
> For example, how would font-lock alter the quotes that Emacs outputs in
> batch diagnostics?

It probably won't, you'll need a different function. However, 
substitute-command-keys wouldn't seem right for that purpose either (are 
there any command keys in diagnostics?).

Re: A simple solution to "Upcoming loss of usability ..."

[Paul Eggert]

Dmitry Gutov wrote:
>
>> I have been seriously considering font-lock approaches but so far they
>> appear to be more confusing and less reliable than the alternatives.
>> For example, how would font-lock alter the quotes that Emacs outputs in
>> batch diagnostics?
>
> It probably won't, you'll need a different function.

OK, that can be the format-message function already discussed.  So the font-lock 
proposal isn't intended to address the problem of diagnostics, or of info or 
help buffers for that matter; it's intended only to address the issue of 
displaying docstrings to a developer who's editing Elisp code.  Is that correct?

Re: A simple solution to "Upcoming loss of usability ..."

[Dmitry Gutov]

On 06/26/2015 01:45 AM, Paul Eggert wrote:

> OK, that can be the format-message function already discussed.

Indeed.

> So the
> font-lock proposal isn't intended to address the problem of diagnostics,
> or of info or help buffers for that matter; it's intended only to
> address the issue of displaying docstrings to a developer who's editing
> Elisp code.  Is that correct?

Not correct. font-lock runs in the help buffers, as well as info (if I'm 
not mistaken), so that approach can be used in those too.

Haven't you seen my recent message with the patch for help-mode?