Escaping a string for substitute-command-keys

Escaping a string for substitute-command-keys

[Clément Pit-Claudel]

Hi emacs-devel,

I think I've seen a thread about this in the past, but I can't find it. What's the expected way to escape a string before its fed to substitute-command-keys, so that substitute-command-keys reconstructs the original string?

The reason for this question is that since 2015 Emacs has been calling substitute-command-keys on help-echo messages, which breaks flycheck and flymake (a simple way to reproduce the issue is to enable flymake in a Python buffer containing just xxx and to hover over the xxx with the mouse: Emacs shows a tooltip containing "Undefined name ’xxx’").

So, what's the right way to escape a string to be fed to substitute-command-keys?  Alternatively, is there be a way to disable the transformation for a particular help-echo string, rather than forcing an unnecessary rountrip?  Maybe we could add a text property, so that if a string is tagged with a 'text-quoting-style property that style is used instead of the default one?

Thanks!
Clément.

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Thu, 3 Oct 2019 09:52:31 -0400
> 
> I think I've seen a thread about this in the past, but I can't find it. What's the expected way to escape a string before its fed to substitute-command-keys, so that substitute-command-keys reconstructs the original string?
> 
> The reason for this question is that since 2015 Emacs has been calling substitute-command-keys on help-echo messages, which breaks flycheck and flymake (a simple way to reproduce the issue is to enable flymake in a Python buffer containing just xxx and to hover over the xxx with the mouse: Emacs shows a tooltip containing "Undefined name ’xxx’").
> 
> So, what's the right way to escape a string to be fed to substitute-command-keys?  Alternatively, is there be a way to disable the transformation for a particular help-echo string, rather than forcing an unnecessary rountrip?  Maybe we could add a text property, so that if a string is tagged with a 'text-quoting-style property that style is used instead of the default one?

If I understand what you are looking for, the answer is in the manual:
quote any character with \= (in a Lisp string, that's "\\=", of
course).  See the node "Keys in Documentation" in the ELisp manual.

If this is not what you want, please tell more.

Re: Escaping a string for substitute-command-keys

[Clément Pit-Claudel]

On 2019-10-03 12:31, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Thu, 3 Oct 2019 09:52:31 -0400
>>
>> I think I've seen a thread about this in the past, but I can't find it. What's the expected way to escape a string before its fed to substitute-command-keys, so that substitute-command-keys reconstructs the original string?
>>
>> The reason for this question is that since 2015 Emacs has been calling substitute-command-keys on help-echo messages, which breaks flycheck and flymake (a simple way to reproduce the issue is to enable flymake in a Python buffer containing just xxx and to hover over the xxx with the mouse: Emacs shows a tooltip containing "Undefined name ’xxx’").
>>
>> So, what's the right way to escape a string to be fed to substitute-command-keys?  Alternatively, is there be a way to disable the transformation for a particular help-echo string, rather than forcing an unnecessary rountrip?  Maybe we could add a text property, so that if a string is tagged with a 'text-quoting-style property that style is used instead of the default one?
> 
> If I understand what you are looking for, the answer is in the manual:
> quote any character with \= (in a Lisp string, that's "\\=", of
> course).  See the node "Keys in Documentation" in the ELisp manual.

I saw that part in the manual, but I was looking for a function that would do that.  Is there an easy way to tell what needs escaping, or should I just escape all `, ', and \? (and if so, should we add a function that does that to subr-x.el or somewhere similar?)

Clément.

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Thu, 3 Oct 2019 12:46:50 -0400
> 
> > If I understand what you are looking for, the answer is in the manual:
> > quote any character with \= (in a Lisp string, that's "\\=", of
> > course).  See the node "Keys in Documentation" in the ELisp manual.
> 
> I saw that part in the manual, but I was looking for a function that would do that.  Is there an easy way to tell what needs escaping, or should I just escape all `, ', and \? (and if so, should we add a function that does that to subr-x.el or somewhere similar?)

Is the problem only with quotes?  Or also with other characters?

I don't think we have a general solution, but yes, escaping every
character that makes trouble would be one way.

Re: Escaping a string for substitute-command-keys

[Clément Pit-Claudel]

On 2019-10-03 13:21, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Thu, 3 Oct 2019 12:46:50 -0400
>>
>>> If I understand what you are looking for, the answer is in the manual:
>>> quote any character with \= (in a Lisp string, that's "\\=", of
>>> course).  See the node "Keys in Documentation" in the ELisp manual.
>>
>> I saw that part in the manual, but I was looking for a function that would do that.  Is there an easy way to tell what needs escaping, or should I just escape all `, ', and \? (and if so, should we add a function that does that to subr-x.el or somewhere similar?)
> 
> Is the problem only with quotes?  Or also with other characters?

Any characters: for example, \\[ should not trigger a replacement.  The use case (displaying warnings and errors as overlays on code) does not require any string transformation, just to display what a tool (such as gcc or python) produced.

Ideally, it would be best to be able to turn off that translation entirely, I think.  I see why it is convenient, but it seems wasteful to mangle a string with escapes only for these escapes to be promptly removed right after.

> I don't think we have a general solution, but yes, escaping every
> character that makes trouble would be one way.

That makes sense.  There's an additional problem that I hadn't thought of at first: if I change the help-echo property on my help-echo strings, even though I'll get the right message when mousing over, now things will break for other renderings that do not use substitute-command-keys (in fact, most places that I see in lisp/ do not use substitute-command-keys on help-echo; most importantly, help-at-pt doesn't). 

In other words, at the moment I can either get "The footer should be: (provide 'xyz)…" in the echo area and "The footer should be: (provide ’xyz)…" when hovering; or "The footer should be: (provide \='xyz)…" in the echo area and "The footer should be: (provide 'xyz)…" when hovering; neither of these are good.

Should all uses of help-echo be fixed to call substitute-command-keys, and should all code that sets help-echo and doesn't want substitutions changed to escape quotes and backlashes?

Thanks for your help,
Clément.

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Thu, 3 Oct 2019 14:28:43 -0400
> 
> > Is the problem only with quotes?  Or also with other characters?
> 
> Any characters: for example, \\[ should not trigger a replacement.  The use case (displaying warnings and errors as overlays on code) does not require any string transformation, just to display what a tool (such as gcc or python) produced.

Then why do you use APIs that are meant for keys and quoted strings?
Why not format the message yourself?

> Ideally, it would be best to be able to turn off that translation entirely, I think.  I see why it is convenient, but it seems wasteful to mangle a string with escapes only for these escapes to be promptly removed right after.

Format it yourself, and you have that, no?  Or what am I missing?

> That makes sense.  There's an additional problem that I hadn't thought of at first: if I change the help-echo property on my help-echo strings, even though I'll get the right message when mousing over, now things will break for other renderings that do not use substitute-command-keys (in fact, most places that I see in lisp/ do not use substitute-command-keys on help-echo; most importantly, help-at-pt doesn't). 
> 
> In other words, at the moment I can either get "The footer should be: (provide 'xyz)…" in the echo area and "The footer should be: (provide ’xyz)…" when hovering; or "The footer should be: (provide \='xyz)…" in the echo area and "The footer should be: (provide 'xyz)…" when hovering; neither of these are good.

Then don't use help-echo, use child frames instead.

> Should all uses of help-echo be fixed to call substitute-command-keys, and should all code that sets help-echo and doesn't want substitutions changed to escape quotes and backlashes?

I think you are using the wrong APIs for the job you need to do.
These APIs perform the substitutions for a reason: they were all
written for displaying documentation, not arbitrary strings.

Re: Escaping a string for substitute-command-keys

[Clément Pit-Claudel]

On 2019-10-03 14:54, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Thu, 3 Oct 2019 14:28:43 -0400
>>
>>> Is the problem only with quotes?  Or also with other characters?
>>
>> Any characters: for example, \\[ should not trigger a replacement.  The use case (displaying warnings and errors as overlays on code) does not require any string transformation, just to display what a tool (such as gcc or python) produced.
> 
> Then why do you use APIs that are meant for keys and quoted strings?
> Why not format the message yourself?

Sorry, I do not understand what you mean.  I want to display information when the mouse hovers on a portion of the buffer.  Isn't the proper way to do that to set a help-echo property on the corresponding text?

The manual says nothing about command-key substitutions, it just says "If text has a string as its help-echo property, then when you move the mouse onto that text, Emacs displays that string in the echo area, or in the tooltip window (see Tooltips)."

Really, all that I'm complaining about is this bit of code in keyboard.c:

  if (STRINGP (help) || NILP (help))
    {
      if (!NILP (Vshow_help_function))
	call1 (Vshow_help_function, Fsubstitute_command_keys (help));
      help_echo_showing_p = STRINGP (help);
    }

My question is why Vshow_help_function is called with its arguments passed through Fsubstitute_command_keys.

>> Ideally, it would be best to be able to turn off that translation entirely, I think.  I see why it is convenient, but it seems wasteful to mangle a string with escapes only for these escapes to be promptly removed right after.
> 
> Format it yourself, and you have that, no?  Or what am I missing?

I think I've been miscommunication :/ Sorry for wasting your time. My text is already formatted exactly how it should look.  My problem is that once I apply it using a help-echo property, it's displayed differently in the echo area and when I hover with the mouse: the pop-up from the mouse runs the contents of help-echo through substitute-command-keys, but help-at-point doesn't.

>> Should all uses of help-echo be fixed to call substitute-command-keys, and should all code that sets help-echo and doesn't want substitutions changed to escape quotes and backlashes?
> 
> I think you are using the wrong APIs for the job you need to do.
> These APIs perform the substitutions for a reason: they were all
> written for displaying documentation, not arbitrary strings.

I'm saying that these APIs are inconsistent: sometimes they do substitution, sometimes they don't.  Tooltips have substitution applied, and help-at-point doesn't perform substitutions.  And at least two clients of that API seem to be using it wrong (both Flycheck and Flymake).

> Then don't use help-echo, use child frames instead.

Are you saying that flymake should grow its own implementation of tooltips, just to work around the fact that help-echo strings are passed through substitute-command-keys?

Thanks for your help,
Clément.

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 4 Oct 2019 09:56:56 -0400
> 
> > Then why do you use APIs that are meant for keys and quoted strings?
> > Why not format the message yourself?
> 
> Sorry, I do not understand what you mean.  I want to display information when the mouse hovers on a portion of the buffer.  Isn't the proper way to do that to set a help-echo property on the corresponding text?

help-echo is for displaying documentation, not for displaying
general-purpose text strings.

> The manual says nothing about command-key substitutions, it just says "If text has a string as its help-echo property, then when you move the mouse onto that text, Emacs displays that string in the echo area, or in the tooltip window (see Tooltips)."

And a little ways down from that text you have a cross-reference to
"Help display", which leads to the description of show-help-function,
and that one does say the output is passed through
substitute-command-keys.

> Really, all that I'm complaining about is this bit of code in keyboard.c:
> 
>   if (STRINGP (help) || NILP (help))
>     {
>       if (!NILP (Vshow_help_function))
> 	call1 (Vshow_help_function, Fsubstitute_command_keys (help));
>       help_echo_showing_p = STRINGP (help);
>     }
> 
> My question is why Vshow_help_function is called with its arguments passed through Fsubstitute_command_keys.

See above.

> >> Ideally, it would be best to be able to turn off that translation entirely, I think.  I see why it is convenient, but it seems wasteful to mangle a string with escapes only for these escapes to be promptly removed right after.
> > 
> > Format it yourself, and you have that, no?  Or what am I missing?
> 
> I think I've been miscommunication :/ Sorry for wasting your time.

No reason to apologize or think you were wasting my time.

> > I think you are using the wrong APIs for the job you need to do.
> > These APIs perform the substitutions for a reason: they were all
> > written for displaying documentation, not arbitrary strings.
> 
> I'm saying that these APIs are inconsistent: sometimes they do substitution, sometimes they don't.

Please tell more about these inconsistencies.  Specifically, which
APIs behave inconsistently?

> > Then don't use help-echo, use child frames instead.
> 
> Are you saying that flymake should grow its own implementation of tooltips

Tooltips by themselves don't substitute, you get in the tooltip the
exact text you've provided.  Here's an example I just trried:

  M-: (tooltip-show "foo `bar' \\[kill-buffer]") RET

The tooltip that popped up didn't do any substitutions, I see the same
text I typed.

The substitution happens specifically in generating the help-echo, and
it happens before we pop up the tooltip.

It is possible that we need to provide some additional facilities to
make this stuff more flexible in specific situations, but I think we
should first be on the same page regarding the existing facilities and
what they are intended for.

Re: Escaping a string for substitute-command-keys

[Stefan Monnier]

> That makes sense.  There's an additional problem that I hadn't thought of at
> first: if I change the help-echo property on my help-echo strings, even
> though I'll get the right message when mousing over, now things will break
> for other renderings that do not use substitute-command-keys (in fact, most
> places that I see in lisp/ do not use substitute-command-keys on help-echo;
> most importantly, help-at-pt doesn't). 

That sounds like a bug, indeed: we need to pass the string
through substitute-command-keys either always or never.


        Stefan

Re: Escaping a string for substitute-command-keys

[Clément Pit-Claudel]

On 2019-10-04 10:17, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Fri, 4 Oct 2019 09:56:56 -0400
>>
>>> Then why do you use APIs that are meant for keys and quoted strings?
>>> Why not format the message yourself?
>>
>> Sorry, I do not understand what you mean.  I want to display information when the mouse hovers on a portion of the buffer.  Isn't the proper way to do that to set a help-echo property on the corresponding text?
> 
> help-echo is for displaying documentation, not for displaying
> general-purpose text strings.

Understood.  Thanks for taking the time to explain.

In my experience, help-echo is very convenient (modulo this small issue with substituting command keys) for the general purpose of associating a small amount of help text to a buffer region, because everything is handled automagically for you.  For example, both flycheck and flymake use it to attach errors or warnings to regions of the buffer.

I have briefly reviewed the places where I use it in my code, and indeed it falls into two categories:

* Places where I want to indicate to the user how to interact with the text (really, a button) under the cursor.  There's are many places in Emacs that use it for this purpose, indeed, but most of them actually don't leverage the command substitution facility:
  lisp/help-mode.el:291:4:  'help-echo (purecopy "mouse-2, RET: visit theme file"))
  lisp/vc/vc-git.el:706:11:		       'help-echo "mouse-3: Show stash menu…
  lisp/progmodes/compile.el:114:30:	                           'help-echo "Compiling; mouse-2: Goto Buffer"
  lisp/help-fns.el:1073:12:			       'help-echo "mouse-2, RET: show value")

* Places where I want to add some general information about the text at point.  There are some places like that in Emacs too:
  lisp/net/shr.el:1499:8:      'help-echo title ;; This is the title of an <abbr> tag
  lisp/net/shr.el:1237:4:	 'help-echo (let ((parsed (url-generic-parse-url ;; This shows the URL a links points to
  lisp/progmodes/flymake.el:647:23:      (default-maybe 'help-echo ;; This attaches a compiler message to a buffer region

The motivation for using help-echo in the second case is that works very easily and very smoothly: users see the text in a tooltip if they hover, and in the echo area otherwise, and you don't have to do much on the package side at all.  But the command-key substitution only makes sense in the first case: neither shr nor flymake want quotes transformed in the help-echos that they set, because these are data coming from the outside. 

> And a little ways down from that text you have a cross-reference to
> "Help display", which leads to the description of show-help-function,
> and that one does say the output is passed through
> substitute-command-keys.

Got it, thanks. My bad.
> No reason to apologize or think you were wasting my time.

Thanks for your patience :)

> Please tell more about these inconsistencies.  Specifically, which
> APIs behave inconsistently?

There are two main places that I know of in Emacs that use help-echo: help-at-pt, specifically display-local-help, which displays the contents of the help-echo in the echo area; and show_help_echo, in keyboard.c, which calls show-help-function after running substitute-command-keys.

The inconsistency is that the former displays the help-echo property unmodified, whereas the latter displays it after running substitute-command-keys.

> Tooltips by themselves don't substitute, you get in the tooltip the
> exact text you've provided.  Here's an example I just trried: […]
> The substitution happens specifically in generating the help-echo, and
> it happens before we pop up the tooltip.

Indeed, you're absolutely correct.  The convenience that help-echo provides is that it provides a very easy way to tell Emacs "display a tooltip with this text when the mouse hovers above this overlay".  I think it's not easy to achieve this result otherwise, which makes help-echo a generally useful primitive.

> It is possible that we need to provide some additional facilities to
> make this stuff more flexible in specific situations, but I think we
> should first be on the same page regarding the existing facilities and
> what they are intended for.

I hope the above helps with that. Thanks again for your help.

Clément.

Re: Escaping a string for substitute-command-keys

[Clément Pit-Claudel]

On 2019-10-04 10:17, Eli Zaretskii wrote:
> help-echo is for displaying documentation, not for displaying
> general-purpose text strings.

Btw, it was surprising to me to see that very little code in Emacs actually takes advantage of this command-substitution facility in help-echo; most help-echo strings that mention commands actually hardcode the default binding.

Also related is the fact that help-echo allows kinds of values: strings or functions.  In Flycheck's case (the one I know the best), it would be enough to say that help-echo is only passed through substitute-command-keys if it's a string, but not if it's a function (the reasoning being that it's easy to run the substitution inside the function, if needed).  Of course, shr and Flymake would then need to switch to using functions instead of strings, as well.

Clément.

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Sat, 5 Oct 2019 00:06:43 -0400
> 
> On 2019-10-04 10:17, Eli Zaretskii wrote:
> > help-echo is for displaying documentation, not for displaying
> > general-purpose text strings.
> 
> Btw, it was surprising to me to see that very little code in Emacs actually takes advantage of this command-substitution facility in help-echo; most help-echo strings that mention commands actually hardcode the default binding.

Most probably because the call to substitute-command-keys there is a
relatively recent addition.

> Also related is the fact that help-echo allows kinds of values: strings or functions.  In Flycheck's case (the one I know the best), it would be enough to say that help-echo is only passed through substitute-command-keys if it's a string, but not if it's a function (the reasoning being that it's easy to run the substitution inside the function, if needed).  Of course, shr and Flymake would then need to switch to using functions instead of strings, as well.

I'm not sure whether this is a feature or a bug.

I guess we need to add some feature to control whether help-echo gets
run through substitute-command-keys, or add an alternative to
help-echo that works exactly like it, but without substitutions.
Patches welcome.

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Sat, 5 Oct 2019 00:03:01 -0400
> 
> > Please tell more about these inconsistencies.  Specifically, which
> > APIs behave inconsistently?
> 
> There are two main places that I know of in Emacs that use help-echo: help-at-pt, specifically display-local-help, which displays the contents of the help-echo in the echo area; and show_help_echo, in keyboard.c, which calls show-help-function after running substitute-command-keys.
> 
> The inconsistency is that the former displays the help-echo property unmodified, whereas the latter displays it after running substitute-command-keys.

Thanks.

I see that the kbd-help property and all the facilities of
help-at-pt.el are not documented at all in the ELisp manual, which I
think is a documentation bug, because in some contexts they override
the similar help-echo facilities.  I also think help-at-pt.el should
indeed pass help-echo through substitute-command-keys, at least by
default.

Having read help-at-pt.el, I now don't understand why we need that
package, since help-echo can be easily displayed in the echo area as
well.  But that ship has sailed, so even if we deprecate this package
now, we should first make it consistent with help-echo.

Would you please file a bug report about these issues?

Thanks.

Re: Escaping a string for substitute-command-keys

[Clément Pit-Claudel]

On 2019-10-05 03:12, Eli Zaretskii wrote:
>> Btw, it was surprising to me to see that very little code in Emacs actually takes advantage of this command-substitution facility in help-echo; most help-echo strings that mention commands actually hardcode the default binding.
> 
> Most probably because the call to substitute-command-keys there is a
> relatively recent addition.

Makes sense.

> I guess we need to add some feature to control whether help-echo gets
> run through substitute-command-keys, or add an alternative to
> help-echo that works exactly like it, but without substitutions.
> Patches welcome.

Sounds good, and I can work on the patch.  I think I like the first solution better, if we can come up with something elegant: the second case will force us to define priorities, and besides we already have kbd-help in addition to help-echo.

What about the following: before performing a conversion, substitute-command-keys could check a property of the string that it receives, say help-echo-conversion, and if that property is nil it would not perform the conversion.

Alternatively, there's the option of performing conversion only on those 'help-echo' properties that are string, and leaving (the output of) the ones that are functions untouched.

What do you think?
Clément.

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Sat, 5 Oct 2019 04:04:34 -0400
> 
> > I guess we need to add some feature to control whether help-echo gets
> > run through substitute-command-keys, or add an alternative to
> > help-echo that works exactly like it, but without substitutions.
> > Patches welcome.
> 
> Sounds good, and I can work on the patch.  I think I like the first solution better, if we can come up with something elegant: the second case will force us to define priorities, and besides we already have kbd-help in addition to help-echo.
> 
> What about the following: before performing a conversion, substitute-command-keys could check a property of the string that it receives, say help-echo-conversion, and if that property is nil it would not perform the conversion.

I'd prefer a property like help-echo-inhibit-substitution that is
non-nil, to inhibit the call to substitute-command-keys.  A value of
nil is easy to confuse with no property at all.

> Alternatively, there's the option of performing conversion only on those 'help-echo' properties that are string, and leaving (the output of) the ones that are functions untouched.

I think they all should be converted, otherwise we have an
inconsistency of the kind you mentioned.

Re: Escaping a string for substitute-command-keys

[Clément Pit-Claudel]

On 2019-10-05 04:13, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Sat, 5 Oct 2019 04:04:34 -0400
>>
>>> I guess we need to add some feature to control whether help-echo gets
>>> run through substitute-command-keys, or add an alternative to
>>> help-echo that works exactly like it, but without substitutions.
>>> Patches welcome.
>>
>> Sounds good, and I can work on the patch.  I think I like the first solution better, if we can come up with something elegant: the second case will force us to define priorities, and besides we already have kbd-help in addition to help-echo.
>>
>> What about the following: before performing a conversion, substitute-command-keys could check a property of the string that it receives, say help-echo-conversion, and if that property is nil it would not perform the conversion.
> 
> I'd prefer a property like help-echo-inhibit-substitution that is
> non-nil, to inhibit the call to substitute-command-keys.  A value of
> nil is easy to confuse with no property at all.

OK, I will work on a patch.  I expect I will have time to work on it in a few weeks from now, but not sooner, unfortunately.

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Sat, 5 Oct 2019 04:24:28 -0400
> 
> > I'd prefer a property like help-echo-inhibit-substitution that is
> > non-nil, to inhibit the call to substitute-command-keys.  A value of
> > nil is easy to confuse with no property at all.
> 
> OK, I will work on a patch.  I expect I will have time to work on it in a few weeks from now, but not sooner, unfortunately.

There's no rush, but if you want this to be in Emacs 27, then I hope
"few weeks" is indeed a few, because I intend to cut the emacs-27
branch "in a few weeks" ;-)

Re: Escaping a string for substitute-command-keys

[Stefan Monnier]

> Having read help-at-pt.el, I now don't understand why we need that
> package, since help-echo can be easily displayed in the echo area as
> well.

The issue isn't "tooltip frame vs echo area" but "mouse vs keyboard", IIRC.


        Stefan

Re: Escaping a string for substitute-command-keys

[Basil L. Contovounesios]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> That makes sense.  There's an additional problem that I hadn't thought of at
>> first: if I change the help-echo property on my help-echo strings, even
>> though I'll get the right message when mousing over, now things will break
>> for other renderings that do not use substitute-command-keys (in fact, most
>> places that I see in lisp/ do not use substitute-command-keys on help-echo;
>> most importantly, help-at-pt doesn't). 
>
> That sounds like a bug, indeed: we need to pass the string
> through substitute-command-keys either always or never.

Another source of inconsistency is {forward,backward}-button, which
display help-echo values in the echo area when called interactively.
Currently, they perform no command name substitution.  Should they?

Thanks,

-- 
Basil

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Clément Pit-Claudel <cpitclaudel@gmail.com>,
>   emacs-devel@gnu.org
> Date: Sat, 05 Oct 2019 11:05:06 -0400
> 
> > Having read help-at-pt.el, I now don't understand why we need that
> > package, since help-echo can be easily displayed in the echo area as
> > well.
> 
> The issue isn't "tooltip frame vs echo area" but "mouse vs keyboard", IIRC.

Not entirely, because tooltips can be shown due to keyboard commands,
e.g. in TTY menus.

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: Clément Pit-Claudel <cpitclaudel@gmail.com>,  Eli
>  Zaretskii
>  <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sat, 05 Oct 2019 16:40:35 +0100
> 
> Another source of inconsistency is {forward,backward}-button, which
> display help-echo values in the echo area when called interactively.
> Currently, they perform no command name substitution.  Should they?

IMO, all displays of help-echo should look the same way.

Re: Escaping a string for substitute-command-keys

[Basil L. Contovounesios]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Basil L. Contovounesios" <contovob@tcd.ie>
>> Cc: Clément Pit-Claudel <cpitclaudel@gmail.com>,  Eli
>>  Zaretskii
>>  <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Sat, 05 Oct 2019 16:40:35 +0100
>> 
>> Another source of inconsistency is {forward,backward}-button, which
>> display help-echo values in the echo area when called interactively.
>> Currently, they perform no command name substitution.  Should they?
>
> IMO, all displays of help-echo should look the same way.

So, would something like the following be acceptable?  Clément, do you
envision this working against any of your planned changes?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Substitute-command-keys-in-button-help-echo-values.patch --]
[-- Type: text/x-diff, Size: 5967 bytes --]

From 745a68d371d20fa3a98e3ea0e270e477d85a11d0 Mon Sep 17 00:00:00 2001
From: "Basil L. Contovounesios" <contovob@tcd.ie>
Date: Sat, 5 Oct 2019 19:51:38 +0100
Subject: [PATCH] Substitute command keys in button help-echo values

For consistency with tooltips displayed via show-help-function, make
forward-button also substitute command keys in the help-echo values
it displays in the echo area.
For discussion, see the following thread:
https://lists.gnu.org/archive/html/emacs-devel/2019-10/msg00090.html
* lisp/button.el (button--help-echo): Pass resulting string through
substitute-command-keys for consistency with show-help-function.
* test/lisp/button-tests.el (button-tests--map): New test keymap.
(button--help-echo-string, button--help-echo-form)
(button--help-echo-function): Use it to test command key
substitution in help-echo strings.
---
 lisp/button.el            | 18 ++++++++++++------
 test/lisp/button-tests.el | 35 ++++++++++++++++++++++-------------
 2 files changed, 34 insertions(+), 19 deletions(-)

diff --git a/lisp/button.el b/lisp/button.el
index 04e77ca904..8d073aaceb 100644
--- a/lisp/button.el
+++ b/lisp/button.el
@@ -468,12 +468,18 @@ push-button
 	t))))
 
 (defun button--help-echo (button)
-  "Evaluate BUTTON's `help-echo' property and return its value."
-  (let ((help (button-get button 'help-echo)))
-    (if (functionp help)
-        (let ((obj (if (overlayp button) button (current-buffer))))
-          (funcall help (selected-window) obj (button-start button)))
-      (eval help lexical-binding))))
+  "Evaluate BUTTON's `help-echo' property and return its value.
+If the result is non-nil, pass it through
+`substitute-command-keys' before returning it, as is done for
+`show-help-function'."
+  (let* ((help (button-get button 'help-echo))
+         (help (if (functionp help)
+                   (funcall help
+                            (selected-window)
+                            (if (overlayp button) button (current-buffer))
+                            (button-start button))
+                 (eval help lexical-binding))))
+    (and help (substitute-command-keys help))))
 
 (defun forward-button (n &optional wrap display-message no-error)
   "Move to the Nth next button, or Nth previous button if N is negative.
diff --git a/test/lisp/button-tests.el b/test/lisp/button-tests.el
index 44a7ea6f6e..73e3a764b3 100644
--- a/test/lisp/button-tests.el
+++ b/test/lisp/button-tests.el
@@ -21,6 +21,12 @@
 
 (require 'ert)
 
+(defvar button-tests--map
+  (let ((map (make-sparse-keymap)))
+    (define-key map "x" #'ignore)
+    map)
+  "Keymap for testing command substitution.")
+
 (ert-deftest button-at ()
   "Test `button-at' behavior."
   (with-temp-buffer
@@ -41,11 +47,13 @@ button--help-echo-string
   "Test `button--help-echo' with strings."
   (with-temp-buffer
     ;; Text property buttons.
-    (let ((button (insert-text-button "text" 'help-echo "text help")))
-      (should (equal (button--help-echo button) "text help")))
+    (let ((button (insert-text-button
+                   "text" 'help-echo "text: \\<button-tests--map>\\[ignore]")))
+      (should (equal (button--help-echo button) "text: x")))
     ;; Overlay buttons.
-    (let ((button (insert-button "overlay" 'help-echo "overlay help")))
-      (should (equal (button--help-echo button) "overlay help")))))
+    (let ((button (insert-button "overlay" 'help-echo
+                                 "overlay: \\<button-tests--map>\\[ignore]")))
+      (should (equal (button--help-echo button) "overlay: x")))))
 
 (ert-deftest button--help-echo-form ()
   "Test `button--help-echo' with forms."
@@ -55,16 +63,17 @@ button--help-echo-form
            (form   `(funcall (let ((,help "lexical form"))
                                (lambda () ,help))))
            (button (insert-text-button "text" 'help-echo form)))
-      (set help "dynamic form")
-      (should (equal (button--help-echo button) "dynamic form")))
+      (set help "dynamic: \\<button-tests--map>\\[ignore]")
+      (should (equal (button--help-echo button) "dynamic: x")))
     ;; Test overlay buttons with lexical scoping.
     (setq lexical-binding t)
     (let* ((help   (make-symbol "help"))
-           (form   `(funcall (let ((,help "lexical form"))
-                               (lambda () ,help))))
+           (form   `(funcall
+                     (let ((,help "lexical: \\<button-tests--map>\\[ignore]"))
+                       (lambda () ,help))))
            (button (insert-button "overlay" 'help-echo form)))
       (set help "dynamic form")
-      (should (equal (button--help-echo button) "lexical form")))))
+      (should (equal (button--help-echo button) "lexical: x")))))
 
 (ert-deftest button--help-echo-function ()
   "Test `button--help-echo' with functions."
@@ -77,9 +86,9 @@ button--help-echo-function
                      (should (eq win owin))
                      (should (eq obj obuf))
                      (should (=  pos opos))
-                     "text function"))
+                     "text: \\<button-tests--map>\\[ignore]"))
            (button (insert-text-button "text" 'help-echo help)))
-      (should (equal (button--help-echo button) "text function"))
+      (should (equal (button--help-echo button) "text: x"))
       ;; Overlay buttons.
       (setq help (lambda (win obj pos)
                    (should (eq win owin))
@@ -88,9 +97,9 @@ button--help-echo-function
                    (should (eq (overlay-buffer obj) obuf))
                    (should (= (overlay-start obj) opos))
                    (should (= pos opos))
-                   "overlay function"))
+                   "overlay: \\<button-tests--map>\\[ignore]"))
       (setq opos (point))
       (setq button (insert-button "overlay" 'help-echo help))
-      (should (equal (button--help-echo button) "overlay function")))))
+      (should (equal (button--help-echo button) "overlay: x")))))
 
 ;;; button-tests.el ends here
-- 
2.23.0


[-- Attachment #3: Type: text/plain, Size: 20 bytes --]


Thanks,

-- 
Basil

Re: Escaping a string for substitute-command-keys

[Clément Pit-Claudel]

On 2019-10-05 15:53, Basil L. Contovounesios wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> From: "Basil L. Contovounesios" <contovob@tcd.ie>
>>> Cc: Clément Pit-Claudel <cpitclaudel@gmail.com>,  Eli
>>>  Zaretskii
>>>  <eliz@gnu.org>,  emacs-devel@gnu.org
>>> Date: Sat, 05 Oct 2019 16:40:35 +0100
>>>
>>> Another source of inconsistency is {forward,backward}-button, which
>>> display help-echo values in the echo area when called interactively.
>>> Currently, they perform no command name substitution.  Should they?
>>
>> IMO, all displays of help-echo should look the same way.
> 
> So, would something like the following be acceptable?  Clément, do you
> envision this working against any of your planned changes?

This looks good to me.  At some point "someone" should do a pass and replace most hardcoded commands (such as mouse-3) with commands to be substituted.

Also, do you want to include the help-at-point update in your patch?

Clément.

Re: Escaping a string for substitute-command-keys

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Date: Sat, 05 Oct 2019 20:53:48 +0100
> Cc: cpitclaudel@gmail.com, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > IMO, all displays of help-echo should look the same way.
> 
> So, would something like the following be acceptable?

This is fine by me.  I'd only announce this in NEWS.