bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Gabriele Nicolardi]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

Hi,
I have the following Emacs Lisp code:

|(progn (y-or-n-p "Test: ") (let ((search-spaces-regexp 
"\\(?:\\n?[\s\t]+\\|\n\\)?")) (y-or-n-p "Test: "))) |

The first prompt from the |y-or-n-p| function appears as expected:

|Test: (y or n) |

However, the second prompt appears differently:

|Test: (‘y’ or ‘n’) |

I’m trying to understand why the second prompt format changes. What 
causes this inconsistency in the |y-or-n-p| prompt?

I suspect it might be related to the |search-spaces-regexp| variable or 
how Emacs handles interactive prompts, but I’m not sure. Any insights or 
explanations would be greatly appreciated!

Best regards,

Gabriele Nicolardi

​

[-- Attachment #2: Type: text/html, Size: 4865 bytes --]

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Eli Zaretskii]

> Date: Sat, 8 Jun 2024 09:03:37 +0200
> From: Gabriele Nicolardi <gabriele@medialab.sissa.it>
> 
> I have the following Emacs Lisp code:
> 
> (progn
>   (y-or-n-p "Test: ")
>   (let ((search-spaces-regexp "\\(?:\\n?[\s\t]+\\|\n\\)?"))
>     (y-or-n-p "Test: ")))
> 
> The first prompt from the y-or-n-p function appears as expected:
> 
> Test: (y or n)
> 
> However, the second prompt appears differently:
> 
> Test: (‘y’ or ‘n’)
> 
> I’m trying to understand why the second prompt format changes. What causes this inconsistency in the
> y-or-n-p prompt?
> 
> I suspect it might be related to the search-spaces-regexp variable or how Emacs handles interactive
> prompts, but I’m not sure. Any insights or explanations would be greatly appreciated!

Stefan, can you please look into this?  It sounds like some issue with
substitute-command-keys:

  (substitute-command-keys "(\\`y' or \\`n') ")
   => #("(y or n) " 1 2 (font-lock-face help-key-binding face help-key-binding) 6 7 (font-lock-face help-key-binding face help-key-binding))

But

  (let ((search-spaces-regexp "\\(?:\\n?[\s\t]+\\|\n\\)?"))
    (substitute-command-keys "(\\`y' or \\`n') "))
   => "(\\‘y’ or \\‘n’) "

I actually don't understand why we use \\`y' and \\`n' in y-or-n-p.
Why those backslashes, and not just `y' and `n'?  That's your change
in commit a36ecc408a.  If I remove the backslashes, the results are
identical whether or not search-spaces-regexp is let-bound.

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Stephen Berman via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Sat, 08 Jun 2024 11:24:50 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> Date: Sat, 8 Jun 2024 09:03:37 +0200
>> From: Gabriele Nicolardi <gabriele@medialab.sissa.it>
>> 
>> I have the following Emacs Lisp code:
>> 
>> (progn
>>   (y-or-n-p "Test: ")
>>   (let ((search-spaces-regexp "\\(?:\\n?[\s\t]+\\|\n\\)?"))
>>     (y-or-n-p "Test: ")))
>> 
>> The first prompt from the y-or-n-p function appears as expected:
>> 
>> Test: (y or n)
>> 
>> However, the second prompt appears differently:
>> 
>> Test: (‘y’ or ‘n’)
>> 
>> I’m trying to understand why the second prompt format changes. What causes this inconsistency in the
>> y-or-n-p prompt?
>> 
>> I suspect it might be related to the search-spaces-regexp variable or how Emacs handles interactive
>> prompts, but I’m not sure. Any insights or explanations would be greatly appreciated!
>
> Stefan, can you please look into this?  It sounds like some issue with
> substitute-command-keys:
>
>   (substitute-command-keys "(\\`y' or \\`n') ")
>    => #("(y or n) " 1 2 (font-lock-face help-key-binding face help-key-binding) 6 7 (font-lock-face help-key-binding face help-key-binding))
>
> But
>
>   (let ((search-spaces-regexp "\\(?:\\n?[\s\t]+\\|\n\\)?"))
>     (substitute-command-keys "(\\`y' or \\`n') "))
>    => "(\\‘y’ or \\‘n’) "
>
> I actually don't understand why we use \\`y' and \\`n' in y-or-n-p.
> Why those backslashes, and not just `y' and `n'?  That's your change
> in commit a36ecc408a.  If I remove the backslashes, the results are
> identical whether or not search-spaces-regexp is let-bound.

Removing the final '?' in the regexp, i.e.

(let ((search-spaces-regexp "\\(?:\\n?[\s\t]+\\|\n\\)"))
  (y-or-n-p "Test: "))

results in the second prompt appearing like the first one.  Likewise
with '*', but not with '+':

(let ((search-spaces-regexp " ?"))
  (y-or-n-p "Test: "))
=> Test: (\‘y’ or \‘n’)

(let ((search-spaces-regexp " *"))
  (y-or-n-p "Test: "))
=> Test: (\‘y’ or \‘n’)

(let ((search-spaces-regexp " +"))
  (y-or-n-p "Test: "))
=> Test: (y or n)

(let ((search-spaces-regexp " "))
  (y-or-n-p "Test: "))
=> Test: (y or n)

Steve Berman

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Stephen Berman via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Sat, 08 Jun 2024 11:20:14 +0200 Stephen Berman <stephen.berman@gmx.net> wrote:

> On Sat, 08 Jun 2024 11:24:50 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
>
>>> Date: Sat, 8 Jun 2024 09:03:37 +0200
>>> From: Gabriele Nicolardi <gabriele@medialab.sissa.it>
>>> 
>>> I have the following Emacs Lisp code:
>>> 
>>> (progn
>>>   (y-or-n-p "Test: ")
>>>   (let ((search-spaces-regexp "\\(?:\\n?[\s\t]+\\|\n\\)?"))
>>>     (y-or-n-p "Test: ")))
>>> 
>>> The first prompt from the y-or-n-p function appears as expected:
>>> 
>>> Test: (y or n)
>>> 
>>> However, the second prompt appears differently:
>>> 
>>> Test: (‘y’ or ‘n’)
>>> 
>>> I’m trying to understand why the second prompt format changes. What causes this inconsistency in the
>>> y-or-n-p prompt?
>>> 
>>> I suspect it might be related to the search-spaces-regexp variable or how Emacs handles interactive
>>> prompts, but I’m not sure. Any insights or explanations would be greatly appreciated!
>>
>> Stefan, can you please look into this?  It sounds like some issue with
>> substitute-command-keys:
>>
>>   (substitute-command-keys "(\\`y' or \\`n') ")
>>    => #("(y or n) " 1 2 (font-lock-face help-key-binding face help-key-binding) 6 7 (font-lock-face help-key-binding face help-key-binding))
>>
>> But
>>
>>   (let ((search-spaces-regexp "\\(?:\\n?[\s\t]+\\|\n\\)?"))
>>     (substitute-command-keys "(\\`y' or \\`n') "))
>>    => "(\\‘y’ or \\‘n’) "
>>
>> I actually don't understand why we use \\`y' and \\`n' in y-or-n-p.
>> Why those backslashes, and not just `y' and `n'?  That's your change
>> in commit a36ecc408a.  If I remove the backslashes, the results are
>> identical whether or not search-spaces-regexp is let-bound.

Without the backslashes the cond-clause in substitute-command-keys
handling sequences starting with "\" is skipped, so "y" and "n" do not
get the help-key-binding face property.

> Removing the final '?' in the regexp, i.e.
>
> (let ((search-spaces-regexp "\\(?:\\n?[\s\t]+\\|\n\\)"))
>   (y-or-n-p "Test: "))
>
> results in the second prompt appearing like the first one.  Likewise
> with '*', but not with '+':
>
> (let ((search-spaces-regexp " ?"))
>   (y-or-n-p "Test: "))
> => Test: (\‘y’ or \‘n’)
>
> (let ((search-spaces-regexp " *"))
>   (y-or-n-p "Test: "))
> => Test: (\‘y’ or \‘n’)
>
> (let ((search-spaces-regexp " +"))
>   (y-or-n-p "Test: "))
> => Test: (y or n)
>
> (let ((search-spaces-regexp " "))
>   (y-or-n-p "Test: "))
> => Test: (y or n)

Stepping through substitute-command-keys in Edebug, I see that when the
regexp ends in '?' or '*' the sexp (key-valid-p k) in
substitute-command-keys returns nil for k set to "y" and then to "n", so
these strings do not get the help-key-binding face property and "(\\`y'
or \\`n') " is returned to y-or-n-p unaltered.  When the regexp does not
end in '?' or '*', (key-valid-p k) returns t for "y" and "n" and these
strings get propertized.

Stepping through key-valid-p, I see that when the regexp ends in '?' or
'*' the sexp (split-string keys " ") returns (#1="" "y" #1#) for keys
set to "y", and key-valid-p loops over this lists, and the first element
"" is an invalid key.  When the regexp does not end in '?' or '*' the
split-string sexp in key-valid-p returns ("y"), and "y" is valid.

And stepping through split-string, I see that when the regexp ends in
'?' or '*', the invocation of string-match in the while-loop with args
REGEXP set to " ", STRING set to "y" and START set to 0 returns 0, which
results in "" being pushed onto the list both before and after "y",
hence returning (#1="" "y" #1#).  When the regexp does not end in '?' or
'*', the string-match invocation returns nil and only "y" is pushed onto
the list.

Steve Berman

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: Gabriele Nicolardi <gabriele@medialab.sissa.it>,  Stefan Kangas
>  <stefankangas@gmail.com>,  71429@debbugs.gnu.org
> Date: Sat, 08 Jun 2024 15:59:11 +0200
> 
> >> I actually don't understand why we use \\`y' and \\`n' in y-or-n-p.
> >> Why those backslashes, and not just `y' and `n'?  That's your change
> >> in commit a36ecc408a.  If I remove the backslashes, the results are
> >> identical whether or not search-spaces-regexp is let-bound.
> 
> Without the backslashes the cond-clause in substitute-command-keys
> handling sequences starting with "\" is skipped, so "y" and "n" do not
> get the help-key-binding face property.

This should be explained in a comment in y-or-n-p.

> Stepping through substitute-command-keys in Edebug, I see that when the
> regexp ends in '?' or '*' the sexp (key-valid-p k) in
> substitute-command-keys returns nil for k set to "y" and then to "n", so
> these strings do not get the help-key-binding face property and "(\\`y'
> or \\`n') " is returned to y-or-n-p unaltered.  When the regexp does not
> end in '?' or '*', (key-valid-p k) returns t for "y" and "n" and these
> strings get propertized.
> 
> Stepping through key-valid-p, I see that when the regexp ends in '?' or
> '*' the sexp (split-string keys " ") returns (#1="" "y" #1#) for keys
> set to "y", and key-valid-p loops over this lists, and the first element
> "" is an invalid key.  When the regexp does not end in '?' or '*' the
> split-string sexp in key-valid-p returns ("y"), and "y" is valid.

Thanks.  To me, this means that key-valid-p should bind
search-spaces-regexp to nil, because otherwise the value will subvert
its contract.  Do you agree?

I added Stefan Monnier to the discussion in the hope he would have
comments to this.

> And stepping through split-string, I see that when the regexp ends in
> '?' or '*', the invocation of string-match in the while-loop with args
> REGEXP set to " ", STRING set to "y" and START set to 0 returns 0, which
> results in "" being pushed onto the list both before and after "y",
> hence returning (#1="" "y" #1#).  When the regexp does not end in '?' or
> '*', the string-match invocation returns nil and only "y" is pushed onto
> the list.

We should ad to split-string's doc string the fact that
search-spaces-regexp affects its results when SEPARATORS includes
whitespace.

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Drew Adams via Bug reports for GNU Emacs, the Swiss army knife of text editors]

FWIW, the question/problem was also posed here:

https://emacs.stackexchange.com/q/81433/105

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Thanks.  To me, this means that key-valid-p should bind
> search-spaces-regexp to nil, because otherwise the value will subvert
> its contract.  Do you agree?

I kind of agree, but I wonder why it would be non-nil at this point.
AFAIK `search-spaces-regexp` is meant for interactive searches, so
let-binding it around code like `substitute-command-keys`,
`split-string`, or `key-valid-p` sounds like a bug.


        Stefan

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Stephen Berman via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Sat, 08 Jun 2024 17:58:17 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Cc: Gabriele Nicolardi <gabriele@medialab.sissa.it>,  Stefan Kangas
>>  <stefankangas@gmail.com>,  71429@debbugs.gnu.org
>> Date: Sat, 08 Jun 2024 15:59:11 +0200
>>
>> >> I actually don't understand why we use \\`y' and \\`n' in y-or-n-p.
>> >> Why those backslashes, and not just `y' and `n'?  That's your change
>> >> in commit a36ecc408a.  If I remove the backslashes, the results are
>> >> identical whether or not search-spaces-regexp is let-bound.
>>
>> Without the backslashes the cond-clause in substitute-command-keys
>> handling sequences starting with "\" is skipped, so "y" and "n" do not
>> get the help-key-binding face property.
>
> This should be explained in a comment in y-or-n-p.

Since this effect of using a backslash is part of what
substitute-command-keys does (and it's commented there: ";; 1C. \`f' is
replaced with a fontified f."), would adding a comment to y-or-n-p be an
exception or would all callers of substitute-command-keys that use this
handling of the backslash also need to have such a comment?

>> Stepping through substitute-command-keys in Edebug, I see that when the
>> regexp ends in '?' or '*' the sexp (key-valid-p k) in
>> substitute-command-keys returns nil for k set to "y" and then to "n", so
>> these strings do not get the help-key-binding face property and "(\\`y'
>> or \\`n') " is returned to y-or-n-p unaltered.  When the regexp does not
>> end in '?' or '*', (key-valid-p k) returns t for "y" and "n" and these
>> strings get propertized.
>>
>> Stepping through key-valid-p, I see that when the regexp ends in '?' or
>> '*' the sexp (split-string keys " ") returns (#1="" "y" #1#) for keys
>> set to "y", and key-valid-p loops over this lists, and the first element
>> "" is an invalid key.  When the regexp does not end in '?' or '*' the
>> split-string sexp in key-valid-p returns ("y"), and "y" is valid.
>
> Thanks.  To me, this means that key-valid-p should bind
> search-spaces-regexp to nil, because otherwise the value will subvert
> its contract.  Do you agree?

The value can break key-valid-p, e.g. by only optionally matching
whitespace.  Maybe that's reason enough to have key-valid-p bind it to
nil.  The OP's use case (mentioned the the stackexchange thread Drew
referred to) seems legitimate, but maybe it can be achieved without
changing search-spaces-regexp.

Steve Berman

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Stephen Berman <stephen.berman@gmx.net>,  gabriele@medialab.sissa.it,
>   stefankangas@gmail.com,  71429@debbugs.gnu.org
> Date: Sat, 08 Jun 2024 11:38:53 -0400
> 
> > Thanks.  To me, this means that key-valid-p should bind
> > search-spaces-regexp to nil, because otherwise the value will subvert
> > its contract.  Do you agree?
> 
> I kind of agree, but I wonder why it would be non-nil at this point.
> AFAIK `search-spaces-regexp` is meant for interactive searches, so
> let-binding it around code like `substitute-command-keys`,
> `split-string`, or `key-valid-p` sounds like a bug.

You are saying that our implementation of search-spaces-regexp is
incorrect?  Because search.c uses its value regardless of whether it
was invoked interactively or not.

Or maybe you are saying that a Lisp program that binds
search-spaces-regexp has a bug, because Lisp code is not supposed to
bind that?

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  gabriele@medialab.sissa.it,
>   stefankangas@gmail.com,  71429@debbugs.gnu.org
> Date: Sat, 08 Jun 2024 17:47:04 +0200
> 
> On Sat, 08 Jun 2024 17:58:17 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >> Without the backslashes the cond-clause in substitute-command-keys
> >> handling sequences starting with "\" is skipped, so "y" and "n" do not
> >> get the help-key-binding face property.
> >
> > This should be explained in a comment in y-or-n-p.
> 
> Since this effect of using a backslash is part of what
> substitute-command-keys does (and it's commented there: ";; 1C. \`f' is
> replaced with a fontified f."), would adding a comment to y-or-n-p be an
> exception or would all callers of substitute-command-keys that use this
> handling of the backslash also need to have such a comment?

Not necessarily all the callers, but some definitely.  This is an
unusual convention, which is mentioned in the doc string of
substitute-command-keys, but not in the ELisp manual, and is used in
our tree not too many times.  The fact that I didn't know about it
should already speak volumes.  And in this case, how can the reader
guess that `y' and `n' are meant as key bindings, not as simple
characters?

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Or maybe you are saying that a Lisp program that binds
> search-spaces-regexp has a bug, because Lisp code is not supposed to
> bind that?

That, yes. Binding it should be done carefully only at those places
where the contained Lisp code is performing a search whose regexp was
built from interactive user input.

If you let bind that var and then call arbitrary code, I'd
expect breakage.


        Stefan

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: stephen.berman@gmx.net,  gabriele@medialab.sissa.it,
>   stefankangas@gmail.com,  71429@debbugs.gnu.org
> Date: Sat, 08 Jun 2024 13:41:14 -0400
> 
> > Or maybe you are saying that a Lisp program that binds
> > search-spaces-regexp has a bug, because Lisp code is not supposed to
> > bind that?
> 
> That, yes. Binding it should be done carefully only at those places
> where the contained Lisp code is performing a search whose regexp was
> built from interactive user input.
> 
> If you let bind that var and then call arbitrary code, I'd
> expect breakage.

OK, thanks.  So I now added the above caveat to the ELisp manual, and
I'm closing this bug.

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> Since this effect of using a backslash is part of what
>> substitute-command-keys does (and it's commented there: ";; 1C. \`f' is
>> replaced with a fontified f."), would adding a comment to y-or-n-p be an
>> exception or would all callers of substitute-command-keys that use this
>> handling of the backslash also need to have such a comment?
>
> Not necessarily all the callers, but some definitely.  This is an
> unusual convention, which is mentioned in the doc string of
> substitute-command-keys, but not in the ELisp manual,

It is documented in (info "(elisp) Keys in Documentation").

See commit 1aef1a6673bc29784effe10d2e01e62b49c0112c and Bug#50804.

bug#71429: Inconsistent y-or-n-p prompt behavior in Emacs Lisp

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> OK, thanks.  So I now added the above caveat to the ELisp manual, and
> I'm closing this bug.

Actually closing.