Escape syntax in docstrings

Escape syntax in docstrings

[Chen Zhaoyang]

Dear friends,

Since when did emacs become stricter when enforcing the escape syntax in
docstrings? On 27.1 (built by debian), this function evals to no error:

(defun slash-in-docstring ()
 "\x -> x"       
 nil)

but on 30.0.50, the above lisp function will get emacs to complain
`(error "Invalid escape character syntax")`. I believe this is a recent
change. I didn't build from the master branch for only about a week; I
recompiled my emacs yesterday from master and found out that
the lisp code ships with the HOL theorem prover broke on me in the
following two functions (their docstrings, rather, the functions
themselves are fine):

(defun hol-input-compose (f g)
  "\x -> concatMap F (G x)"
  (lexical-let ((f1 f) (g1 g))
    (lambda (x) (hol-input-concat-map f1 (funcall g1 x)))))

(defun hol-input-or (f g)
  "\x -> F x ++ G x"
  (lexical-let ((f1 f) (g1 g))
    (lambda (x) (append (funcall f1 x) (funcall g1 x)))))

The fix is trivial of course
(https://github.com/HOL-Theorem-Prover/HOL/commit/e9e9506209e82b8037b8a4066b7fd672e961c08a),
but going through the recent logs I don't quite find commits explicitly
about escape character in docstrings (I think commit b8e7061232f `Remove
recursion from character escape handling in reader` probably has
something to do with it, but I am not so literate in C).

I am aware that the agda project also uses slash to represent lambdas in
their elisp code
(https://github.com/agda/agda/blob/35bcdbc04e47e49a436b5a097312a672e0ad0074/src/data/emacs-mode/agda-input.el#L56,
courtesy of <ski> on #emacs):

(defun agda-input-compose (f g)
  "\x -> concatMap F (G x)"
    (lambda (x) (agda-input-concat-map f (funcall g x))))

Once this change lands in the next release, this lisp function will certainly break.

Can we document this change of behavior? 

-- 
Chen Zhaoyang

Re: Escape syntax in docstrings

[Mattias Engdegård]

23 mars 2023 kl. 21.46 skrev Chen Zhaoyang <chenzhauyang@gmail.com>:

> Since when did emacs become stricter when enforcing the escape syntax in
> docstrings? On 27.1 (built by debian), this function evals to no error:
> 
> (defun slash-in-docstring ()
> "\x -> x"

Yes, this was changed recently, but I feel vindicated by your report: "\x" never meant what the author thought it did (or at least what I think the author thought it did): it is a hex escape without digits and used to be equivalent to "\000", yielding the control character NUL. Run `C-h f slash-in-docstring` in Emacs 29 or older and see for yourself.

The probability of anyone writing \x expecting NUL is, well, nil. It's guaranteed to be a mistake.

In fact, if you alpha-rename that doc string to "\y -> y" then Emacs won't complain but the doc string will still be wrong and behave as if you wrote "y -> y". (The 'relint' tool finds these things as an extra service.)

(That Emacs doesn't reject undefined character escape sequences such as "\y" is not only a constant source of bugs, it also makes it difficult for us to add new escape sequences without worrying about breaking some (misguided) code. Almost every other language complain about this.)

Thanks for your report -- I scanned all packages on my disk for occurrences of \x without digits when making the change and found none. Shame on me for not using HOL!

> (defun agda-input-compose (f g)
>  "\x -> concatMap F (G x)"

Good catch. Would you tell the Agdas about it?

> Can we document this change of behavior? 

Will do.

Re: Escape syntax in docstrings

[Chen Zhaoyang]

Mattias Engdegård <mattias.engdegard@gmail.com> writes:

> Yes, this was changed recently, but I feel vindicated by your report: "\x" never
> meant what the author thought it did (or at least what I think the author
> thought it did): it is a hex escape without digits and used to be equivalent to
> "\000", yielding the control character NUL. Run `C-h f slash-in-docstring` in
> Emacs 29 or older and see for yourself.

Thank you for you clarification! I totally did not know about the fact
that "\x" means NUL in docstrings. 
I do agree that using the slash to represent λ in docstring is not so
good to begin with.

>> (defun agda-input-compose (f g)
>>  "\x -> concatMap F (G x)"
>
> Good catch. Would you tell the Agdas about it?

Yeah, I am grepping for cases like this and will submit a patch to them soon.

>> Can we document this change of behavior? 
>
> Will do.

Cool, thank you again!

-- 
Chen Zhaoyang

Re: Escape syntax in docstrings

[Ruijie Yu via Emacs development discussions.]

Hello,

Chen Zhaoyang <chenzhauyang@gmail.com> writes:

> Dear friends,
>
> Since when did emacs become stricter when enforcing the escape syntax in
> docstrings? On 27.1 (built by debian), this function evals to no error:
>
> (defun slash-in-docstring ()
>  "\x -> x"
>  nil)
>
> [...]
>
> Can we document this change of behavior?

This issue has been reported in bug#62224 as well.  Apparently after the
OP reported the bug, Eli made changes to the documentation accordingly
-- although I myself don't know where exactly to look.

--
Best,


RY

Re: Escape syntax in docstrings

[Chen Zhaoyang]

Ruijie Yu <ruijie@netyu.xyz> writes:

> This issue has been reported in bug#62224 as well.  Apparently after the
> OP reported the bug, Eli made changes to the documentation accordingly
> -- although I myself don't know where exactly to look.

Thank you for letting me know.

-- 
Chen Zhaoyang

Re: Escape syntax in docstrings

[Mattias Engdegård]

23 mars 2023 kl. 23.32 skrev Chen Zhaoyang <chenzhauyang@gmail.com>:

> I totally did not know about the fact that "\x" means NUL in docstrings. 

Most likely even the original author of that code in the Lisp reader did totally not know about that.

> I do agree that using the slash to represent λ in docstring is not so
> good to begin with.

There is always a tension in doc strings between their dual roles as comments to be read in the source, and as strings to be displayed in help text. Text mark-up in doc strings and comments adds an extra layer to the problem.

The best we can do is trying to catch mistakes as early as possible.

>> Good catch. Would you tell the Agdas about it?
> 
> Yeah, I am grepping for cases like this and will submit a patch to them soon.

Excellent. There is now a special \x check in relint as well.

Re: Escape syntax in docstrings

[David Ongaro]

Chen Zhaoyang <chenzhauyang@gmail.com> writes:

> I do agree that using the slash to represent λ in docstring is not so
> good to begin with.

I want to point out that this doesn't mean you have to replace every
backslash by fn. You can just use an escaped backslash (i.e. \\) and it
will yield a \ in the help.

Re: Escape syntax in docstrings

[James Cloos]

>>>>> "ME" == Mattias Engdegård <mattias.engdegard@gmail.com> writes:

ME> Run `C-h f slash-in-docstring` in Emacs 29 or older and see for yourself.

i was curious to see what that doc would say.

but it did not work in 29, 28, 27 or 26.  (those compiles are all still
hanging around, unused, on my main workstation…)

‘grep -r slash-in-docstring /usr/share/emacs/’ also failed to find anything.

what should one have gotten to read?

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: Escape syntax in docstrings

[Chen Zhaoyang]

The following message is a courtesy copy of an article
that has been posted to gmane.emacs.devel as well.

David Ongaro <david.ongaro@hamburg.de> writes:

> I want to point out that this doesn't mean you have to replace every
> backslash by fn. You can just use an escaped backslash (i.e. \\) and it
> will yield a \ in the help.

That is true indeed. I changed it to `fn` mainly because the familiarity
of this form in ML. It would make it more sense to use the backslash in
the context of languages still backslash to represent lambdas, which is a
form that they inherited from the Cambridge LCF, I believe.

-- 
Chen Zhaoyang

Re: Escape syntax in docstrings

[Eli Zaretskii]

> From: Chen Zhaoyang <chenzhauyang@gmail.com>
> Cc: 
> Newsgroup: gmane.emacs.devel
> Date: Thu, 23 Mar 2023 23:44:46 -0500
> 
> Ruijie Yu <ruijie@netyu.xyz> writes:
> 
> > This issue has been reported in bug#62224 as well.  Apparently after the
> > OP reported the bug, Eli made changes to the documentation accordingly
> > -- although I myself don't know where exactly to look.
> 
> Thank you for letting me know.

See "General Escape Syntax" in the ELisp Reference manual.

Re: Escape syntax in docstrings

[Chen Zhaoyang]

The following message is a courtesy copy of an article
that has been posted to gmane.emacs.devel as well.

> Eli Zaretskii <eliz@gnu.org> writes:
>
> See "General Escape Syntax" in the ELisp Reference manual.

Thank you, Eli.

-- 
Chen Zhaoyang

Re: Escape syntax in docstrings

[Philip Kaludercic]

Mattias Engdegård <mattias.engdegard@gmail.com> writes:

>> (defun agda-input-compose (f g)
>>  "\x -> concatMap F (G x)"
>
> Good catch. Would you tell the Agdas about it?

Agda has already been informed, but it is bad documentation string to
begin with.

Re: Escape syntax in docstrings

[Robert Pluim]

>>>>> On Thu, 23 Mar 2023 23:02:08 +0100, Mattias Engdegård <mattias.engdegard@gmail.com> said:

    Mattias> (That Emacs doesn't reject undefined character escape sequences such
    Mattias> as "\y" is not only a constant source of bugs, it also makes it
    Mattias> difficult for us to add new escape sequences without worrying about
    Mattias> breaking some (misguided) code. Almost every other language complain
    Mattias> about this.)

How strict do you want to be? Emacsʼ own source code has some decidely
non-supported escape sequences, especially when specifying characters
using the `?\c' syntax. I guess a first step could be to add a
byte-compile warning for those analagous to
`lread--unescaped-character-literals', and then maybe expand it to
strings as well.

Robert
-- 

Re: Escape syntax in docstrings

[Andreas Schwab]

On Mär 28 2023, Robert Pluim wrote:

>>>>>> On Thu, 23 Mar 2023 23:02:08 +0100, Mattias Engdegård <mattias.engdegard@gmail.com> said:
>
>     Mattias> (That Emacs doesn't reject undefined character escape sequences such
>     Mattias> as "\y" is not only a constant source of bugs, it also makes it
>     Mattias> difficult for us to add new escape sequences without worrying about
>     Mattias> breaking some (misguided) code. Almost every other language complain
>     Mattias> about this.)
>
> How strict do you want to be? Emacsʼ own source code has some decidely
> non-supported escape sequences, especially when specifying characters
> using the `?\c' syntax.

The elisp manual is explicit that a redundant backslash in ?\c is
ignored, and even recommends its use to avoid confusion in some cases.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Escape syntax in docstrings

[Robert Pluim]

>>>>> On Tue, 28 Mar 2023 14:27:17 +0200, Andreas Schwab <schwab@suse.de> said:

    Andreas> On Mär 28 2023, Robert Pluim wrote:
    >>>>>>> On Thu, 23 Mar 2023 23:02:08 +0100, Mattias Engdegård <mattias.engdegard@gmail.com> said:
    >> 
    Mattias> (That Emacs doesn't reject undefined character escape sequences such
    Mattias> as "\y" is not only a constant source of bugs, it also makes it
    Mattias> difficult for us to add new escape sequences without worrying about
    Mattias> breaking some (misguided) code. Almost every other language complain
    Mattias> about this.)
    >> 
    >> How strict do you want to be? Emacsʼ own source code has some decidely
    >> non-supported escape sequences, especially when specifying characters
    >> using the `?\c' syntax.

    Andreas> The elisp manual is explicit that a redundant backslash in ?\c is
    Andreas> ignored, and even recommends its use to avoid confusion in some cases.

OK, so weʼd only need to check in strings then. I think the list of
required changes would be a lot shorter, but Iʼve not explicitly
checked.

Robert
--