bug#61281: “`(a \, b)” equals to “`(a . , b)”

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Xie Shynur]

[-- Attachment #1: Type: text/plain, Size: 389 bytes --]

In `*ielm*`:

```
ELISP> `(emacs-version \, emacs-version)
(emacs-version . "28.2")
```

________________

In `custom-file`:

```
(custom-set-variables
 '(var `(a . ,b)))
```

Then change some options by GUI menu, and click `Save Options`:

```
(custom-set-variables
 '(var `(a \, b))
 '(changed-option new-value))
```

________________

Is it a feature or bug?

[-- Attachment #2: Type: text/html, Size: 3883 bytes --]

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

Good catch!

`(a \, b) returns (a . <value-of-b>)

Looks like a bug to me.

The code in backquote.el that causes this is this bit
of the definition of function backquote-process:
(eq (car s) backquote-unquote-symbol), where the value
of variable backquote-unquote-symbol is the symbol \,

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

And this goes back at least as far as Emacs 20!

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> Good catch!
>
> `(a \, b) returns (a . <value-of-b>)
>
> Looks like a bug to me.

Isn't `(a \, b) just another read syntax for `(a . (\, b)), which is
another syntax for `(a . ,b) ?

With other words, I think `(a \, b) and `(a . ,b) are different read
syntaxes for the same expression, equivalent to (cons 'a b).

Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > Good catch!
> >
> > `(a \, b) returns (a . <value-of-b>)
> >
> > Looks like a bug to me.
> 
> Isn't `(a \, b) just another read syntax for
> `(a . (\, b)), which is another syntax for
> `(a . ,b) ?

Well, yes.  And that's no doubt why we get that.

But (\, b) shouldn't be handled as ,b.  I know
that it is - in Elisp.  I don't think it should
be.  To me, that spells (faulty) implementation
leaking out.

> With other words, I think `(a \, b) and `(a . ,b) are different read
> syntaxes for the same expression, equivalent to (cons 'a b).

Yes, but see above.  I think \, should be read
as the symbol whose print name is ",".  To me,
`(a \, b) should be treated like (a foo b): a
list of 3 symbols - no evaluation.  And `(a \,b)
should be treated as a list of two symbols,
whose print names are "a" and ",b".

The symbol \, should be read as just a symbol.
The same is not true of just an unescaped comma
- outside a backquoted sexp that raises an error,
and inside one it's handled specially as part of
the backquote syntax.

\, is not just ,

I don't have another Lisp interpreter, but I'm
guessing that Common Lisp does what I expect.
(For Common Lisp also, \ escapes a character.) 

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> Yes, but see above.  I think \, should be read
> as the symbol whose print name is ",".

That's the case.

>  To me, `(a \, b) should be treated like (a foo b): a list of 3
> symbols - no evaluation.

No evaluation by backquote, you mean?  Yes, you need to say `(a ,'\, b).
Is this really different in other Lisps (isn't `,' a reader macro in
Common Lisp)?

>  And `(a \,b) should be treated as a list of two symbols, whose print
> names are "a" and ",b".

That's also the case.


I don't decide about this, but when we changed the semantics of `,' like
you suggest, we will probably break a lot of code for no real gain (I
think the semantics in Elisp clear and easy to understand), so this
sounds like a very bad idea to me.

Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> No evaluation by backquote, you mean?  Yes, you need to say `(a ,'\, b).
> Is this really different in other Lisps (isn't `,' a reader macro in
> Common Lisp)?

I mean, what should the Elisp reader return when reading ",foo"?  It
must be some expression, and backquote will have to handle this
expression differently than quote for the thing to work.

Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > No evaluation by backquote, you mean?  Yes, you need
> > to say `(a ,'\, b).
> > Is this really different in other Lisps (isn't `,' a
> > reader macro in Common Lisp)?

I think I spoke to this in my previous msg
today.  Let me know, if you think not.

> I mean, what should the Elisp reader return
> when reading ",foo"?

Dunno.  (read ",foo") and (read ", foo" return
(\, foo).  Why?  (Just because the current
implementation of backquote depends on that?)

(read "'foo") returns just 'foo - it doesn't
return (quote foo).  (read "`foo") returns
`foo, not (backquote foo).  (read "`,foo")
returns `,foo.  And (read "\\,foo") returns
\,foo.

Maybe (read ",foo") should rather return
",foo" - but of course the Lisp reader would
need to behave accordingly.  And maybe
(read ", foo") should return ",".

> It must be some expression, and backquote
> will have to handle this expression differently
> than quote for the thing to work.

See above.

Notice the error msg from (eval ',foo):

Debugger entered--Lisp error: (void-function \,)
  ,foo
  eval(,foo)
  (progn (eval ',foo))

Nothing in (normal) Lisp syntax shows the use
of comma as a function.  ,foo doesn't look
like function-call syntax, does it?

And here's the error from either (read ",")
or (eval (read ",")):

 End of file during parsing

Yes, an error should be reported in each case,
but I think it should just say that comma is
undefined outside of backquote.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> Dunno.  (read ",foo") and (read ", foo" return
> (\, foo).  Why?  (Just because the current
> implementation of backquote depends on that?)

As I said: any other format would lead to the same "problem", no?
Unless ",X" is something "meta" that has no reader-construct-less
expansion at all.

> (read "'foo") returns just 'foo - it doesn't
> return (quote foo).  (read "`foo") returns
> `foo, not (backquote foo).  (read "`,foo")
> returns `,foo.

You forget that the printer is able to produce this format.  Try `car'.

> Notice the error msg from (eval ',foo):
>
> Debugger entered--Lisp error: (void-function \,)
>   ,foo
>   eval(,foo)
>   (progn (eval ',foo))
>
> Nothing in (normal) Lisp syntax shows the use
> of comma as a function.  ,foo doesn't look
> like function-call syntax, does it?

Another side effect of ,X being equivalent to (\, X.).  That's the only
thing you need to remember.  When you eval that, you should not be
surprised that it's evaluated like ... Lisp evaluates things.

> And here's the error from either (read ",")
> or (eval (read ",")):
>
>  End of file during parsing
>
> Yes, an error should be reported in each case,
> but I think it should just say that comma is
> undefined outside of backquote.

S-exps are defined recursively.  ",X" is read syntax of a valid s-exp,
and I don't think we want to make the reader raise an error for it.  But
the reader expects an expression following the ",".  And a single "," is
_not_the read syntax of the symbol with the name "," (that is "\,":
(symbol-name (read "\\,")) --> ",").

So "End of file during parsing" is an appropriate message: no complete
expression could be read, but something that looks like the start of an
expression.


Ok, so everything is about that you don't want that ,X and (\, X) are
equivalent.  All your arguments were of the kind "the implications are
surprising".  But you never answered the core question: what should ,X
expand to instead that would not have any of these implications?  Else
all you say is that it's surprising when you lack knowledge.


OTOH it seems not easy to find the information ,X == (\, X) somewhere.
Is there a place where there is said something about the kind of
expression the reader construct ,X produces?  I didn't find anything in
a rush.  It should be explained, else this thing indeed can lead to
surprises, as the one reported here.

Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > Notice the error msg from (eval ',foo):
> >
> > Debugger entered--Lisp error: (void-function \,)
> >   ,foo
> >   eval(,foo)
> >   (progn (eval ',foo))
> >
> > Nothing in (normal) Lisp syntax shows the use
> > of comma as a function.  ,foo doesn't look
> > like function-call syntax, does it?
> 
> Another side effect of ,X being equivalent to (\, X.).

Yes, that's what I was saying.

> That's the only thing you need to remember.

I think you're making a virtue out of necessity. ;-)

Yes, that's the way comma is implemented inside
backquote in Elisp.  So yes, just remember that
implementation factoid.
 
> > And here's the error from either (read ",")
> > or (eval (read ",")):
> >
> >  End of file during parsing
> >
> > Yes, an error should be reported in each case,
> > but I think it should just say that comma is
> > undefined outside of backquote.
> 
> S-exps are defined recursively.  ",X" is read
> syntax of a valid s-exp,

That it is so is only because Elisp implements it
as that particular read macro.

And the question is about "\,", not ",".

(setq ,X  42) ; => 42
(setq \,X 42) ; => (wrong-type-argument symbolp (\, X))

> Ok, so everything is about that you don't want
> that ,X and (\, X) are equivalent.

You can say that, I suppose.  I'd instead say that
it's about being able to escape a comma inside a
backquote - just like elsewhere, so it's just
treated like a symbol character, even in the case
where it's the only char in the symbol name.

(I'd be interested in what the case is in Common
Lisp, including what a typical implementation is.)

> All your arguments were of the kind "the implications are
> surprising".  But you never answered the core question: what should ,X
> expand to instead that would not have any of these implications?  Else
> all you say is that it's surprising when you lack knowledge.

If you say so.  I haven't said anything about the
implementation: what "," should expand to.  I'd
say that if unescaped its behavior should be to
do what it does now.

(FWIW, I don't think I said that the behavior or
their implications are surprising.  But yes, I
didn't expect "\," to not escape out of the
backquote handling of ",".  I didn't expect comma
to be any different from period or @.)

The question is whether \, and , should have the
same behavior.  Certainly \z and z have the same
behavior.  But character z has no special behavior
inside a backquote.

\@ and @ don't have the same behavior inside a
backquote.  And neither do \. and . -- only \,
and , have the same behavior.  To me, that's just
an implementation/design thing, not something
normal or inevitable.  Not a big deal, not the
end of the world.  I minor unfortunate thing
(gotcha).

> OTOH it seems not easy to find the information ,X == (\, X) somewhere.
> Is there a place where there is said something about the kind of
> expression the reader construct ,X produces?  I didn't find anything in
> a rush.  It should be explained, else this thing indeed can lead to
> surprises, as the one reported here.

I pointed to the comments in the code.  They tell
the story.  But I don't think there's any such
explanation/description in the doc.  Normally we
wouldn't need anything like that -- we'd consider
that to be just implementation/plumbing.  But in
this case it seems that users need to know the
implementation if they're really to understand
the behavior.  But only if they need to use a
symbol named "," normally inside backquote -- a
rare case, surely.

Again, the bug is certainly a tiny corner case.
It's not like users can't use backquote syntax
without knowing this aspect of the implementation.
It's not elegant, but it works pretty well.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> And the question is about "\,", not ",".
>
> (setq ,X  42) ; => 42
> (setq \,X 42) ; => (wrong-type-argument symbolp (\, X))

The other way round.

"\,X" is interpreted as symbol, just as "\,".

What's the question about "\,"?


> (I'd be interested in what the case is in Common
> Lisp, including what a typical implementation is.)

AFAIU, there are different implementations.  I tested two random CL and
two random scheme interpreters.  Of those 4, 3 expanded
,X --> (unquote X), and in one case ,X was read as an atomic expression.

> The question is whether \, and , should have the
> same behavior.  Certainly \z and z have the same
> behavior.  But character z has no special behavior
> inside a backquote.

I think the necessity to escape the comma to get the symbol has been
introduced to avoid ambiguities with uses of the reader macro.

> \@ and @ don't have the same behavior inside a
> backquote.  And neither do \. and . -- only \,
> and , have the same behavior.  To me, that's just
> an implementation/design thing, not something
> normal or inevitable.  Not a big deal, not the
> end of the world.  I minor unfortunate thing
> (gotcha).

In any case, much too late to change it.

> I pointed to the comments in the code.  They tell the story.  But I
> don't think there's any such explanation/description in the doc.

Yes, it would be good to add something.


Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > And the question is about "\,", not ",".
> >
> > (setq ,X  42) ; => 42
> > (setq \,X 42) ; => (wrong-type-argument symbolp (\, X))
> 
> The other way round.

Yes, I accidentally reversed the comments.  Thx.

> "\,X" is interpreted as symbol, just as "\,".

Yes, backslash escapes the special meaning of
comma -- but only outside backquote.

Unescaped comma by itself is not interpreted as
a symbol inside backquote or outside it.

And leaving aside the X:

(setq ,  42)  ; => (wrong-number-of-arguments setq 1)
              ;    from expansion to (setq (\, 42))
(setq \, 42)  ; => 42

Comma is special in raising an error outside
backquote.  But there, escaping it makes it
just symbol syntax.  Not so inside backquote:
there, escaping it has no effect -- "\," and
"," behave the same inside backquote.

> What's the question about "\,"?

Why is it equivalent to "," inside backquote?

> > (I'd be interested in what the case is in Common
> > Lisp, including what a typical implementation is.)
> 
> AFAIU, there are different implementations.  I tested two random CL and
> two random scheme interpreters.  Of those 4, 3 expanded
> ,X --> (unquote X), and in one case ,X was read as an atomic expression.

Sorry, I don't know what that means.  How is
"\," syntax treated inside backquote in those
interpreters?

If different CLs have different behavior in this
regard, then I wonder what the spec says about it
(if anything).  I didn't notice anything.

> > The question is whether \, and , should have the
> > same behavior.  Certainly \z and z have the same
> > behavior.  But character z has no special behavior
> > inside a backquote.
> 
> I think the necessity to escape the comma to get the symbol has been
> introduced to avoid ambiguities with uses of the reader macro.

Not sure what you mean.  I'd think it's to give
you a way to use comma in a symbol name.  Same as
for the other chars you need to escape for that,
such as "(".

Escaping a comma outside backquote works, to give
you symbol syntax, but that doesn't work inside
backquote?  Why doesn't it?  I don't mean "why" in
terms of "how so".  I mean why _should_ that be
the case?  Is there a good reason for this behavior
(in terms of useful behavior, not in terms of
implementation)?  I don't see any, so far.

> > \@ and @ don't have the same behavior inside a
> > backquote.  And neither do \. and . -- only \,
> > and , have the same behavior.
> >
> > To me, that's just an implementation/design
> > thing, not something normal or inevitable.
> > Not a big deal, not the end of the world.
> > [A] minor unfortunate thing (gotcha).
> 
> In any case, much too late to change it.
> 
> > I pointed to the comments in the code.  They tell the story.  But I
> > don't think there's any such explanation/description in the doc.
> 
> Yes, it would be good to add something.

Glad we agree on something. ;-)

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Ihor Radchenko]

Drew Adams <drew.adams@oracle.com> writes:

> ...  But only if they need to use a
> symbol named "," normally inside backquote -- a
> rare case, surely.

I recall one user had a need to macro-expand something that indented to
be passed to another macro-expand. We did not find a way to retain ","
in the macro-expanded sexp.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > ...  But only if they need to use a
> > symbol named "," normally inside backquote -- a
> > rare case, surely.
> 
> I recall one user had a need to macro-expand something that indented to
> be passed to another macro-expand. We did not find a way to retain ","
> in the macro-expanded sexp.

Thanks.  All the more reason why it would be good
for the bug to be fixed.  It may be a corner case,
but apparently it really exists.

[I'm guessing Emacs autocorrected "intended" to
"indented".]

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> > I recall one user had a need to macro-expand something that indented to
> > be passed to another macro-expand. We did not find a way to retain ","
> > in the macro-expanded sexp.
>
> Thanks.  All the more reason why it would be good for the bug to be
> fixed.  It may be a corner case, but apparently it really exists.

Unless I'm misunderstanding, this is a contradictory argument: to retain
a single "," for another macroexpand, you are actually _using_ this
implementation detail (in the outer expansion), so it is of no use to
change the ,X --> (\, X) expansion because then, the outer macro
expansion would handle the single "," as well.

And if the reader syntax "," was meant, this kind of "problem" exists in
other Lisps as well.

Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Ihor Radchenko <yantar92@posteo.net> writes:

> I recall one user had a need to macro-expand something that indented to
> be passed to another macro-expand. We did not find a way to retain ","
> in the macro-expanded sexp.

I guess you are describing the common difficulties related to multiple
levels of backquotes (nested backquotes).  These are tricky (for human
brains) and you need to know little "tricks" to get what you want (took
me a while to discover how to deal with nested backquotes, maybe we
should have examples in the manual?).

Anyway, you (only) need to use trivial quoting, it is not necessary and
probably not good style to use the symbol "," instead of the reader
construct in human written code.


Compare:

#+begin_src emacs-lisp
;; Substitution at the same place at multiple levels:
(let ((f 'my-function))
  ``(when (funcall ,,f) (do-something)))
==>
 `(when (funcall ,my-function)
    (do-something))

;; Substitution once, by outside level backquote:
(let ((f 'my-function))
  ``(when (funcall ,',f) (do-something)))
==>
 `(when (funcall ,'my-function)
    (do-something))

;; Substitution once, by the inside backquote
``(when (funcall ,,'f) (do-something))
==>
  `(when (funcall ,f)
     (do-something))

  or simpler:

``(when (funcall ,f) (do-something))
==>
  `(when (funcall ,f)
     (do-something))
#+end_src

You probably tried to get some of these cases work, and it's not trivial
to get to a solution the first time one encounters this problem.

But this has not directly a relation to what we discuss here.  If you
rewrite ,X as (\, X) you have additional ways to express the above
things, looking like (,'\, X) etc (though these are less readable).

But, unless I guessed wrong what you were originally trying to achieve,
you just have to solve the nested-backquotes problems like in other
Lisps, Elisp is not preventing you to use any of these syntaxes.  It's
just that trying the (\, X) rewrite doesn't bring you closer to the
solution of the original problem.


Michael.

bug#61281: Double backquote expansion and ", " (was: bug#61281: “`(a \, b)” equals to “`(a . , b)”)

[Ihor Radchenko]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> But this has not directly a relation to what we discuss here.  If you
> rewrite ,X as (\, X) you have additional ways to express the above
> things, looking like (,'\, X) etc (though these are less readable).

I may indeed be misunderstanding the case I raised vs. the discussed bug.
Let's branch this off.

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> I recall one user had a need to macro-expand something that indented to
>> be passed to another macro-expand. We did not find a way to retain ","
>> in the macro-expanded sexp.
>
> I guess you are describing the common difficulties related to multiple
> levels of backquotes (nested backquotes).  These are tricky (for human
> brains) and you need to know little "tricks" to get what you want (took
> me a while to discover how to deal with nested backquotes, maybe we
> should have examples in the manual?).

I would appreciate having such examples in the manual.
Also, CCing Adam, who originally raised the question.

> Anyway, you (only) need to use trivial quoting, it is not necessary and
> probably not good style to use the symbol "," instead of the reader
> construct in human written code.

Could you elaborate?

> Compare:
>
> #+begin_src emacs-lisp
> ;; Substitution at the same place at multiple levels:
> (let ((f 'my-function))
>   ``(when (funcall ,,f) (do-something)))
> ==>
>  `(when (funcall ,my-function)
>     (do-something))
>
> ;; Substitution once, by outside level backquote:
> (let ((f 'my-function))
>   ``(when (funcall ,',f) (do-something)))
> ==>
>  `(when (funcall ,'my-function)
>     (do-something))
>
> ;; Substitution once, by the inside backquote
> ``(when (funcall ,,'f) (do-something))
> ==>
>   `(when (funcall ,f)
>      (do-something))
>
>   or simpler:
>
> ``(when (funcall ,f) (do-something))
> ==>
>   `(when (funcall ,f)
>      (do-something))
> #+end_src
>
> You probably tried to get some of these cases work, and it's not trivial
> to get to a solution the first time one encounters this problem.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#61281: Double backquote expansion and ", "

[Michael Heerdegen]

Ihor Radchenko <yantar92@posteo.net> writes:

> > Anyway, you (only) need to use trivial quoting, it is not necessary and
> > probably not good style to use the symbol "," instead of the reader
> > construct in human written code.

> Could you elaborate?

On what?  You said "We did not find a way to retain "," in the
macro-expanded sexp.", I tried to demonstrate how to handle this.  As
you see, it is not necessary to use "," the symbol, the examples all use
the reader construct syntax and quoting.

If that doesn't answer your question, please ask more specific
questions.

> > Compare:
> >
> > #+begin_src emacs-lisp
> > ;; Substitution at the same place at multiple levels:
> > (let ((f 'my-function))
> >   ``(when (funcall ,,f) (do-something)))
> > ==>
> >  `(when (funcall ,my-function)
> >     (do-something))
> >
> > ;; Substitution once, by outside level backquote:
> > (let ((f 'my-function))
> >   ``(when (funcall ,',f) (do-something)))
> > ==>
> >  `(when (funcall ,'my-function)
> >     (do-something))
> >
> > ;; Substitution once, by the inside backquote
> > ``(when (funcall ,,'f) (do-something))
> > ==>
> >   `(when (funcall ,f)
> >      (do-something))
> >
> >   or simpler:
> >
> > ``(when (funcall ,f) (do-something))
> > ==>
> >   `(when (funcall ,f)
> >      (do-something))
> > #+end_src
> >
> > You probably tried to get some of these cases work, and it's not trivial
> > to get to a solution the first time one encounters this problem.



Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Andreas Schwab]

On Feb 06 2023, Michael Heerdegen wrote:

> S-exps are defined recursively.  ",X" is read syntax of a valid s-exp,
> and I don't think we want to make the reader raise an error for it.

FWIW, clisp generates an error if the comma occurs outside of backquote.
If you quote the comma, it is treated as an ordinary symbol character.

$ clisp -q -x ',X'
*** - READ: comma is illegal outside of backquote
$ clisp -q -x '\,X'
*** - SYSTEM::READ-EVAL-PRINT: variable |,X| has no value

$ clisp -q -x '`\,X'
|,X|

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > S-exps are defined recursively.  ",X" is read syntax of a valid s-exp,
> > and I don't think we want to make the reader raise an error for it.
> 
> FWIW, clisp generates an error if the comma occurs outside of backquote.
> If you quote the comma, it is treated as an ordinary symbol character.
> 
> $ clisp -q -x ',X'
> *** - READ: comma is illegal outside of backquote
> $ clisp -q -x '\,X'
> *** - SYSTEM::READ-EVAL-PRINT: variable |,X| has no value
> 
> $ clisp -q -x '`\,X'
> |,X|

Thanks for that info, confirming what clisp does.
Those are all what I'd expect/hope: the \,x
inside the backquote sexp is read as the symbol
|,x|.

The Elisp question (bug) is really only about a
_bare_ escaped comma, not one that's immediately
followed by symbol chars (e.g. x).

E.g., what do these give?

$ clisp -q -x '`\,'
$ clisp -q -x '`(\, x)'

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Andreas Schwab]

In clisp, backquote is implemented via system::backquote,
system::unquote and system::splice.  Thus the symbol |,| has no special
meaning.

$ clisp -q -x $'(car \'`,X)'
SYSTEM::BACKQUOTE
$ clisp -q -x $'(cadr \'`,X)'
(SYSTEM::UNQUOTE X)
$ clisp -q -x $'(cadr \'`(,@X))'
((SYSTEM::SPLICE X))
$ clisp -q -x "'(system::backquote (system::unquote x))"
`,X
$ clisp -q -x "'(system::backquote (system::splice x))"
`,@X
$ clisp -q -x '`,@x'
*** - READ: the syntax `,@form is invalid

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> In clisp, backquote is implemented via system::backquote,
> system::unquote and system::splice.  Thus the symbol |,|
> has no special meaning.

If the symbol |,| has no special meaning inside backquote
then that answers my question.

Still, could you please show these uses of bare "\," 
explicitly?

 $ clisp -q -x '`\,'
 $ clisp -q -x '`(\, x)'

You showed the following, which are not in question.
Elisp does the same thing for these - it (correctly)
reads "\,X" as the symbol named ",X", even inside
backquote sexps.

>> $ clisp -q -x ',X'
>> *** - READ: comma is illegal outside of backquote
>> $ clisp -q -x '\,X'
>> *** - SYSTEM::READ-EVAL-PRINT: variable |,X| has no value
>>
>> $ clisp -q -x '`\,X'
>> |,X|

It's only _bare_ "\," that Elisp treats the same
as an unescaped comma (","), provoking eval of
what follows it.  What does clisp do for bare "\,"
(i.e., not immediately followed by symbol chars)?

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> Still, could you please show these uses of bare "\,"
> explicitly?
>
>  $ clisp -q -x '`\,'
>  $ clisp -q -x '`(\, x)'

| micha> clisp -q -x '`\,'
| |,|
| micha> clisp -q -x '`(\, x)'
| (|,| X)

In Emacs we could as well change the symbol (or whatever object) used as
"tag" for the expansion of the `,` reader construct.  But unless this
solves some practical problem, since there is the cost of breaking code,
I don't understand why we are still discussing this.

You would suggest to make a change like this, Drew?


Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > Still, could you please show these uses of bare "\,"
> > explicitly?
> >
> >  $ clisp -q -x '`\,'
> >  $ clisp -q -x '`(\, x)'
> 
> | micha> clisp -q -x '`\,'
> | |,|
> | micha> clisp -q -x '`(\, x)'
> | (|,| X)

Great.  So clisp handles the case right (IMO).  It
doesn't seem to have this bug, at least.

> In Emacs we could as well change the symbol (or
> whatever object) used as "tag" for the expansion
> of the `,` reader construct.  But unless this
> solves some practical problem, since there is
> the cost of breaking code, I don't understand
> why we are still discussing this.
> 
> You would suggest to make a change like this, Drew?

I can't speak to the code change that should be
made.  (But I think I understand you, and yes,
I think something like that is what's needed.)

I do wish the code behaved solidly, with no such
"capture" of occurrences of "\,".  But I've
already agreed with you that I doubt anyone will
fix this, and maybe it's too risky to bother
doing that.

I don't see the risk so much (if at all) in that
some code might exist that depends on the bugged
behavior.  I seriously doubt that.  But (1) I
don't know that, and nested etc. backquote
expressions can get very complicated.

I do expect some risk in someone trying to fix
this and ending up causing more problems.
That's the code breakage I'd worry about.
That's a reason I won't be pretending to have
the right code fix.

Assuming the bug isn't going to be fixed, I'd
propose we just describe the problem in a
comment.  That can be done with words or with
code examples.  This thread is already full of
both. ;-)

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Andreas Schwab]

On Feb 08 2023, Drew Adams wrote:

>> > Still, could you please show these uses of bare "\,"
>> > explicitly?
>> >
>> >  $ clisp -q -x '`\,'
>> >  $ clisp -q -x '`(\, x)'
>> 
>> | micha> clisp -q -x '`\,'
>> | |,|
>> | micha> clisp -q -x '`(\, x)'
>> | (|,| X)
>
> Great.  So clisp handles the case right (IMO).

It just happens to use different symbols to implement backquote.  That
Elisp uses symbols with the same spelling as the reader macros is not a
bug, only an implementation detail.  For clisp, system::unquote has the
same magic bahaviour for backquote handling as \, does for Elisp.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> I don't see the risk so much (if at all) in that
> some code might exist that depends on the bugged
> behavior.  I seriously doubt that.

I see lots of places alone in the Emacs sources that rely on the behavior.

> Assuming the bug isn't going to be fixed, I'd propose we just describe
> the problem in a comment.

Where should this comment be located?

Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > I don't see the risk so much (if at all) in that
> > some code might exist that depends on the bugged
> > behavior.  I seriously doubt that.
> 
> I see lots of places alone in the Emacs sources that rely on the
> behavior.

Maybe we mean different things by the bugged behavior?
Do you see lots of places where the Emacs sources use "\,"?

> 
> > Assuming the bug isn't going to be fixed, I'd propose we just describe
> > the problem in a comment.
> 
> Where should this comment be located?

As part of the comment you pointed to, which tries
to describe how the code works/behaves?

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> Maybe we mean different things by the bugged behavior?  Do you see
> lots of places where the Emacs sources use "\,"?

Yes, there are indeed some.  A common one is (eq \, (car-safe EXPR)) to
test whether some expression is an "unquote expression".  Used in
code-walker like things - byte compiler, stuff to save expressions in
files, pcase, such things.

Any package can make use of the reader constructs in non-trivial ways (I
mean, uses unrelated to backquote).  And the only way to treat such
expressions is to test for the expansion.

> > > Assuming the bug isn't going to be fixed, I'd propose we just describe
> > > the problem in a comment.
> >
> > Where should this comment be located?
>
> As part of the comment you pointed to, which tries
> to describe how the code works/behaves?

See my other comment then.


Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Andreas Schwab]

On Feb 07 2023, Drew Adams wrote:

>> In clisp, backquote is implemented via system::backquote,
>> system::unquote and system::splice.  Thus the symbol |,|
>> has no special meaning.
>
> If the symbol |,| has no special meaning inside backquote
> then that answers my question.

That symbol has no special meaning at all.  (A reader macro is not a
symbol.)

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Jim Porter]

On 2/4/2023 8:32 PM, Michael Heerdegen wrote:
> I don't decide about this, but when we changed the semantics of `,' like
> you suggest, we will probably break a lot of code for no real gain (I
> think the semantics in Elisp clear and easy to understand), so this
> sounds like a very bad idea to me.

What about emitting a warning? That shouldn't break anything, and it 
might catch some bugs. (I'm not sure if a warning is easy to add for 
this though...)

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Jim Porter <jporterbugs@gmail.com> writes:

> What about emitting a warning? That shouldn't break anything, and it
> might catch some bugs. (I'm not sure if a warning is easy to add for
> this though...)

You mean the reader should emit a warning?  Note that at least
expressions printed by Emacs itself might be printed like this, e.g.

  (nth 1 '`(a . ,b)) ==> (a \, b)

so you would get warnings for automatically generated code.

OTOH, is it that likely that people write something like the above by
accident?  If the symbol "," (with read syntax "\,") appears outside
backquote or pcase patterns at a place where it is funcalled or
referenced as a variable you already get a compiler warning.  And are we
sure that nobody ever wants to use that syntax by purpose?


BTW, if you are working with backquote expressions a lot you are seeing
such commas in apparently "normal" (undotted) lists regularly (because
the printer seems to prefer that syntax over using a dotted list, like
in the above example).  It surprises you only once or twice.


Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> You mean the reader should emit a warning?  Note that at least
> expressions printed by Emacs itself might be printed like this, e.g.
>
>   (nth 1 '`(a . ,b)) ==> (a \, b)
>
> so you would get warnings for automatically generated code.

An example is even in the original report.

Maybe the printer could be taught to print (... . ,X) in this form
instead of (... \, X).

Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Jim Porter]

On 2/5/2023 4:18 PM, Michael Heerdegen wrote:
> Michael Heerdegen <michael_heerdegen@web.de> writes:
> 
>> You mean the reader should emit a warning?  Note that at least
>> expressions printed by Emacs itself might be printed like this, e.g.
>>
>>    (nth 1 '`(a . ,b)) ==> (a \, b)
>>
>> so you would get warnings for automatically generated code.
> 
> An example is even in the original report.

Well, that's what I get for trying to send a message shortly before 
going to sleep.

> Maybe the printer could be taught to print (... . ,X) in this form
> instead of (... \, X).

This would probably be nice though (assuming it's sufficiently easy). 
The current printing form would likely confuse a novice programmer if 
they encountered it.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > Yes, but see above.  I think \, should be
> > read as the symbol whose print name is ",".
> 
> That's the case.

Inside a backquote, an escaped comma's handled
as if it were unescaped.  That's the bug.
 
> > To me, `(a \, b) should be treated like
> > (a foo b): a list of 3 symbols - no evaluation.
> 
> No evaluation by backquote, you mean?

I mean that \, is not (should not be) the same
as comma whether inside or outside a backquote.

It is (should be) only an UNescaped comma that
(1) when inside backquote introduces evaluation
of what follows it and (2) raises an error when
not inside a backquote.

> Yes, you need to say `(a ,'\, b).

When I said "no evaluation" I meant that \,
shouldn't cause evaluation inside backquote. 
I wasn't talking about evaluating the symbol
\, itself - I'm not interested in doing that.

I'm talking about the special interpretation
of \, here, i.e., its being handled just like
an UNescaped comma, instead of just like any
other symbol (e.g. foo).

> Is this really different in other Lisps 
> (isn't `,' a reader macro in Common Lisp)?

It may be, dunno, but I'd bet that Common Lisp
treats unescaped comma differently from escaped
comma.

I meant that reading \, should be like reading
foo, whether inside backquote or not.  The only
thing that should introduce evaluation within
a backquote is an UNescaped comma.

\, is not ,

Dunno about the reader macro question.  (But
I'm guessing that the overall difference here
is how the CL reader works, yes.)

> I don't decide about this, but when we changed
> the semantics of `,' like you suggest, we will
> probably break a lot of code for no real gain
> (I think the semantics in Elisp clear and easy
> to understand), so this sounds like a very bad
> idea to me.

I'm not saying the bug should be fixed.  I have
the same concern you express.  Not that I really
think that "a lot of code" will break (I don't),
but that (1) some code might depend on this
behavior and so break, and (more importantly)
(2) it's likely that the new code will introduce
other (new) backquote bugs.  There be dragons.

What I expect is that the bug will be closed as
"wont-fix".  This should nevertheless be seen
as a (minor) bug.  It should be noted somewhere
(at least called out in code comments).

Whether it's important enough to merit mention
in the Elisp manual, I don't know.  Normally
it's not, but someone actually stumbling on this
would need to check the code to discover it and
would need to recognize that it's a bug by design.

File backquote.el says clearly at the outset:

;; When the Lisp reader sees `(...), it generates
;;   (\` (...)).
;;
;; When it sees ,... inside such a backquote form,
;;   it generates (\, ...).
;;
;; For ,@... it generates (\,@ ...).

It's _good_ that it says this.  That's a pretty
clear description of the implementation / design.

(Except that the use of (...) and ... in what the
reader sees doesn't make clear what those stand
for.  Backquote applies to unparenthesized sexps
also.)

It could also usefully say that this doesn't
provide exactly the usual Lisp backquote behavior.
It could call out what we're talking about:
_escaped_ comma is (wrongly) treated as unescaped
comma inside a backquote.  That's the design,
AFAICT, and that's not the traditional behavior
(IIRC).

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> > > Yes, but see above.  I think \, should be
> > > read as the symbol whose print name is ",".
> >
> > That's the case.
>
> Inside a backquote, an escaped comma's handled
> as if it were unescaped.  That's the bug.

What do you mean by "escaped" and "unescaped" comma?  "\," is the read
syntax for the symbol named ",", is that what you mean by "escaped"
comma?  And the unescaped comma is the reader macro?

Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

(Michael, I started to reply to your message
by explaining what I meant by escaping and
unescaping, but I think you do know what I
have in mind.)

I guess an argument can be made that there's
no bug here.  I don't think that's right, but
I can see an argument for it.

Outside backquote (and outside a string etc.)
you need to escape a comma: "\,".  Otherwise,
an error is raised.  Escaping it means it's
read as part of a symbol name.

Comma is special in this way.  And an escaped
comma read on its own produces the symbol
named ",".

It's the need to escape a comma normally that
makes me think that the same kind of escaping
should remove the special behavior that comma
has inside backquote, and just have it be
treated by the reader as a symbol there.

For characters other than comma, which don't
raise an error without escaping generally, a
backslash (aside from particular contexts)
generally is a no-op: \X is the same as X for
a character X.

Because of that, you could make an argument
that that's what should happen for comma
inside backquote: a backslash to escape it
should have no effect.  That's clearly what
_is_ happening, in any case.

I don't think that's the most logical behavior,
because comma in Lisp otherwise behaves so
differently if escaped or not (just read as
a symbol-constituent char when escaped).

And not the best behavior, because it makes it
impossible to use a symbol named "," within a
backquote, without having it get the special
backquote comma behavior.  You can't remove
the special behavior that backquote gives it.

Normally, the special behavior of any char can
be removed, to include it in a symbol name,
including to use it alone as a symbol name.

In particular, note that @ doesn't have the
same problem that comma has.  You can remove
its special meaning there, to make it just be
read as a symbol, by backslash-escaping it:

(setq \@  '(3 4))
(setq foo '(4 5))

`(a ,\@ foo) ; ==> (a (3 4) foo)
`(a ,@  foo) ; ==> (a 4 5)

If @ behaved like comma here, then both of
those backquote sexps would result in (a 4 5).

Likewise period - behaves as I would expect:
backslash-escaping it removes its special
backquote behavior.

(setq \. '(1 2))

`(a .  ,foo) ; ==> (a 4 5)
`(a \. ,foo) ; ==> (a \. (4 5))

If period behaved as comma does then both of
those backquote sexps would result in (a 4 5).

Of course, there's no crying _need_ to use
symbols named ".", "@" and "," as variables,
functions, etc. inside a backquote.  But why
not, and why not be consistent among all such
special chars?

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

[-- Attachment #1: Type: text/plain, Size: 1365 bytes --]

Xie Shynur <one.last.kiss@outlook.com> writes:

> ELISP> `(emacs-version \, emacs-version)
> (emacs-version . "28.2")

> ```
> (custom-set-variables
>  '(var `(a . ,b)))
> ```
>
> Then change some options by GUI menu, and click `Save Options`:
>
> ```
> (custom-set-variables
>  '(var `(a \, b))
>  '(changed-option new-value))
> ```

> Is it a feature or bug?

It doesn't harm (it's not wrong), you see an implementation detail
exposed, but it's not nice either.

After discussing this and peripheral stuff with others I would suggest
(I try to give a complete summary of the useful suggestions here) to do
these things:


(1) We should try to teach the printer to prefer the syntax
(... . ,X) over (... \, X).  The latter is equivalent but in most
cases the first version will be easier to interpret.  This would fix
the above case about saving custom variables (readability).


(2) We should (in the recently added function docstrings and the manual)
explain that the reader constructs `X, ,X and ,@X are expanded to (or
equivalent to) (\` X), (\, X) and (\,@ X) respectively, where the cars
are the symbols with the names "`", "," and ",@".

While this is an implementation detail, not knowing about that fact
leaves the semantics of expressions like above unclear, which is not
good.


(3) Fix the header in backquote.el as suggested by Drew, e.g. like this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-lisp-emacs-lisp-backquote.el-Small-fix-in-commentary.patch --]
[-- Type: text/x-diff, Size: 1025 bytes --]

From 7c8bc97263a1b6d009a11b32d6e62e82fe14e997 Mon Sep 17 00:00:00 2001
From: Michael Heerdegen <michael_heerdegen@web.de>
Date: Mon, 6 Feb 2023 04:47:12 +0100
Subject: [PATCH] ; * lisp/emacs-lisp/backquote.el: Small fix in commentary

---
 lisp/emacs-lisp/backquote.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lisp/emacs-lisp/backquote.el b/lisp/emacs-lisp/backquote.el
index 84527234207..de14b5cd42f 100644
--- a/lisp/emacs-lisp/backquote.el
+++ b/lisp/emacs-lisp/backquote.el
@@ -25,9 +25,9 @@

 ;;; Commentary:

-;; When the Lisp reader sees `(...), it generates (\` (...)).
-;; When it sees ,... inside such a backquote form, it generates (\, ...).
-;; For ,@... it generates (\,@ ...).
+;; When the Lisp reader sees `X it generates (\` X).
+;; When it sees ,X it generates (\, X).  For ,@X it generates
+;; (\,@ X).

 ;; This backquote will generate calls to the backquote-list* form.
 ;; Both a function version and a macro version are included.
--
2.30.2


[-- Attachment #3: Type: text/plain, Size: 20 bytes --]



Thanks,

Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> (2) We should (in the recently added function docstrings and the manual)
> explain that the reader constructs `X, ,X and ,@X are expanded to (or
> equivalent to) (\` X), (\, X) and (\,@ X) respectively, where the cars
> are the symbols with the names "`", "," and ",@".

Yes and no, no?

I see a difference between escaped comma and
escaped comma before @.  The above "are expanded
to" doesn't cover this, I think.  For example:

`(a  ,@ foo) ; ==> (a 4 5)
`(a \,@foo)  ; ==> (a \,@ foo) - good
`(a \,@ foo) ; ==> (a \,@ foo) - good

`(a  , foo) ; ==> (a 4 5)
`(a \,foo)  ; ==> (a \,foo) - good
`(a \, foo) ; ==> (a 4 5)   - bad, the bug case

The \,@ cases and the \,foo case "work" because
\, immediately followed by any escaped char or
unescaped whitespace etc. works.  It's only \,
followed by unescaped whitespace etc. that
doesn't work.

> While this is an implementation detail, not knowing about that fact
> leaves the semantics of expressions like above unclear, which is not
> good.

Agreed, but I don't think just describing those
expansions that way is sufficient.  For one
thing, what's X?  Whether certain chars follow
the comma immediately makes a difference.

> (3) Fix the header in backquote.el as suggested by Drew, e.g. like this:

+;; When the Lisp reader sees `X it generates (\` X).
+;; When it sees ,X it generates (\, X).  For ,@X it generates
+;; (\,@ X).

I don't think that's sufficient - see above.
,X and ,@X aren't handled the same, and it
matters what X is.  X is not necessarily a new
sexp.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> `(a  ,@ foo) ; ==> (a 4 5)
> `(a \,@foo)  ; ==> (a \,@ foo) - good
> `(a \,@ foo) ; ==> (a \,@ foo) - good
> 
> `(a  , foo) ; ==> (a 4 5)
> `(a \,foo)  ; ==> (a \,foo) - good
> `(a \, foo) ; ==> (a 4 5)   - bad, the bug case

Sorry for the typos in the comments.
This is what it is:

1. `(a  ,@ foo) ; ==> (a 4 5)
2. `(a \,@foo)  ; ==> (a \,@foo)  - good
3. `(a \,@ foo) ; ==> (a \,@ foo) - good

4. `(a  , foo) ; ==> (a (4 5))
5. `(a \,foo)  ; ==> (a \,foo) - good
6. `(a \, foo) ; ==> (a 4 5)   - bad, bug case

Note that #6 is not the same as #4.  It's not
just that the backslash made no difference.
#6 is the same as #1: the backslash made the
comma act like ,@.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> > (2) We should (in the recently added function docstrings and the manual)
> > explain that the reader constructs `X, ,X and ,@X are expanded to (or
> > equivalent to) (\` X), (\, X) and (\,@ X) respectively, where the cars
> > are the symbols with the names "`", "," and ",@".
>
> Yes and no, no?
>
> I see a difference between escaped comma and
> escaped comma before @.  The above "are expanded
> to" doesn't cover this, I think.  For example:
> [...]
> `(a \, foo) ; ==> (a 4 5)   - bad, the bug case

It's a logical consequence (although not an obvious one, but also not
totally unobvious) of the fact that this expression is equivalent to
`(a . ,foo).

If the Elisp printer would not sometimes output a syntax like what you
call a "bug case", this would be such an extremely rare appearing corner
case that I really would not mention it explicitly.  Really, this would
make people more wonder about why they need to know that, and distract
from understanding the main points.


> The \,@ cases and the \,foo case "work" because
> \, immediately followed by any escaped char or
> unescaped whitespace etc. works.  It's only \,
> followed by unescaped whitespace etc. that
> doesn't work.

`(a \,@ foo) would correspond to `(a . ,@foo), which would be illegal as
backquote expression.
Relying on the return value of that expression is calling for trouble.
I have the same opinion about this as above.


> Agreed, but I don't think just describing those
> expansions that way is sufficient.  For one
> thing, what's X?  Whether certain chars follow
> the comma immediately makes a difference.

The author is allowed to add more details about how the reader parses
that character of course.

> +;; When the Lisp reader sees `X it generates (\` X).
> +;; When it sees ,X it generates (\, X).  For ,@X it generates
> +;; (\,@ X).
>
> I don't think that's sufficient - see above.
> ,X and ,@X aren't handled the same, and it
> matters what X is.  X is not necessarily a new
> sexp.

For the file header this is enough IMO, it only clarifies what the
backquote macro gets to see and needs to handle.


Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > I see a difference between escaped comma and
> > escaped comma before @.  The above "are expanded
> > to" doesn't cover this, I think.  For example:
> > [...]
> > `(a \, foo) ; ==> (a 4 5)   - bad, the bug case
> 
> It's a logical consequence (although not an obvious one, but also not
> totally unobvious) of the fact that this expression is equivalent to
> `(a . ,foo).

Again, I think you're describing the implementation,
not why it _should_ behave this way.

IF
we assume (whether or not this bug will be fixed),
as I thought we agreed (?), that it makes more sense
for bare "\," within backquote, just as outside it,
and just as for UNbare escaped comma (e.g. "\,xyz"),
whether inside or outside backquote, to escape/undo
the backquote-meaning of ","

THEN
the sexp `(a \, foo) should just return (a \, foo),
a list of three symbols.

> > The \,@ cases and the \,foo case "work" because
> > \, immediately followed by any escaped char or
> > unescaped whitespace etc. works.  It's only \,
> > followed by unescaped whitespace etc. that
> > doesn't work.
> 
> `(a \,@ foo) would correspond to `(a . ,@foo), which 
> would be illegal as backquote expression.

Whether "illegal" or not, in Elisp it already
works as I would expect, and as I showed:

3. `(a \,@ foo) ; ==> (a \,@ foo) - good

Elisp correctly escapes the comma, to be a normal
symbol constituent, and the symbol name ",@" is
used.  Nothing in the backquote sexp calls for
any evaluation; the result is the same list of
3 symbols as the input to `.

> Relying on the return value of that expression 
> is calling for trouble.

How so?  On what basis do you think it's "illegal"?

> I have the same opinion about this as above.

Which is that it runs counter to what you say
the Elisp implementation gives?  But clearly the
Elisp implementation doesn't agree.

The only bug, IMO, is that an escaped isolated
comma is not actually escaped from the backquote
interpretation/behavior of comma.  There's no bug
for an escaped comma that's immediately followed
by other symbol chars (including @).
 
> > Agreed, but I don't think just describing those
> > expansions that way is sufficient.  For one
> > thing, what's X?  Whether certain chars follow
> > the comma immediately makes a difference.
> 
> The author is allowed to add more details about
> how the reader parses that character of course.

Author of the code?  The original comment?  My
mail?  I don't have a suggestion for the comment
improvement.  Unless it's to say what I've said
in this thread, perhaps showing some of the cases.
Or maybe just refer to this bug thread in the
comments.  And perhaps adding a FIXME to confirm
that there's a bug - if, that is, the Deciders
agree.

> > +;; When the Lisp reader sees `X it generates (\` X).
> > +;; When it sees ,X it generates (\, X).  For ,@X 
> > +;; it generates (\,@ X).
> >
> > I don't think that's sufficient - see above.
> > ,X and ,@X aren't handled the same, and it
> > matters what X is.  X is not necessarily a new
> > sexp.
> 
> For the file header this is enough IMO, it only clarifies 
> what the backquote macro gets to see and needs to handle.

The behavior depends on what X is allowed to be.
Is it something that parses (is read) as a separate
sexp?  Is it any sequence of chars?  Any sequence
of symbol chars?

I guess at this point we understand each other and
can just agree to disagree.  And I guess we agree
that it's unlikely that the bug will be fixed.  And
if it were fixed there may be some code that depends
on the bugged behavior (very unlikely, IMO).  And
(more likely), the fix might introduce new bugs, as
this seems tricky.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> THEN the sexp `(a \, foo) should just return (a \, foo),
> a list of three symbols.

I missed that you are still talking about this thing.  Yes, you can call
this a bug, but given that we agreed that it is unlikely that the
behavior will change, I somehow thought you were describing some other
thing.


> > Relying on the return value of that expression
> > is calling for trouble.
>
> How so?  On what basis do you think it's "illegal"?

On the basis these things are implemented in Elisp.  It is likely that
this will not change, so I think we should warn users to avoid using the
symbols with names ",@" and "," in backquote expressions directly (you
can still prevent them being handled with quoting of course, so it's not
forbidden to use them).  Because of this "bug" and that the behavior
depends on implementation details (also that of ",@", maybe it's also
broken in some cases or works only by luck, dunno).


> > > Agreed, but I don't think just describing those
> > > expansions that way is sufficient.  For one
> > > thing, what's X?  Whether certain chars follow
> > > the comma immediately makes a difference.
> >
> > The author is allowed to add more details about
> > how the reader parses that character of course.
>
> Author of the code?  The original comment?  My
> mail?

The author of the documentation improvement I suggested, Drew.  I don't
know who that will be.


> The behavior depends on what X is allowed to be.
> Is it something that parses (is read) as a separate
> sexp?  Is it any sequence of chars?  Any sequence
> of symbol chars?

It is obvious that the Lisp implementation of the "`" macro receives
symbolic expressions.  It's a Lisp library.  And only a comment in a
file header, not the manual.


> I guess at this point we understand each other and
> can just agree to disagree.

I think we actually agree on all things more or less but talked past
each other.


Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > The behavior depends on what X is allowed to be.
> > Is it something that parses (is read) as a
> > separate sexp?  Is it any sequence of chars?
> > Any sequence of symbol chars?
> 
> It is obvious that the Lisp implementation of 
> the "`" macro receives symbolic expressions.

This is about the Lisp reader.  When you write
",X" in that explanatory comment it's ambiguous
whether "X" is an arbitrary sequence of chars
or some Lisp sexp (read separately after reading
the ",").  ",abcd" is handled differently than
", abcd", as we've gone over several times now.

The bug (not "bug") is the handling of bare
"\,".  There's no such bug for "\,abcd" or even
"\,@" or "\,@<X>", for arbitrary <X> (arbitrary
text or a sexp).

> > I guess at this point we understand each
> > other and can just agree to disagree.
> 
> I think we actually agree on all things more
> or less but talked past each other.

Maybe.  I think rather that we understand each
other and probably agree about the facts.  But
I think we may have different interpretations
of the facts.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> > It is obvious that the Lisp implementation of
> > the "`" macro receives symbolic expressions.
>
> This is about the Lisp reader.  When you write
> ",X" in that explanatory comment it's ambiguous
> whether "X" is an arbitrary sequence of chars
> or some Lisp sexp (read separately after reading
> the ",").  ",abcd" is handled differently than
> ", abcd", as we've gone over several times now.

So you want to add to the text that X is an expression?  Or anything
else?  We surely don't want to explain the complete parsing process
there.  What do you suggest to write in that comment?


Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > > It is obvious that the Lisp implementation of
> > > the "`" macro receives symbolic expressions.
> >
> > This is about the Lisp reader.  When you write
> > ",X" in that explanatory comment it's ambiguous
> > whether "X" is an arbitrary sequence of chars
> > or some Lisp sexp (read separately after reading
> > the ",").  ",abcd" is handled differently than
> > ", abcd", as we've gone over several times now.
> 
> So you want to add to the text that X is an expression?  Or anything
> else?  We surely don't want to explain the complete parsing process
> there.  What do you suggest to write in that comment?

Dunno.  I didn't intend to use ",X" at all.  That
was from you.

I think that should particularly be pointed out in
comments is this bug: that "\," evaluates, just
like "," does, when inside backquote.  And it even
splices, like ",@" does.  This isn't obvious, even
if it might be a rare/corner case.

"\," that is not immediately followed by a symbol
char is handled as if you'd written ",@" instead.
One would (I would) expect a bare "\," to be read
as a symbol with no special behavior, just as
reading "abc" is (but remove all the double-quotes
when reading this, of course).

I'm sorry, but now I'm just repeating myself.  I
really don't have anything more/new to say about
this.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Michael Heerdegen]

Drew Adams <drew.adams@oracle.com> writes:

> Dunno.  I didn't intend to use ",X" at all.  That
> was from you.
>
> I think that should particularly be pointed out in
> comments is this bug: that "\," evaluates, just
> like "," does, when inside backquote.  And it even
> splices, like ",@" does.  This isn't obvious, even
> if it might be a rare/corner case.

So you think that somebody that is able to look up and understand the
Lisp part of the implementation of backquote expansion will - after
reading the comment that the reader construct expands like ,X -> (\, X)
- will _not_ understand that this implies that the symbol \, appearing in
a list inside a backquote expression has to be interpreted by backquote?

Here we disagree, I can't imagine such a person.


Michael.

bug#61281: “`(a \, b)” equals to “`(a . , b)”

[Drew Adams]

> > Dunno.  I didn't intend to use ",X" at all.  That
> > was from you.
> >
> > I think that should particularly be pointed out in
> > comments is this bug: that "\," evaluates, just
> > like "," does, when inside backquote.  And it even
> > splices, like ",@" does.  This isn't obvious, even
> > if it might be a rare/corner case.
> 
> So you think that somebody that is able to look up and understand the
> Lisp part of the implementation of backquote expansion will - after
> reading the comment that the reader construct expands like ,X -> (\, X)
> - will _not_ understand that this implies that the symbol \, appearing in
> a list inside a backquote expression has to be interpreted by backquote?
> 
> Here we disagree, I can't imagine such a person.

I really have nothing more to say on this.
I would just be repeating myself.  In any case,
we've agreed that it's unlikely that the bug
will be fixed.  And although I thought at one
time that you agreed the bug existed, I'm not
so sure now that you did.  And as, IIUC,
Andreas points out, clisp apparently just
substitutes a different char for "," internally
when it handles backquote, which just punts the
problem to a different corner case (different
char).  So I guess nothing will be done, and
maybe nothing reasonable can be done, for this.