bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Federico Tedin]

According to the Elisp docs
(https://www.gnu.org/software/emacs/manual/html_node/elisp/Dotted-Pair-Notation.html),
one should be able to evaluate e.g.:
    (. 1)
to:
    1

This works correctly in Emacs 28.1 However, in the emacs-29 branch
(bd07cec) this results in:
    *** Read error ***  Invalid read syntax: "."

Since the description in the docs are the same in the emacs-29 I
assume this is a bug.
I believe this may have been caused by the changes implementing the
nonrecursive Lisp reader.
I believe the fix would be roughly:
- in read0 (lread.c), when handling c == '.', ensure we handle not
only the case where the top of the read stack is RE_list but *also*
RE_list_start.
- if the top of the stack was effectively RE_list_start, then the top
of the stack needs to be manipulated somehow so that at the end of
read0 the correct value is returned, when emptying the stack.

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Eli Zaretskii]

> From: Federico Tedin <federicotedin@gmail.com>
> Date: Tue, 7 Mar 2023 02:14:02 +0100
> 
> According to the Elisp docs
> (https://www.gnu.org/software/emacs/manual/html_node/elisp/Dotted-Pair-Notation.html),
> one should be able to evaluate e.g.:
>     (. 1)
> to:
>     1
> 
> This works correctly in Emacs 28.1 However, in the emacs-29 branch
> (bd07cec) this results in:
>     *** Read error ***  Invalid read syntax: "."
> 
> Since the description in the docs are the same in the emacs-29 I
> assume this is a bug.
> I believe this may have been caused by the changes implementing the
> nonrecursive Lisp reader.
> I believe the fix would be roughly:
> - in read0 (lread.c), when handling c == '.', ensure we handle not
> only the case where the top of the read stack is RE_list but *also*
> RE_list_start.
> - if the top of the stack was effectively RE_list_start, then the top
> of the stack needs to be manipulated somehow so that at the end of
> read0 the correct value is returned, when emptying the stack.

Stefan and Mattias, can you please see how to fix this regression?

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Mattias Engdegård]

Eli, thank you for bringing this to our attention.

>> one should be able to evaluate e.g.:
>>    (. 1)
>> to:
>>    1

Federico, it would be very easy to change the reader into behaving that way and I'll do that if required, but before I or anyone else change code or docs, and above all much more important and interesting would be to hear exactly why it matters to you and how you were affected by this corner of the reader semantics.

Could be it that you saw the manual passage, decided to try it out -- which is good, we want more people to do that -- and observed that Emacs and the manual didn't agree on that point?

As far as I can tell while researching ahead of the previous changed, the documented (old) reader semantics was merely emergent behaviour of an under-constrained implementation, never a purposeful design for user convenience.

No other Lisp (Common Lisp, Scheme etc) implementation known to me provides such a reader 'feature', and no evidence of any use of it was found at the time. This is why your report is of such interest: did it actually break existing code, and if so, how exactly?

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Federico Tedin]

> Federico, it would be very easy to change the reader into behaving that way and I'll do that if required, but before I or anyone else change code or docs, and above all much more important and interesting would be to hear exactly why it matters to you and how you were affected by this corner of the reader semantics.
> Could be it that you saw the manual passage, decided to try it out -- which is good, we want more people to do that -- and observed that Emacs and the manual didn't agree on that point?
> As far as I can tell while researching ahead of the previous changed, the documented (old) reader semantics was merely emergent behaviour of an under-constrained implementation, never a purposeful design for user convenience.
> No other Lisp (Common Lisp, Scheme etc) implementation known to me provides such a reader 'feature', and no evidence of any use of it was found at the time. This is why your report is of such interest: did it actually break existing code, and if so, how exactly?

Hey Mattias. The reason I found this is actually a bit funny. I am
working on re-implementing the Elisp interpreter (and other parts of
Emacs) in Go, as a hobby/learning/fun project. My initial
implementation of the reader was based on the old version of Emacs'
reader (i.e. with read0 and read1 calling each other). Recently I
decided to port over the changes made to the Emacs reader
(nonrecursive reader), and immediately after I finished, I ran my test
suite and realized that some cases were failing. All of them were
related to the trailing dot notation, specifically when the initial
elements were omitted. The project itself can be found here
(https://github.com/federicotdn/pimacs), the reader is in read.go and
the test cases in interpreter_test.go. I implemented a fix in my code
in order to still be able to read these expressions (though it's a bit
ugly at the moment).

I agree that using the (. x) notation is not really needed, I haven't
used it in any of my code myself. And specially if no other Lisp
implements it, it would make sense to consider its removal! Though I
am not sure how these types of changes are handled in Emacs.

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Mattias Engdegård]

7 mars 2023 kl. 18.24 skrev Federico Tedin <federicotedin@gmail.com>:

> The reason I found this is actually a bit funny. I am
> working on re-implementing the Elisp interpreter (and other parts of
> Emacs) in Go, as a hobby/learning/fun project.

Delighted to hear about that! Best of luck, and do let us know about what else of interest you discover, whether you think they are bugs in Emacs or not.

The manual should include a formal syntax that describes what the Lisp reader accepts. It would benefit many users, not just those seeking to clone Emacs.

> I agree that using the (. x) notation is not really needed, I haven't
> used it in any of my code myself.

That's the general conclusion on this side as well: by disallowing the quirk we do the user a much greater service than by letting it pass, since any occurrence is far more likely to be a mistake than actually intended.

Let's correct the manual in Emacs 29 so that it corresponds to the code.

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Wed, 8 Mar 2023 11:37:27 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>,
>         62020@debbugs.gnu.org
> 
> Let's correct the manual in Emacs 29 so that it corresponds to the code.

Any specific correction you had in mind?

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Mattias Engdegård]

8 mars 2023 kl. 15.01 skrev Eli Zaretskii <eliz@gnu.org>:

>> Let's correct the manual in Emacs 29 so that it corresponds to the code.
> 
> Any specific correction you had in mind?

Just removing the part where we say that (. X) is valid, and perhaps clarify that the dot must follow (and precede) a value. That is, if documentation fixes are still fine for emacs-29.

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Fri, 10 Mar 2023 11:11:35 +0100
> Cc: federicotedin@gmail.com, monnier@iro.umontreal.ca, 62020@debbugs.gnu.org
> 
> 8 mars 2023 kl. 15.01 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> >> Let's correct the manual in Emacs 29 so that it corresponds to the code.
> > 
> > Any specific correction you had in mind?
> 
> Just removing the part where we say that (. X) is valid, and perhaps clarify that the dot must follow (and precede) a value. That is, if documentation fixes are still fine for emacs-29.

Documentation changes are always fine for a release branch, but I'd
prefer to discuss a specific patch, if you don't mind posting one.

Thanks.

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Mattias Engdegård]

[-- Attachment #1: Type: text/plain, Size: 302 bytes --]

10 mars 2023 kl. 12.55 skrev Eli Zaretskii <eliz@gnu.org>:

> Documentation changes are always fine for a release branch, but I'd
> prefer to discuss a specific patch, if you don't mind posting one.

Not at all. There's not much to see here -- the minimal change would just remove a paragraph:


[-- Attachment #2: reader-dotted-doc.diff --]
[-- Type: application/octet-stream, Size: 673 bytes --]

diff --git a/doc/lispref/objects.texi b/doc/lispref/objects.texi
index 99a3c073971..2fe7e6db560 100644
--- a/doc/lispref/objects.texi
+++ b/doc/lispref/objects.texi
@@ -1007,13 +1007,6 @@ Dotted Pair Notation
 @end example
 @end ifnottex
 
-  As a somewhat peculiar side effect of @code{(a b . c)} and
-@code{(a . (b . c))} being equivalent, for consistency this means
-that if you replace @code{b} here with the empty sequence, then it
-follows that @code{(a . c)} and @code{(a . ( . c))} are equivalent,
-too.  This also means that @code{( .  c)} is equivalent to @code{c},
-but this is seldom used.
-
 @node Association List Type
 @subsubsection Association List Type
 

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Sat, 11 Mar 2023 10:32:48 +0100
> Cc: federicotedin@gmail.com, monnier@iro.umontreal.ca, 62020@debbugs.gnu.org
> 
> 10 mars 2023 kl. 12.55 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> > Documentation changes are always fine for a release branch, but I'd
> > prefer to discuss a specific patch, if you don't mind posting one.
> 
> Not at all. There's not much to see here -- the minimal change would just remove a paragraph:
> 
> diff --git a/doc/lispref/objects.texi b/doc/lispref/objects.texi
> index 99a3c073971..2fe7e6db560 100644
> --- a/doc/lispref/objects.texi
> +++ b/doc/lispref/objects.texi
> @@ -1007,13 +1007,6 @@ Dotted Pair Notation
>  @end example
>  @end ifnottex
>  
> -  As a somewhat peculiar side effect of @code{(a b . c)} and
> -@code{(a . (b . c))} being equivalent, for consistency this means
> -that if you replace @code{b} here with the empty sequence, then it
> -follows that @code{(a . c)} and @code{(a . ( . c))} are equivalent,
> -too.  This also means that @code{( .  c)} is equivalent to @code{c},
> -but this is seldom used.
> -

OK, thanks.

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Mattias Engdegård]

11 mars 2023 kl. 13.19 skrev Eli Zaretskii <eliz@gnu.org>:

>>> Documentation changes are always fine for a release branch, but I'd
>>> prefer to discuss a specific patch, if you don't mind posting one.
>> 
>> Not at all. There's not much to see here -- the minimal change would just remove a paragraph:
> 
> OK, thanks.

Pushed to emacs-29.

bug#62020: Lisp reader: dotted pair notation not working when initial elements are omitted

[Federico Tedin]

> Delighted to hear about that! Best of luck, and do let us know about what else of interest you discover, whether you think they are bugs in Emacs or not.

Yes no problem, will do!

> >>> Documentation changes are always fine for a release branch, but I'd
> >>> prefer to discuss a specific patch, if you don't mind posting one.
> >>
> >> Not at all. There's not much to see here -- the minimal change would just remove a paragraph:
> >
> > OK, thanks.
>
> Pushed to emacs-29.

Thanks for the fix !