bug#63722: 29.0.91; (elisp) Skip to end #@00 does not work

bug#63722: 29.0.91; (elisp) Skip to end #@00 does not work

[richard newton]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

With emacs 29.0.91 pretest, the "skip to end" syntax #@00 no longer seems
to work. For example, with the three lines below in a buffer (eval-buffer)
gives the error - End of file during parsing. With previous emacs versions
it did not complain.

(setq testStr "zz")
#@00
should be ignored

In GNU Emacs 29.0.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.37, cairo version 1.16.0) of 2023-05-18 built on n26-hp
Windowing system distributor 'The X.Org Foundation', version 11.0.12201009
System Description: Debian GNU/Linux 12 (bookworm)

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
WEBP X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

[-- Attachment #2: Type: text/html, Size: 921 bytes --]

bug#63722: 29.0.91; (elisp) Skip to end #@00 does not work

[Eli Zaretskii]

> From: richard newton <richardn26@gmail.com>
> Date: Thu, 25 May 2023 16:19:42 +0200
> 
> With emacs 29.0.91 pretest, the "skip to end" syntax #@00 no longer seems to work. For example,
> with the three lines below in a buffer (eval-buffer) gives the error - End of file during parsing. With
> previous emacs versions it did not complain.
> 
> (setq testStr "zz")
> #@00
> should be ignored

Thanks.

Mattias, this is due to commit b903507b36, which fixed bug#55676.  It
introduced a subtle change in the behavior of read0 in this case.

Does the below look like the correct way of fixing this regression?

diff --git a/src/lread.c b/src/lread.c
index 342d367..6ee2829 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -3400,8 +3400,9 @@ read_bool_vector (Lisp_Object readcharfun)
 }
 
 /* Skip (and optionally remember) a lazily-loaded string
-   preceded by "#@".  */
-static void
+   preceded by "#@".  Return non-zero if we skip to EOB,
+   otherwise zero.  */
+static bool
 skip_lazy_string (Lisp_Object readcharfun)
 {
   ptrdiff_t nskip = 0;
@@ -3429,7 +3430,7 @@ skip_lazy_string (Lisp_Object readcharfun)
 	{
 	  /* #@00 means "skip to end" */
 	  skip_dyn_eof (readcharfun);
-	  return;
+	  return true;
 	}
     }
 
@@ -3476,6 +3477,8 @@ skip_lazy_string (Lisp_Object readcharfun)
   else
     /* Skip that many bytes.  */
     skip_dyn_bytes (readcharfun, nskip);
+
+  return false;
 }
 
 /* Given a lazy-loaded string designator VAL, return the actual string.
@@ -3933,7 +3936,8 @@ read0 (Lisp_Object readcharfun, bool locate_syms)
 	    /* #@NUMBER is used to skip NUMBER following bytes.
 	       That's used in .elc files to skip over doc strings
 	       and function definitions that can be loaded lazily.  */
-	    skip_lazy_string (readcharfun);
+	    if (skip_lazy_string (readcharfun))
+	      return unbind_to (base_pdl, Qnil);
 	    goto read_obj;
 
 	  case '$':

bug#63722: 29.0.91; (elisp) Skip to end #@00 does not work

[Mattias Engdegård]

25 maj 2023 kl. 20.44 skrev Eli Zaretskii <eliz@gnu.org>:

>> With emacs 29.0.91 pretest, the "skip to end" syntax #@00 no longer seems to work. For example,
>> with the three lines below in a buffer (eval-buffer) gives the error - End of file during parsing. With
>> previous emacs versions it did not complain.
>> 
>> (setq testStr "zz")
>> #@00
>> should be ignored

Thank you for reporting this. The undocumented #@00 behaviour has indeed been co-opted by parts of the community as an 'pretend end-of-file occurs here' mechanism, but it actually works as if only the value `nil` were left in the buffer. This is fine in files containing Lisp source where a top-level nil has no effect, but could cause trouble if used in Lisp data files.

Obviously we are going to keep the old reader behaviour for compatibility for now since there is little to gain from not doing so.

Ideally we should stop handling comments and whitespace in readevalloop because the actual reader (read0) is perfectly capable of doing that. We need a way to distinguish 'EOF before anything could be read' from 'EOF when we have read an incomplete value', both of which currently signal `end-of-file`. Not sure how best to do that in a compatible way, however.

> Does the below look like the correct way of fixing this regression?

Thank you and yes, roughly speaking -- a fix has now been pushed to emacs-29, and I'm closing the bug.

bug#63722: 29.0.91; (elisp) Skip to end #@00 does not work

[Michael Albinus]

Mattias Engdegård <mattiase@acm.org> writes:

Hi Mattias,

> The undocumented #@00 behaviour has indeed been co-opted by parts of
> the community as an 'pretend end-of-file occurs here' mechanism,

Should this be documented then?

Best regards, Michael.

bug#63722: 29.0.91; (elisp) Skip to end #@00 does not work

[Mattias Engdegård]

26 maj 2023 kl. 11.43 skrev Michael Albinus <michael.albinus@gmx.de>:

>> The undocumented #@00 behaviour has indeed been co-opted by parts of
>> the community as an 'pretend end-of-file occurs here' mechanism,
> 
> Should this be documented then?

I don't think we should do that right now -- the behaviour (read as nil) is not ideal and should not be made permanent; we still have the option to change it (but not in emacs 29, of course).

We could also come up with a less obscure replacement syntax, given that there may be a genuine need.

bug#63722: 29.0.91; (elisp) Skip to end #@00 does not work

[Michael Albinus]

Mattias Engdegård <mattiase@acm.org> writes:

Hi Mattias,

>>> The undocumented #@00 behaviour has indeed been co-opted by parts of
>>> the community as an 'pretend end-of-file occurs here' mechanism,
>> 
>> Should this be documented then?
>
> I don't think we should do that right now -- the behaviour (read as
> nil) is not ideal and should not be made permanent; we still have the
> option to change it (but not in emacs 29, of course).
>
> We could also come up with a less obscure replacement syntax, given
> that there may be a genuine need.

Everything right. But it is used meanwhile in the wild ... perhaps we
should document then, that this usage is deprecated?

Best regards, Michael.

bug#63722: 29.0.91; (elisp) Skip to end #@00 does not work

[Eli Zaretskii]

> Cc: richardn26@gmail.com, 63722@debbugs.gnu.org
> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Fri, 26 May 2023 11:56:12 +0200
> 
> Mattias Engdegård <mattiase@acm.org> writes:
> 
> >>> The undocumented #@00 behaviour has indeed been co-opted by parts of
> >>> the community as an 'pretend end-of-file occurs here' mechanism,
> >> 
> >> Should this be documented then?
> >
> > I don't think we should do that right now -- the behaviour (read as
> > nil) is not ideal and should not be made permanent; we still have the
> > option to change it (but not in emacs 29, of course).
> >
> > We could also come up with a less obscure replacement syntax, given
> > that there may be a genuine need.
> 
> Everything right. But it is used meanwhile in the wild ... perhaps we
> should document then, that this usage is deprecated?

I see no reason to deprecate it, let alone remove it, FWIW.

bug#63722: 29.0.91; (elisp) Skip to end #@00 does not work

[Mattias Engdegård]

26 maj 2023 kl. 12.36 skrev Eli Zaretskii <eliz@gnu.org>:

>> Everything right. But it is used meanwhile in the wild ... perhaps we
>> should document then, that this usage is deprecated?
> 
> I see no reason to deprecate it, let alone remove it, FWIW.

I prefer neither documenting nor deprecating it now.
If someone asks me on the record, I will neither confirm nor deny its existence.