bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Philipp Stephani]

Insert into *scratch*:

(defun foo () (syntax-propertize-rules ((concat ""))))

Now try to instrument this using C-u C-M-x.  This will lead to an error
"Eager macro-expansion failure":

Debugger entered--Lisp error: (error "Eager macro-expansion failure: (void-function edeb...")
  signal(error ("Eager macro-expansion failure: (void-function edeb..."))
  error("Eager macro-expansion failure: %S" (void-function edebug-after))
  internal-macroexpand-for-load((setq elisp--eval-defun-result (let ((print-level nil) (print-length nil)) (progn (defalias 'foo #'(lambda nil (edebug-enter ... ... ...)))))) t)
  eval-region(146 200 nil #f(compiled-function (ignore) #<bytecode -0x5a01b3e4778c94e>))  ; Reading at buffer position 186
  elisp--eval-defun()
  #f(compiled-function (edebug-it) "Evaluate top-level form around point and instrument it if EDEBUG-IT is non-nil.\nInteractively, EDEBUG-IT is the prefix argument.\nIf `edebug-all-defs' is non-nil, that inverts the meaning of EDEBUG-IT\nand the prefix argument: this function will instrument the form\nunless EDEBUG-IT is non-nil.  The command `edebug-all-defs' toggles\nthe value of the variable `edebug-all-defs'.\n\nIf point isn't in a top-level form, evaluate the first top-level\nform after point.  If there is no top-level form after point,\nevaluate the first preceding top-level form.\n\nIf the current defun is actually a call to `defvar' or `defcustom',\nevaluating it this way resets the variable using its initial value\nexpression (using the defcustom's :set function if there is one), even\nif the variable already has some other value.  (Normally `defvar' and\n`defcustom' do not alter the value if there already is one.)  In an\nanalogous way, evaluating a `defface' overrides any customizations of\nthe face, so that it becomes defined exactly as the `defface' expression\nsays.\n\nIf `eval-expression-debug-on-error' is non-nil, which is the default,\nthis command arranges for all errors to enter the debugger.\n\nIf acting on a `defun' for FUNCTION, and the function was\ninstrumented, `Edebug: FUNCTION' is printed in the echo area.  If not\ninstrumented, just FUNCTION is printed.\n\nIf not acting on a `defun', the result of evaluation is displayed in\nthe echo area.  This display is controlled by the variables\n`eval-expression-print-length' and `eval-expression-print-level',\nwhich see." (interactive "P") #<bytecode 0xb135734182b2df2>)(nil)
  edebug--eval-defun(#f(compiled-function (edebug-it) "Evaluate top-level form around point and instrument it if EDEBUG-IT is non-nil.\nInteractively, EDEBUG-IT is the prefix argument.\nIf `edebug-all-defs' is non-nil, that inverts the meaning of EDEBUG-IT\nand the prefix argument: this function will instrument the form\nunless EDEBUG-IT is non-nil.  The command `edebug-all-defs' toggles\nthe value of the variable `edebug-all-defs'.\n\nIf point isn't in a top-level form, evaluate the first top-level\nform after point.  If there is no top-level form after point,\nevaluate the first preceding top-level form.\n\nIf the current defun is actually a call to `defvar' or `defcustom',\nevaluating it this way resets the variable using its initial value\nexpression (using the defcustom's :set function if there is one), even\nif the variable already has some other value.  (Normally `defvar' and\n`defcustom' do not alter the value if there already is one.)  In an\nanalogous way, evaluating a `defface' overrides any customizations of\nthe face, so that it becomes defined exactly as the `defface' expression\nsays.\n\nIf `eval-expression-debug-on-error' is non-nil, which is the default,\nthis command arranges for all errors to enter the debugger.\n\nIf acting on a `defun' for FUNCTION, and the function was\ninstrumented, `Edebug: FUNCTION' is printed in the echo area.  If not\ninstrumented, just FUNCTION is printed.\n\nIf not acting on a `defun', the result of evaluation is displayed in\nthe echo area.  This display is controlled by the variables\n`eval-expression-print-length' and `eval-expression-print-level',\nwhich see." (interactive "P") #<bytecode 0xb135734182b2df2>) (4))
  apply(edebug--eval-defun #f(compiled-function (edebug-it) "Evaluate top-level form around point and instrument it if EDEBUG-IT is non-nil.\nInteractively, EDEBUG-IT is the prefix argument.\nIf `edebug-all-defs' is non-nil, that inverts the meaning of EDEBUG-IT\nand the prefix argument: this function will instrument the form\nunless EDEBUG-IT is non-nil.  The command `edebug-all-defs' toggles\nthe value of the variable `edebug-all-defs'.\n\nIf point isn't in a top-level form, evaluate the first top-level\nform after point.  If there is no top-level form after point,\nevaluate the first preceding top-level form.\n\nIf the current defun is actually a call to `defvar' or `defcustom',\nevaluating it this way resets the variable using its initial value\nexpression (using the defcustom's :set function if there is one), even\nif the variable already has some other value.  (Normally `defvar' and\n`defcustom' do not alter the value if there already is one.)  In an\nanalogous way, evaluating a `defface' overrides any customizations of\nthe face, so that it becomes defined exactly as the `defface' expression\nsays.\n\nIf `eval-expression-debug-on-error' is non-nil, which is the default,\nthis command arranges for all errors to enter the debugger.\n\nIf acting on a `defun' for FUNCTION, and the function was\ninstrumented, `Edebug: FUNCTION' is printed in the echo area.  If not\ninstrumented, just FUNCTION is printed.\n\nIf not acting on a `defun', the result of evaluation is displayed in\nthe echo area.  This display is controlled by the variables\n`eval-expression-print-length' and `eval-expression-print-level',\nwhich see." (interactive "P") #<bytecode 0xb135734182b2df2>) (4))
  eval-defun((4))
  funcall-interactively(eval-defun (4))
  call-interactively(eval-defun nil nil)
  command-execute(eval-defun)

This fails on master and emacs-29, but not on emacs-28, so it's a
regression that I think should be fixed before releasing Emacs 29.


In GNU Emacs 30.0.50 (build 27, x86_64-pc-linux-gnu, GTK+ Version
 3.24.37, cairo version 1.16.0) of 2023-07-04
Repository revision: b8811f854ff2ea97c929d9f3f1674258ffc14cc1
Repository branch: syntax-propertize-rules-edebug
Windowing system distributor 'The X.Org Foundation', version 11.0.12101007
System Description: Debian GNU/Linux rodete

Configured using:
 'configure --enable-gcc-warnings=warn-only
 --enable-gtk-deprecation-warnings --without-pop --with-mailutils
 --enable-checking=all --enable-check-lisp-object-type --with-modules
 'CFLAGS=-O0 -ggdb3''

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG SECCOMP
SOUND THREADS TIFF TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM XINPUT2 XPM
GTK3 ZLIB

Important settings:
  value of $LC_MONETARY: en_DK.UTF-8
  value of $LC_NUMERIC: en_DK.UTF-8
  value of $LC_TIME: en_DK.UTF-8
  value of $LANG: en_DK.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug phst skeleton pcase ffap thingatpt url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util url-parse auth-source cl-seq
eieio eieio-core cl-macs json map byte-opt gv bytecomp byte-compile
url-vars rx message sendmail mailcap yank-media dired dired-loaddefs
rfc822 mml mml-sec password-cache epa derived epg rfc6068 epg-config
gnus-util time-date mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils
gmm-utils mailheader gnutls puny elp dbus xml subr-x compile
text-property-search comint ansi-osc ansi-color ring cl-loaddefs cl-lib
rmc iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo gtk x-toolkit xinput2 x multi-tty move-toolbar
make-network-process emacs)

Memory information:
((conses 16 70027 11803) (symbols 48 8340 0) (strings 32 23360 2773)
 (string-bytes 1 713597) (vectors 16 16358)
 (vector-slots 8 219441 18429) (floats 8 45 47) (intervals 56 257 0)
 (buffers 976 11))

-- 
Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich.  Falls Sie diese fälschlicherweise erhalten haben
sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie
alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail
an die falsche Person gesendet wurde.

This e-mail is confidential.  If you received this communication by mistake,
please don’t forward it to anyone else, please erase all copies and
attachments, and please let me know that it has gone to the wrong person.

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Tue, 04 Jul 2023 15:35:01 +0200
> 
> This fails on master and emacs-29, but not on emacs-28, so it's a
> regression that I think should be fixed before releasing Emacs 29.

The part about this not being a problem in Emacs 28 is not true: if I
try this in Emacs 28.2, I see in *Messages*:

  Edebug: foo
  Eager macro-expansion failure: (void-function edebug-after)

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Philipp Stephani]

Am Di., 4. Juli 2023 um 15:42 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Tue, 04 Jul 2023 15:35:01 +0200
> >
> > This fails on master and emacs-29, but not on emacs-28, so it's a
> > regression that I think should be fixed before releasing Emacs 29.
>
> The part about this not being a problem in Emacs 28 is not true: if I
> try this in Emacs 28.2, I see in *Messages*:
>
>   Edebug: foo
>   Eager macro-expansion failure: (void-function edebug-after)

Ah, indeed. So it looks like the actual change between 28 and 29 is
that macroexpansion failures are now hard errors.
The easiest way to fix this is to replace the first `form' in the
edebug spec with `sexp'.

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Tue, 4 Jul 2023 16:06:01 +0200
> Cc: 64459@debbugs.gnu.org
> 
> Am Di., 4. Juli 2023 um 15:42 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
> >
> > > From: Philipp Stephani <p.stephani2@gmail.com>
> > > Date: Tue, 04 Jul 2023 15:35:01 +0200
> > >
> > > This fails on master and emacs-29, but not on emacs-28, so it's a
> > > regression that I think should be fixed before releasing Emacs 29.
> >
> > The part about this not being a problem in Emacs 28 is not true: if I
> > try this in Emacs 28.2, I see in *Messages*:
> >
> >   Edebug: foo
> >   Eager macro-expansion failure: (void-function edebug-after)
> 
> Ah, indeed. So it looks like the actual change between 28 and 29 is
> that macroexpansion failures are now hard errors.
> The easiest way to fix this is to replace the first `form' in the
> edebug spec with `sexp'.

Adding Stefan.

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> (defun foo () (syntax-propertize-rules ((concat ""))))
>
> Now try to instrument this using C-u C-M-x.  This will lead to an error
> "Eager macro-expansion failure":
>
> Debugger entered--Lisp error: (error "Eager macro-expansion failure: (void-function edeb...")
>   signal(error ("Eager macro-expansion failure: (void-function edeb..."))

I think the patch below should fix this problem.
Can you confirm that it fixes it for you in a more realistic setting?


        Stefan


diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
index d610fa005cc..002a24af18b 100644
--- a/lisp/emacs-lisp/syntax.el
+++ b/lisp/emacs-lisp/syntax.el
@@ -249,11 +249,12 @@ syntax-propertize-rules
 Note: There may be at most nine back-references in the REGEXPs of
 all RULES in total."
   (declare (debug (&rest &or symbolp    ;FIXME: edebug this eval step.
-                         (form &rest
-                               (numberp
-                                [&or stringp ;FIXME: Use &wrap
-                                     ("prog1" [&or stringp def-form] def-body)
-                                     def-form])))))
+                         (def-form
+                          &rest
+                          (numberp
+                           [&or stringp ;FIXME: Use &wrap
+                                ("prog1" [&or stringp form] def-body)
+                                form])))))
   (let ((newrules nil))
     (while rules
       (if (symbolp (car rules))

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
> index d610fa005cc..002a24af18b 100644
> --- a/lisp/emacs-lisp/syntax.el
> +++ b/lisp/emacs-lisp/syntax.el
> @@ -249,11 +249,12 @@ syntax-propertize-rules
>  Note: There may be at most nine back-references in the REGEXPs of
>  all RULES in total."
>    (declare (debug (&rest &or symbolp    ;FIXME: edebug this eval step.
> -                         (form &rest
> -                               (numberp
> -                                [&or stringp ;FIXME: Use &wrap
> -                                     ("prog1" [&or stringp def-form] def-body)
> -                                     def-form])))))
> +                         (def-form
> +                          &rest
> +                          (numberp
> +                           [&or stringp ;FIXME: Use &wrap
> +                                ("prog1" [&or stringp form] def-body)
> +                                form])))))
>    (let ((newrules nil))
>      (while rules
>        (if (symbolp (car rules))

No, this one introduces a regression.  Try that one:


        Stefan


diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
index d610fa005cc..41db2981d8c 100644
--- a/lisp/emacs-lisp/syntax.el
+++ b/lisp/emacs-lisp/syntax.el
@@ -249,11 +249,12 @@ syntax-propertize-rules
 Note: There may be at most nine back-references in the REGEXPs of
 all RULES in total."
   (declare (debug (&rest &or symbolp    ;FIXME: edebug this eval step.
-                         (form &rest
-                               (numberp
-                                [&or stringp ;FIXME: Use &wrap
-                                     ("prog1" [&or stringp def-form] def-body)
-                                     def-form])))))
+                         (def-form
+                          &rest
+                          (numberp
+                           [&or stringp ;FIXME: Use &wrap
+                                ("prog1" [&or stringp def-form] def-body)
+                                def-form])))))
   (let ((newrules nil))
     (while rules
       (if (symbolp (car rules))

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Philipp Stephani]

Am Mi., 5. Juli 2023 um 01:46 Uhr schrieb Stefan Monnier
<monnier@iro.umontreal.ca>:
>
> > diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
> > index d610fa005cc..002a24af18b 100644
> > --- a/lisp/emacs-lisp/syntax.el
> > +++ b/lisp/emacs-lisp/syntax.el
> > @@ -249,11 +249,12 @@ syntax-propertize-rules
> >  Note: There may be at most nine back-references in the REGEXPs of
> >  all RULES in total."
> >    (declare (debug (&rest &or symbolp    ;FIXME: edebug this eval step.
> > -                         (form &rest
> > -                               (numberp
> > -                                [&or stringp ;FIXME: Use &wrap
> > -                                     ("prog1" [&or stringp def-form] def-body)
> > -                                     def-form])))))
> > +                         (def-form
> > +                          &rest
> > +                          (numberp
> > +                           [&or stringp ;FIXME: Use &wrap
> > +                                ("prog1" [&or stringp form] def-body)
> > +                                form])))))
> >    (let ((newrules nil))
> >      (while rules
> >        (if (symbolp (car rules))
>
> No, this one introduces a regression.  Try that one:
>

Yeah, that works at least in the cases I've tested. Thanks!

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Eli Zaretskii]

> Cc: 64459@debbugs.gnu.org
> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Wed, 5 Jul 2023 11:34:06 +0200
> 
> Am Mi., 5. Juli 2023 um 01:46 Uhr schrieb Stefan Monnier
> <monnier@iro.umontreal.ca>:
> >
> > > diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
> > > index d610fa005cc..002a24af18b 100644
> > > --- a/lisp/emacs-lisp/syntax.el
> > > +++ b/lisp/emacs-lisp/syntax.el
> > > @@ -249,11 +249,12 @@ syntax-propertize-rules
> > >  Note: There may be at most nine back-references in the REGEXPs of
> > >  all RULES in total."
> > >    (declare (debug (&rest &or symbolp    ;FIXME: edebug this eval step.
> > > -                         (form &rest
> > > -                               (numberp
> > > -                                [&or stringp ;FIXME: Use &wrap
> > > -                                     ("prog1" [&or stringp def-form] def-body)
> > > -                                     def-form])))))
> > > +                         (def-form
> > > +                          &rest
> > > +                          (numberp
> > > +                           [&or stringp ;FIXME: Use &wrap
> > > +                                ("prog1" [&or stringp form] def-body)
> > > +                                form])))))
> > >    (let ((newrules nil))
> > >      (while rules
> > >        (if (symbolp (car rules))
> >
> > No, this one introduces a regression.  Try that one:
> >
> 
> Yeah, that works at least in the cases I've tested. Thanks!

Stefan, this is for master, right?

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> > > diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
>> > > index d610fa005cc..002a24af18b 100644
>> > > --- a/lisp/emacs-lisp/syntax.el
>> > > +++ b/lisp/emacs-lisp/syntax.el
>> > > @@ -249,11 +249,12 @@ syntax-propertize-rules
>> > >  Note: There may be at most nine back-references in the REGEXPs of
>> > >  all RULES in total."
>> > >    (declare (debug (&rest &or symbolp    ;FIXME: edebug this eval step.
>> > > -                         (form &rest
>> > > -                               (numberp
>> > > -                                [&or stringp ;FIXME: Use &wrap
>> > > -                                     ("prog1" [&or stringp def-form] def-body)
>> > > -                                     def-form])))))
>> > > +                         (def-form
>> > > +                          &rest
>> > > +                          (numberp
>> > > +                           [&or stringp ;FIXME: Use &wrap
>> > > +                                ("prog1" [&or stringp form] def-body)
>> > > +                                form])))))
>> > >    (let ((newrules nil))
>> > >      (while rules
>> > >        (if (symbolp (car rules))
>> >
>> > No, this one introduces a regression.  Try that one:
>> >
>> 
>> Yeah, that works at least in the cases I've tested. Thanks!
>
> Stefan, this is for master, right?

Yes: if you want it for `emacs-29`, I do think it's perfectly safe (it
just replaces the one `form` with `def-form`), but I don't see any
urgency here.


        Stefan

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Philipp Stephani <p.stephani2@gmail.com>,  64459@debbugs.gnu.org
> Date: Wed, 05 Jul 2023 08:43:15 -0400
> 
> >> > > diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
> >> > > index d610fa005cc..002a24af18b 100644
> >> > > --- a/lisp/emacs-lisp/syntax.el
> >> > > +++ b/lisp/emacs-lisp/syntax.el
> >> > > @@ -249,11 +249,12 @@ syntax-propertize-rules
> >> > >  Note: There may be at most nine back-references in the REGEXPs of
> >> > >  all RULES in total."
> >> > >    (declare (debug (&rest &or symbolp    ;FIXME: edebug this eval step.
> >> > > -                         (form &rest
> >> > > -                               (numberp
> >> > > -                                [&or stringp ;FIXME: Use &wrap
> >> > > -                                     ("prog1" [&or stringp def-form] def-body)
> >> > > -                                     def-form])))))
> >> > > +                         (def-form
> >> > > +                          &rest
> >> > > +                          (numberp
> >> > > +                           [&or stringp ;FIXME: Use &wrap
> >> > > +                                ("prog1" [&or stringp form] def-body)
> >> > > +                                form])))))
> >> > >    (let ((newrules nil))
> >> > >      (while rules
> >> > >        (if (symbolp (car rules))
> >> >
> >> > No, this one introduces a regression.  Try that one:
> >> >
> >> 
> >> Yeah, that works at least in the cases I've tested. Thanks!
> >
> > Stefan, this is for master, right?
> 
> Yes: if you want it for `emacs-29`, I do think it's perfectly safe (it
> just replaces the one `form` with `def-form`), but I don't see any
> urgency here.

I don't see any urgency, either, since this problem existed at least
since Emacs 28.

So please install on master, and thanks.

bug#64459: 30.0.50; Edebug can't instrument certain syntax-propertize-rules forms

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> So please install on master, and thanks.

Installed, thanks,


        Setfan