all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* bug#54079: 29.0.50; Method dispatching eratically fails
@ 2022-02-21  0:12 Michael Heerdegen
  2022-02-21  1:14 ` Michael Heerdegen
  2022-02-21  3:16 ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  0 siblings, 2 replies; 36+ messages in thread
From: Michael Heerdegen @ 2022-02-21  0:12 UTC (permalink / raw)
  To: 54079


Hello,

this happens on trunk.

I extended register.el with new register content types.  This involves a
lot of method definitions like

  (cl-defmethod register-val-jump-to ((val TYPE) arg))

where TYPE refers to some of my self defined container types for
register contents.

I byte compile my init file (where these definitions live).

My problem: the dispatching of these methods is often erratically
broken.  The methods then are not used although their type should match.
The according test for VAL being of the TYPE, e.g. by using the
according type predicate, succeeds.

This happens randomly.  Sometimes it's broken from the beginning of a
session, sometimes not, sometimes it breaks later while working.  When
it happens, sometimes not all types are affected.  Sometimes reloading
my init file fixes the problem, sometimes not.  I don't see a clear
pattern.

Maybe there is a relation to the symbol-with-position patch (just
guessing)?

That's all I can say.  So far I failed to reproduce something like this
in emacs -Q, I don't have a recipe, and maybe it's even my fault, I have
no clue.  Maybe the behavior depends on what was loaded (and how, and in
which order) before the file is compiled.

TIA,

Michael.








^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-02-21  0:12 bug#54079: 29.0.50; Method dispatching eratically fails Michael Heerdegen
@ 2022-02-21  1:14 ` Michael Heerdegen
  2022-03-04 16:12   ` Lars Ingebrigtsen
  2022-02-21  3:16 ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  1 sibling, 1 reply; 36+ messages in thread
From: Michael Heerdegen @ 2022-02-21  1:14 UTC (permalink / raw)
  To: 54079

Michael Heerdegen <michael_heerdegen@web.de> writes:

> That's all I can say.  So far I failed to reproduce something like this
> in emacs -Q, I don't have a recipe, and maybe it's even my fault, I have
> no clue.  Maybe the behavior depends on what was loaded (and how, and in
> which order) before the file is compiled.

Or even just recompiling the file has an impact, I do that often.

BTW, I sometimes saw symbols with positions in the `symbol-function'
binding of related stuff - dunno if that is normal or even part of the
problem.

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-02-21  0:12 bug#54079: 29.0.50; Method dispatching eratically fails Michael Heerdegen
  2022-02-21  1:14 ` Michael Heerdegen
@ 2022-02-21  3:16 ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-02-21  4:21   ` Michael Heerdegen
  1 sibling, 1 reply; 36+ messages in thread
From: Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-02-21  3:16 UTC (permalink / raw)
  To: Michael Heerdegen; +Cc: 54079

Michael Heerdegen [2022-02-21 01:12:56] wrote:
> I extended register.el with new register content types.  This involves a
> lot of method definitions like
>
>   (cl-defmethod register-val-jump-to ((val TYPE) arg))
>
> where TYPE refers to some of my self defined container types for
> register contents.

Can you show the actual code (the bodies of the `cl-defmethod`s aren't
important, but the TYPE part could, or maybe if they're inside something
else (maybe an `eval-and-compile` or something like that?))?

How is the type defined?  `cl-defstruct`?  `defclass`?

> My problem: the dispatching of these methods is often erratically
> broken.  The methods then are not used although their type should match.
> The according test for VAL being of the TYPE, e.g. by using the
> according type predicate, succeeds.

What does `type-of` return on VAL?


        Stefan






^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-02-21  3:16 ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-02-21  4:21   ` Michael Heerdegen
  2022-02-22 23:55     ` Michael Heerdegen
  0 siblings, 1 reply; 36+ messages in thread
From: Michael Heerdegen @ 2022-02-21  4:21 UTC (permalink / raw)
  To: 54079; +Cc: monnier

Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> Can you show the actual code (the bodies of the `cl-defmethod`s aren't
> important, but the TYPE part could,

The TYPE parts all simply look like

  (val my-command-register)

where my-command-register is defined with `cl-defstruct'.

> or maybe if they're inside something
> else (maybe an `eval-and-compile` or something like that?))?

Some definitions are wrapped in a lambda and pushed to
`after-init-hook' using this macro:

(defmacro after-init (&rest args)
  `(add-hook 'after-init-hook (lambda () ,@args) 'append))

The others things are top-level definitions.

> What does `type-of` return on VAL?

Will answer that when it happens the next time.

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-02-21  4:21   ` Michael Heerdegen
@ 2022-02-22 23:55     ` Michael Heerdegen
  2022-02-23  0:27       ` Michael Heerdegen
  0 siblings, 1 reply; 36+ messages in thread
From: Michael Heerdegen @ 2022-02-22 23:55 UTC (permalink / raw)
  To: 54079; +Cc: monnier

Michael Heerdegen <michael_heerdegen@web.de> writes:

> > What does `type-of` return on VAL?

`type-of' returns the expected value, the symbol which is the name of
the class.  The return value is not different whether it works or not.

This for example returns t:

  (eq (type-of (cdr (assoc ?A register-alist))) 'my-window+pos-register)

OTOH, in the *trace-output* I am currently seeing this:

1 -> (my-register-val-describe-nicely #s(#<symbol my-window+pos-register at 704865> :#<symbol window at 704894> #<window 173> :#<symbol pos at 704901> #<marker at 699485 in .gnu-emacs.el>))      00:41:15.227
  (my-register-val-describe-nicely #s(#<symbol my-window+pos-register at 704865> :#<symbol window at 704894> #<window 173> :#<symbol pos at 704901> #<marker at 699485 in .gnu-emacs.el>))
  (register-val-describe #s(#<symbol my-window+pos-register at 704865> :#<symbol window at 704894> #<window 173> :#<symbol pos at 704901> #<marker at 699485 in .gnu-emacs.el>) nil)
  (describe-register-1 65)
  (register-describe-oneline 65)
  (my-register-preview (65 . #s(#<symbol my-window+pos-register at 704865> :#<symbol window at 704894> #<window 173> :#<symbol pos at 704901> #<marker at 699485 in .gnu-emacs.el>)))
  (my-register-preview-function (65 . #s(#<symbol my-window+pos-register at 704865> :#<symbol window at 704894> #<window 173> :#<symbol pos at 704901> #<marker at 699485 in .gnu-emacs.el>)))
  (my-register-preview--around-ad #f(compiled-function (buffer &optional show-empty) "Pop up a window to show register preview in BUFFER.\nIf SHOW-EMPTY is non-nil show the window even if no registers.\nFormat of each entry is controlled by the variable `register-preview-function'." #<bytecode 0x1e3c7c22968242f1>) "*Register Preview*")
  (register-preview "*Register Preview*")
  (timer-event-handler [t 25109 29850 770721 nil #f(compiled-function () #<bytecode 0x884d14a4ec654c5>) nil nil 4000 nil])
  (read-key #("Jump to register: " 0 18 (face minibuffer-prompt)))
  (register-read-with-preview "Jump to register: ")
  (command-execute jump-to-register)
1 <- my-register-val-describe-nicely: nil

Definitions are

  (cl-defstruct my-window+pos-register "Doc" window pos)

and

(cl-defgeneric my-register-val-describe-nicely (val)
  "Alternative to `register-val-describe' - the return value is used from this one."
  ;; This should support colors at least for some preview functions
  (ignore val)
  nil)

(cl-defmethod my-register-val-describe-nicely ((val my-window+pos-register))
  (my-register-val-describe-nicely (my-window+pos-register-pos val)))

The default implementation is used and the matching implementation is
ignored.

I have now re-evaluated some of the definitions.  Sometimes this helps -
ATM it didn't.  It made the symbols with positions disappear in the
trace however, now lines are traced like

(my-register-val-describe-nicely #s(my-window+pos-register :window #<window 263 on .gnu-emacs.el> :pos #<marker at 2820 in .gnu-emacs.el>))

So I'm not sure if symbols with positions are related.

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-02-22 23:55     ` Michael Heerdegen
@ 2022-02-23  0:27       ` Michael Heerdegen
  2022-02-23  0:47         ` Michael Heerdegen
  0 siblings, 1 reply; 36+ messages in thread
From: Michael Heerdegen @ 2022-02-23  0:27 UTC (permalink / raw)
  To: 54079; +Cc: monnier

Michael Heerdegen <michael_heerdegen@web.de> writes:

> So I'm not sure if symbols with positions are related.

Dunno if and how this is related: I found out that when I byte-compile
cl-generic.el in current Emacs session, trying to M-x gnus suddenly
gives me:

| Debugger entered--Lisp error: (invalid-function #<symbol let at 15662>)
|   (#<symbol let at 15662> ((#<symbol plain-args at 15668> nil) (#<symbol specializers at 15694> nil) (#<symbol mandatory at 15723> #<symbol t at 15733>)) (let ((--dolist-tail-- #<symbol args at 15756>)) (while --dolist-tail-- (let ((#<symbol arg at 15752> (car --dolist-tail--))) (setq #<symbol plain-args at 16815> (cons (cond (... ...) (... ...) (t ...)) #<symbol plain-args at 16815>)) (setq --dolist-tail-- (cdr --dolist-tail--))))) (#<symbol cons at 16835> (#<symbol nreverse at 16841> #<symbol specializers at 16850>) (#<symbol nreverse at 16877> (#<symbol delq at 16887> nil #<symbol plain-args at 16896>))))
|   (cl--generic-split-args (engine artlist criteria))
|   (cl-generic-define gnus-search-grep-search (engine artlist criteria) nil)
|   (byte-code "\300\301\302\301\303\304#\305#\210\306\301\304\307\304\310%\210\300\311\312\313!\"\210\300\314\315\313!\"\210\300\316\314\"\210\317\316\320\321#\210\322\313\323\314#\324\313..." [defalias gnus-search-grep-search cl-generic-define (engine artlist criteria) nil "Run a secondary grep search over a list of prelimi..." cl-generic-define-method ((engine gnus-search-grep) artlist criteria) #f(compiled-function (engine artlist criteria) #<bytecode -0x1d311dd1a946ba32>) gnus-search-process-p eieio-make-class-predicate gnus-search-process gnus-search-process--eieio-childp eieio-make-child-predicate gnus-search-process-child-p make-obsolete "use (cl-typep ... 'gnus-search-process) instead" "25.1" define-symbol-prop cl-deftype-satisfies eieio-defclass-internal ((proc-buffer :initarg :p
 roc-buffer :type buffer :documentation "A temporary buffer this engine uses for its\n    se...")) (:abstract t :documentation "A mixin class for engines that do their searching ...")] 6)
|   (require gnus-search)
|   (eval-buffer #<buffer  *load*> nil "/home/micha/gnu-emacs/.gnus.el" nil t)  ; Reading at buffer position 231
|   (load-with-code-conversion "/home/micha/gnu-emacs/.gnus.el" "/home/micha/gnu-emacs/.gnus.el" nil t)
|   (load "/home/micha/gnu-emacs/.gnus.el" nil t)
|   (gnus-read-init-file)
|   (gnus-1 nil nil nil)
|   (gnus nil)

Never saw this until now.  No problem when I do not compile
cl-generic.el in the session.

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-02-23  0:27       ` Michael Heerdegen
@ 2022-02-23  0:47         ` Michael Heerdegen
  2022-03-04  2:08           ` Michael Heerdegen
  0 siblings, 1 reply; 36+ messages in thread
From: Michael Heerdegen @ 2022-02-23  0:47 UTC (permalink / raw)
  To: 54079; +Cc: monnier

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Never saw this until now.  No problem when I do not compile
> cl-generic.el in the session.

This short recipe gives me the same error in emacs -Q:

(byte-compile-file
 "/home/micha/software/emacs/lisp/emacs-lisp/cl-generic.el")
(require 'gnus-search)

"Same error" means same error message.  I didn't get a backtrace in
emacs -Q because even the debugger didn't function: M-: (debug) also
errors.

Ok - hope we now have something to start with.

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-02-23  0:47         ` Michael Heerdegen
@ 2022-03-04  2:08           ` Michael Heerdegen
  2022-03-04 19:11             ` Alan Mackenzie
  2022-03-05 16:37             ` Alan Mackenzie
  0 siblings, 2 replies; 36+ messages in thread
From: Michael Heerdegen @ 2022-03-04  2:08 UTC (permalink / raw)
  To: 54079; +Cc: monnier

Michael Heerdegen <michael_heerdegen@web.de> writes:

> This short recipe gives me the same error in emacs -Q:
>
> (byte-compile-file
>  "/home/micha/software/emacs/lisp/emacs-lisp/cl-generic.el")
> (require 'gnus-search)

Small addition: it seems that all issues mentioned disappear when I
avoid any byte compilation in my session.  Compilation of `defstruct's
seems to be the culprit (hypothesis).

Would be good to fix this problem not too late.  I think it will
irritate or distract other people, too.

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-02-21  1:14 ` Michael Heerdegen
@ 2022-03-04 16:12   ` Lars Ingebrigtsen
  0 siblings, 0 replies; 36+ messages in thread
From: Lars Ingebrigtsen @ 2022-03-04 16:12 UTC (permalink / raw)
  To: Michael Heerdegen; +Cc: Alan Mackenzie, 54079

Michael Heerdegen <michael_heerdegen@web.de> writes:

> BTW, I sometimes saw symbols with positions in the `symbol-function'
> binding of related stuff - dunno if that is normal or even part of the
> problem.

I've seen this issue, too, and it involved symbols with positions for
me, too.

I've added Alan to the CCs; perhaps he has some comments.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-04  2:08           ` Michael Heerdegen
@ 2022-03-04 19:11             ` Alan Mackenzie
  2022-03-05 16:37             ` Alan Mackenzie
  1 sibling, 0 replies; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-04 19:11 UTC (permalink / raw)
  To: Michael Heerdegen, Lars Ingebrigtsen; +Cc: 54079, monnier

Hello, Michael and Lars.

On Fri, Mar 04, 2022 at 03:08:05 +0100, Michael Heerdegen wrote:
> Michael Heerdegen <michael_heerdegen@web.de> writes:

> > This short recipe gives me the same error in emacs -Q:

> > (byte-compile-file
> >  "/home/micha/software/emacs/lisp/emacs-lisp/cl-generic.el")
> > (require 'gnus-search)

Thanks for such an easy to use recipe (despite the troubles you had
coming up with it ;-).

> Small addition: it seems that all issues mentioned disappear when I
> avoid any byte compilation in my session.  Compilation of `defstruct's
> seems to be the culprit (hypothesis).

The biggest clue is in that error message

    #<symbol let at 15662>

..  At position 15662 in file cl-generic.el, there is indeed a `let'.
It's the first symbol inside a defun, and that defun is inside an
eval-and-compile.

It seems that the positions are not being removed from the evaluation of
the defun form done by eval-and-compile.  This is the bug.

> Would be good to fix this problem not too late.  I think it will
> irritate or distract other people, too.

It irritates me.  ;-)  I'll fix it.  Maybe this evening, maybe over the
weekend.

> Michael.

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-04  2:08           ` Michael Heerdegen
  2022-03-04 19:11             ` Alan Mackenzie
@ 2022-03-05 16:37             ` Alan Mackenzie
  2022-03-05 17:57               ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
                                 ` (2 more replies)
  1 sibling, 3 replies; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-05 16:37 UTC (permalink / raw)
  To: Michael Heerdegen, Lars Ingebrigtsen; +Cc: acm, 54079, monnier

Hello, Michael.

On Fri, Mar 04, 2022 at 03:08:05 +0100, Michael Heerdegen wrote:
> Michael Heerdegen <michael_heerdegen@web.de> writes:

> > This short recipe gives me the same error in emacs -Q:
> >
> > (byte-compile-file
> >  "/home/micha/software/emacs/lisp/emacs-lisp/cl-generic.el")
> > (require 'gnus-search)

> Small addition: it seems that all issues mentioned disappear when I
> avoid any byte compilation in my session.  Compilation of `defstruct's
> seems to be the culprit (hypothesis).

> Would be good to fix this problem not too late.  I think it will
> irritate or distract other people, too.

I think this problem can only happen for defuns/defvars/defconts inside
eval-when-compile or eval-and-compile.

Would you try out the following patch, please, which I believe fixes the
bug.

Thanks!



diff --git a/src/data.c b/src/data.c
index 1526cc0c73..0854b4efb4 100644
--- a/src/data.c
+++ b/src/data.c
@@ -942,6 +942,13 @@ The return value is undefined.  */)
   (register Lisp_Object symbol, Lisp_Object definition, Lisp_Object docstring)
 {
   CHECK_SYMBOL (symbol);
+  /* If we're in a byte compilation, ensure the definition's symbols
+     are stripped of their positions. */
+  if (symbols_with_pos_enabled
+      && SYMBOL_WITH_POS_P (symbol)
+      && Ffboundp (Qbyte_run_strip_symbol_positions))
+    call1 (Qbyte_run_strip_symbol_positions, definition);
+
   if (!NILP (Vpurify_flag)
       /* If `definition' is a keymap, immutable (and copying) is wrong.  */
       && !KEYMAPP (definition))
@@ -4352,6 +4359,8 @@ This variable cannot be set; trying to do so will signal an error.  */);
 Bind this to non-nil in applications such as the byte compiler.  */);
   symbols_with_pos_enabled = false;
 
+  DEFSYM (Qbyte_run_strip_symbol_positions, "byte-run-strip-symbol-positions");
+
   DEFSYM (Qwatchers, "watchers");
   DEFSYM (Qmakunbound, "makunbound");
   DEFSYM (Qunlet, "unlet");
diff --git a/src/eval.c b/src/eval.c
index 294d79e67a..1b1fde3a20 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -794,6 +794,12 @@ usage: (defvar SYMBOL &optional INITVALUE DOCSTRING)  */)
       if (!NILP (XCDR (tail)) && !NILP (XCDR (XCDR (tail))))
 	error ("Too many arguments");
       Lisp_Object exp = XCAR (tail);
+      /* If we're in a byte compilation, ensure the definition's
+	 symbols are stripped of their positions. */
+      if (symbols_with_pos_enabled
+	  && SYMBOL_WITH_POS_P (sym)
+	  && Ffboundp (Qbyte_run_strip_symbol_positions))
+	call1 (Qbyte_run_strip_symbol_positions, exp);
 
       tem = Fdefault_boundp (sym);
       tail = XCDR (tail);
@@ -861,6 +867,14 @@ usage: (defconst SYMBOL INITVALUE [DOCSTRING])  */)
     }
 
   Finternal__define_uninitialized_variable (sym, docstring);
+
+  /* If we're in a byte compilation, ensure the definition's symbols
+     are stripped of their positions. */
+  if (symbols_with_pos_enabled
+      && SYMBOL_WITH_POS_P (sym)
+      && Ffboundp (Qbyte_run_strip_symbol_positions))
+    call1 (Qbyte_run_strip_symbol_positions, XCAR (XCDR (args)));
+
   tem = eval_sub (XCAR (XCDR (args)));
   if (!NILP (Vpurify_flag))
     tem = Fpurecopy (tem);


> Michael.


-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply related	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-05 16:37             ` Alan Mackenzie
@ 2022-03-05 17:57               ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-03-05 19:00                 ` Alan Mackenzie
  2022-03-05 18:19               ` Eli Zaretskii
  2022-03-06  2:19               ` Michael Heerdegen
  2 siblings, 1 reply; 36+ messages in thread
From: Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-03-05 17:57 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

> I think this problem can only happen for defuns/defvars/defconts inside
> eval-when-compile or eval-and-compile.
> Would you try out the following patch, please, which I believe fixes the
> bug.

Any chance we could fix it in `bytecomp.el` by calling
`byte_run_strip_symbol_positions` before evaluating the contents of
those forms?

Maybe we should have a function to check for the absence of
sympos which we could call (when in debugging mode only) from
defvar/defalias and friends, which could give us useful backtraces
showing where/when the sympos "leak"?


        Stefan


> diff --git a/src/data.c b/src/data.c
> index 1526cc0c73..0854b4efb4 100644
> --- a/src/data.c
> +++ b/src/data.c
> @@ -942,6 +942,13 @@ The return value is undefined.  */)
>    (register Lisp_Object symbol, Lisp_Object definition, Lisp_Object docstring)
>  {
>    CHECK_SYMBOL (symbol);
> +  /* If we're in a byte compilation, ensure the definition's symbols
> +     are stripped of their positions. */
> +  if (symbols_with_pos_enabled
> +      && SYMBOL_WITH_POS_P (symbol)
> +      && Ffboundp (Qbyte_run_strip_symbol_positions))
> +    call1 (Qbyte_run_strip_symbol_positions, definition);
> +
>    if (!NILP (Vpurify_flag)
>        /* If `definition' is a keymap, immutable (and copying) is wrong.  */
>        && !KEYMAPP (definition))
> @@ -4352,6 +4359,8 @@ This variable cannot be set; trying to do so will signal an error.  */);
>  Bind this to non-nil in applications such as the byte compiler.  */);
>    symbols_with_pos_enabled = false;
>  
> +  DEFSYM (Qbyte_run_strip_symbol_positions, "byte-run-strip-symbol-positions");
> +
>    DEFSYM (Qwatchers, "watchers");
>    DEFSYM (Qmakunbound, "makunbound");
>    DEFSYM (Qunlet, "unlet");
> diff --git a/src/eval.c b/src/eval.c
> index 294d79e67a..1b1fde3a20 100644
> --- a/src/eval.c
> +++ b/src/eval.c
> @@ -794,6 +794,12 @@ usage: (defvar SYMBOL &optional INITVALUE DOCSTRING)  */)
>        if (!NILP (XCDR (tail)) && !NILP (XCDR (XCDR (tail))))
>  	error ("Too many arguments");
>        Lisp_Object exp = XCAR (tail);
> +      /* If we're in a byte compilation, ensure the definition's
> +	 symbols are stripped of their positions. */
> +      if (symbols_with_pos_enabled
> +	  && SYMBOL_WITH_POS_P (sym)
> +	  && Ffboundp (Qbyte_run_strip_symbol_positions))
> +	call1 (Qbyte_run_strip_symbol_positions, exp);
>  
>        tem = Fdefault_boundp (sym);
>        tail = XCDR (tail);
> @@ -861,6 +867,14 @@ usage: (defconst SYMBOL INITVALUE [DOCSTRING])  */)
>      }
>  
>    Finternal__define_uninitialized_variable (sym, docstring);
> +
> +  /* If we're in a byte compilation, ensure the definition's symbols
> +     are stripped of their positions. */
> +  if (symbols_with_pos_enabled
> +      && SYMBOL_WITH_POS_P (sym)
> +      && Ffboundp (Qbyte_run_strip_symbol_positions))
> +    call1 (Qbyte_run_strip_symbol_positions, XCAR (XCDR (args)));
> +
>    tem = eval_sub (XCAR (XCDR (args)));
>    if (!NILP (Vpurify_flag))
>      tem = Fpurecopy (tem);
>
>
>> Michael.






^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-05 16:37             ` Alan Mackenzie
  2022-03-05 17:57               ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-03-05 18:19               ` Eli Zaretskii
  2022-03-05 19:07                 ` Alan Mackenzie
  2022-03-06  2:19               ` Michael Heerdegen
  2 siblings, 1 reply; 36+ messages in thread
From: Eli Zaretskii @ 2022-03-05 18:19 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: michael_heerdegen, acm, larsi, 54079, monnier

> Date: Sat, 5 Mar 2022 16:37:16 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: acm@muc.de, 54079@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
>    CHECK_SYMBOL (symbol);
> +  /* If we're in a byte compilation, ensure the definition's symbols
> +     are stripped of their positions. */
> +  if (symbols_with_pos_enabled
> +      && SYMBOL_WITH_POS_P (symbol)
> +      && Ffboundp (Qbyte_run_strip_symbol_positions))
> +    call1 (Qbyte_run_strip_symbol_positions, definition);

The first two conditions should be in reverse order, for speedier
code, shouldn't they?





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-05 17:57               ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-03-05 19:00                 ` Alan Mackenzie
  2022-03-05 23:01                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  0 siblings, 1 reply; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-05 19:00 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

Hello, Stefan.

On Sat, Mar 05, 2022 at 12:57:09 -0500, Stefan Monnier wrote:
> > I think this problem can only happen for defuns/defvars/defconts inside
> > eval-when-compile or eval-and-compile.
> > Would you try out the following patch, please, which I believe fixes the
> > bug.

> Any chance we could fix it in `bytecomp.el` by calling
> `byte_run_strip_symbol_positions` before evaluating the contents of
> those forms?

I've spent quite a large part of the day trying to get that to work,
without success.  The point is, we want to strip the positions here only
when we're writing the value to a defun/defvar/defconst.
eval-when/and-compile doesn't know what its forms will be doing, and
there's no easy way to find out - for example, there might be a defun in
a nested eval-when/and-compile.  Checking in the defun/defvar/defconst
special forms is direct and economical.

> Maybe we should have a function to check for the absence of
> sympos which we could call (when in debugging mode only) from
> defvar/defalias and friends, which could give us useful backtraces
> showing where/when the sympos "leak"?

That's an interesting idea.  Such a function would differ from
byte-run-strip-symbol-positions only a little bit.

>         Stefan

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-05 18:19               ` Eli Zaretskii
@ 2022-03-05 19:07                 ` Alan Mackenzie
  0 siblings, 0 replies; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-05 19:07 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: michael_heerdegen, larsi, 54079, monnier

Hello, Eli.

On Sat, Mar 05, 2022 at 20:19:43 +0200, Eli Zaretskii wrote:
> > Date: Sat, 5 Mar 2022 16:37:16 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: acm@muc.de, 54079@debbugs.gnu.org, monnier@iro.umontreal.ca
> > 
> >    CHECK_SYMBOL (symbol);
> > +  /* If we're in a byte compilation, ensure the definition's symbols
> > +     are stripped of their positions. */
> > +  if (symbols_with_pos_enabled
> > +      && SYMBOL_WITH_POS_P (symbol)
> > +      && Ffboundp (Qbyte_run_strip_symbol_positions))
> > +    call1 (Qbyte_run_strip_symbol_positions, definition);

> The first two conditions should be in reverse order, for speedier
> code, shouldn't they?

Maybe.  A slight slowdown would happen if the first condition was true
whilst the second was not.  Yes, that could happen whilst `require'ing a
..elc file, which will happen regularly.  I'll swap them around.

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-05 19:00                 ` Alan Mackenzie
@ 2022-03-05 23:01                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  0 siblings, 0 replies; 36+ messages in thread
From: Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-03-05 23:01 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

> eval-when/and-compile doesn't know what its forms will be doing, and
> there's no easy way to find out - for example, there might be a defun in
> a nested eval-when/and-compile.

I don't think we need to find out: whatever they do, they should do it
on stripped symbols.


        Stefan






^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-05 16:37             ` Alan Mackenzie
  2022-03-05 17:57               ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-03-05 18:19               ` Eli Zaretskii
@ 2022-03-06  2:19               ` Michael Heerdegen
  2022-03-08 19:28                 ` Alan Mackenzie
  2 siblings, 1 reply; 36+ messages in thread
From: Michael Heerdegen @ 2022-03-06  2:19 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Lars Ingebrigtsen, 54079, monnier

Alan Mackenzie <acm@muc.de> writes:

> > > (byte-compile-file
> > >  "/home/micha/software/emacs/lisp/emacs-lisp/cl-generic.el")
> > > (require 'gnus-search)

> Would you try out the following patch, please, which I believe fixes the
> bug.
> [...patch...]

That fixed the above case - but unfortunately not the failing method
dispatching for my register stuff after compiling some unrelated random
files.

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-06  2:19               ` Michael Heerdegen
@ 2022-03-08 19:28                 ` Alan Mackenzie
  2022-03-08 19:53                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-03-09  4:10                   ` Michael Heerdegen
  0 siblings, 2 replies; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-08 19:28 UTC (permalink / raw)
  To: Michael Heerdegen; +Cc: Lars Ingebrigtsen, 54079, monnier

Hello, Michael.

On Sun, Mar 06, 2022 at 03:19:02 +0100, Michael Heerdegen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > > > (byte-compile-file
> > > >  "/home/micha/software/emacs/lisp/emacs-lisp/cl-generic.el")
> > > > (require 'gnus-search)

> > Would you try out the following patch, please, which I believe fixes the
> > bug.
> > [...patch...]

> That fixed the above case - but unfortunately not the failing method
> dispatching for my register stuff after compiling some unrelated random
> files.

Sorry for not making more substantial progress on this bug.

With the patch for bug #54248 in place (the bug you closed), does the
following patch do anything for the current bug?



diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index 9be44a8d5a..fb9f70bd67 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -499,9 +499,10 @@ byte-compile-initial-macro-environment
                                        (byte-compile-new-defuns
                                         byte-compile-new-defuns))
                                    (setf result
-                                         (byte-compile-eval
+                                         (byte-run-strip-symbol-positions
+                                          (byte-compile-eval
                                            (byte-compile-top-level
-                                            (byte-compile-preprocess form)))))))
+                                            (byte-compile-preprocess form))))))))
                               (list 'quote result))))
     (eval-and-compile . ,(lambda (&rest body)
                            (byte-compile-recurse-toplevel
@@ -512,9 +513,10 @@ byte-compile-initial-macro-environment
                               ;; or byte-compile-file-form.
                               (let* ((print-symbols-bare t) ; Possibly redundant binding.
                                      (expanded
-                                      (macroexpand--all-toplevel
-                                       form
-                                       macroexpand-all-environment)))
+                                      (byte-run-strip-symbol-positions
+                                       (macroexpand--all-toplevel
+                                        form
+                                        macroexpand-all-environment))))
                                 (eval expanded lexical-binding)
                                 expanded)))))
     (with-suppressed-warnings


If not, could you possibly come up with a repeatable recipe for
reproducing the bug?  Or have you done this already?

> Michael.

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply related	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-08 19:28                 ` Alan Mackenzie
@ 2022-03-08 19:53                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-03-08 20:48                     ` Alan Mackenzie
  2022-03-09  4:10                   ` Michael Heerdegen
  1 sibling, 1 reply; 36+ messages in thread
From: Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-03-08 19:53 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

> diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
> index 9be44a8d5a..fb9f70bd67 100644
> --- a/lisp/emacs-lisp/bytecomp.el
> +++ b/lisp/emacs-lisp/bytecomp.el
> @@ -499,9 +499,10 @@ byte-compile-initial-macro-environment
>                                         (byte-compile-new-defuns
>                                          byte-compile-new-defuns))
>                                     (setf result
> -                                         (byte-compile-eval
> +                                         (byte-run-strip-symbol-positions
> +                                          (byte-compile-eval
>                                             (byte-compile-top-level
> -                                            (byte-compile-preprocess form)))))))
> +                                            (byte-compile-preprocess form))))))))

I'd expect the reverse: strip first and then eval the result.
Why should we not strip the form passed to `byte-compile-eval`?
Does `byte-compile-top-level` already return a stripped form of code?
And why bother stripping the result of `byte-compile-eval`?

> @@ -512,9 +513,10 @@ byte-compile-initial-macro-environment
>                                ;; or byte-compile-file-form.
>                                (let* ((print-symbols-bare t) ; Possibly redundant binding.
>                                       (expanded
> -                                      (macroexpand--all-toplevel
> -                                       form
> -                                       macroexpand-all-environment)))
> +                                      (byte-run-strip-symbol-positions
> +                                       (macroexpand--all-toplevel
> +                                        form
> +                                        macroexpand-all-environment))))
>                                  (eval expanded lexical-binding)
>                                  expanded)))))
>      (with-suppressed-warnings

Fundamentally, `eval` should always strip before doing its job.  Yes,
I know, it might be a bit expensive, but we should probably define
a local function in `bytecomp.el` which does strip+eval and use that
instead of `eval` (both here and in `byte-compile-eval`).
WDYT?


        Stefan






^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-08 19:53                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-03-08 20:48                     ` Alan Mackenzie
  2022-03-08 21:03                       ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  0 siblings, 1 reply; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-08 20:48 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Michael Heerdegen, acm, Lars Ingebrigtsen, 54079

Hello, Stefan.

On Tue, Mar 08, 2022 at 14:53:07 -0500, Stefan Monnier wrote:
> > diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
> > index 9be44a8d5a..fb9f70bd67 100644
> > --- a/lisp/emacs-lisp/bytecomp.el
> > +++ b/lisp/emacs-lisp/bytecomp.el
> > @@ -499,9 +499,10 @@ byte-compile-initial-macro-environment
> >                                         (byte-compile-new-defuns
> >                                          byte-compile-new-defuns))
> >                                     (setf result
> > -                                         (byte-compile-eval
> > +                                         (byte-run-strip-symbol-positions
> > +                                          (byte-compile-eval
> >                                             (byte-compile-top-level
> > -                                            (byte-compile-preprocess form)))))))
> > +                                            (byte-compile-preprocess form))))))))

> I'd expect the reverse: strip first and then eval the result.
> Why should we not strip the form passed to `byte-compile-eval`?

It's an edge case either way, but the form being evaluated might be a
`byte-compile', in which case it's (much) better to leave the positions
in place during this operation.

> Does `byte-compile-top-level` already return a stripped form of code?

Compiled code is always stripped, at least since the weekend!

> And why bother stripping the result of `byte-compile-eval`?

Because it might be the result of evaluating a defun (or defvar or
defconst).  This was the situation which gave rise to the bug.


> > @@ -512,9 +513,10 @@ byte-compile-initial-macro-environment
> >                                ;; or byte-compile-file-form.
> >                                (let* ((print-symbols-bare t) ; Possibly redundant binding.
> >                                       (expanded
> > -                                      (macroexpand--all-toplevel
> > -                                       form
> > -                                       macroexpand-all-environment)))
> > +                                      (byte-run-strip-symbol-positions
> > +                                       (macroexpand--all-toplevel
> > +                                        form
> > +                                        macroexpand-all-environment))))
> >                                  (eval expanded lexical-binding)
> >                                  expanded)))))
> >      (with-suppressed-warnings

> Fundamentally, `eval` should always strip before doing its job.

Except when what it's evaluating is a defun, defmacrro, defsubst, etc.
Then it would be better to evaluate SWPs (which would work, since we're
inside a compilation, where enable-symbols-with-pos has been bound).
But here EXPANDED has been stripped before being evaluated, so I'm not
sure what you're saying here.

> Yes, I know, it might be a bit expensive, but we should probably
> define a local function in `bytecomp.el` which does strip+eval and use
> that instead of `eval` (both here and in `byte-compile-eval`).  WDYT?

I don't think stripping is really all that expensive.  There are one or
two .el files in Emacs (ucs-normalize.el springs to mind) which have
very large lists with vectors in them, yet they don't seem noticeably to
slow down the Emacs build.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-08 20:48                     ` Alan Mackenzie
@ 2022-03-08 21:03                       ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-03-09 17:42                         ` Alan Mackenzie
  0 siblings, 1 reply; 36+ messages in thread
From: Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-03-08 21:03 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

>> I'd expect the reverse: strip first and then eval the result.
>> Why should we not strip the form passed to `byte-compile-eval`?
> It's an edge case either way, but the form being evaluated might be a
> `byte-compile', in which case it's (much) better to leave the positions
> in place during this operation.

I don't understand the scenario you're thinking of.
Are you thinking of something like `(eval-when-compile (byte-compile ...))?
Does that ever happen in real life?

>> Does `byte-compile-top-level` already return a stripped form of code?
> Compiled code is always stripped, at least since the weekend!

OK, so no need to strip, go.

>> And why bother stripping the result of `byte-compile-eval`?
> Because it might be the result of evaluating a defun (or defvar or
> defconst).

AFAIK sympos should only appear within the compiler pipeline between the
"read" and the "emit resulting bytecode".  They may be passed to various
functions and macros along the way, but I can't think of any scenario
where they'd end up returned by `(byte-compile-)eval`.

> This was the situation which gave rise to the bug.

Could you give some details about how it played out?
[ Either here or as a comment in the code.  ]

>> Fundamentally, `eval` should always strip before doing its job.
> Except when what it's evaluating is a defun, defmacrro, defsubst, etc.

Why?

> Then it would be better to evaluate SWPs (which would work, since we're
> inside a compilation, where enable-symbols-with-pos has been bound).
> But here EXPANDED has been stripped before being evaluated, so I'm not
> sure what you're saying here.

I was suggesting to move the strip from the computation of `expanded` to
the `eval` call.

>> Yes, I know, it might be a bit expensive, but we should probably
>> define a local function in `bytecomp.el` which does strip+eval and use
>> that instead of `eval` (both here and in `byte-compile-eval`).  WDYT?
> I don't think stripping is really all that expensive.  There are one or
> two .el files in Emacs (ucs-normalize.el springs to mind) which have
> very large lists with vectors in them, yet they don't seem noticeably to
> slow down the Emacs build.

So maybe we should redefine `eval` as "strip and then eval"?


        Stefan






^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-08 19:28                 ` Alan Mackenzie
  2022-03-08 19:53                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-03-09  4:10                   ` Michael Heerdegen
  2022-03-09 17:29                     ` Alan Mackenzie
  1 sibling, 1 reply; 36+ messages in thread
From: Michael Heerdegen @ 2022-03-09  4:10 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Lars Ingebrigtsen, 54079, monnier

Alan Mackenzie <acm@muc.de> writes:

> With the patch for bug #54248 in place (the bug you closed), does the
> following patch do anything for the current bug?

Could be...I quickly tried several times and the issue didn't show up.
But I will have to use the patch for a while to be sure.

Thanks,

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-09  4:10                   ` Michael Heerdegen
@ 2022-03-09 17:29                     ` Alan Mackenzie
  2022-03-16  1:44                       ` Michael Heerdegen
  0 siblings, 1 reply; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-09 17:29 UTC (permalink / raw)
  To: Michael Heerdegen; +Cc: acm, Lars Ingebrigtsen, 54079, monnier

Hello, Michael.

On Wed, Mar 09, 2022 at 05:10:16 +0100, Michael Heerdegen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > With the patch for bug #54248 in place (the bug you closed), does the
> > following patch do anything for the current bug?

> Could be...I quickly tried several times and the issue didn't show up.
> But I will have to use the patch for a while to be sure.

OK.  So the longer I hear nothing from you, the better.  ;-)  Thanks!

> Thanks,

> Michael.

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-08 21:03                       ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-03-09 17:42                         ` Alan Mackenzie
  2022-03-09 18:06                           ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  0 siblings, 1 reply; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-09 17:42 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

Hello, Stefan.

On Tue, Mar 08, 2022 at 16:03:12 -0500, Stefan Monnier wrote:
> >> I'd expect the reverse: strip first and then eval the result.
> >> Why should we not strip the form passed to `byte-compile-eval`?
> > It's an edge case either way, but the form being evaluated might be a
> > `byte-compile', in which case it's (much) better to leave the positions
> > in place during this operation.

> I don't understand the scenario you're thinking of.
> Are you thinking of something like `(eval-when-compile (byte-compile ...))?

Yes.

> Does that ever happen in real life?

Probably exceedingly seldomly.

What's to be gained by not catering to this unusual case?  What do we
lose?

> >> Does `byte-compile-top-level` already return a stripped form of code?
> > Compiled code is always stripped, at least since the weekend!

> OK, so no need to strip, go.

> >> And why bother stripping the result of `byte-compile-eval`?
> > Because it might be the result of evaluating a defun (or defvar or
> > defconst).

> AFAIK sympos should only appear within the compiler pipeline between the
> "read" and the "emit resulting bytecode".  They may be passed to various
> functions and macros along the way, but I can't think of any scenario
> where they'd end up returned by `(byte-compile-)eval`.

> > This was the situation which gave rise to the bug.

> Could you give some details about how it played out?
> [ Either here or as a comment in the code.  ]

Michael byte compiled cl-generic.el.  This created cl-generic.elc
correctly, but also left uncompiled forms in the function cells of the
symbols defun'd inside an eval-{when,and}-compile.  These forms
contained symbols with positions.

> >> Fundamentally, `eval` should always strip before doing its job.
> > Except when what it's evaluating is a defun, defmacrro, defsubst, etc.

> Why?

Because that evaluated form might later be byte compiled, and the SWPs
will be needed for that.

> > Then it would be better to evaluate SWPs (which would work, since we're
> > inside a compilation, where enable-symbols-with-pos has been bound).
> > But here EXPANDED has been stripped before being evaluated, so I'm not
> > sure what you're saying here.

> I was suggesting to move the strip from the computation of `expanded` to
> the `eval` call.

> >> Yes, I know, it might be a bit expensive, but we should probably
> >> define a local function in `bytecomp.el` which does strip+eval and use
> >> that instead of `eval` (both here and in `byte-compile-eval`).  WDYT?
> > I don't think stripping is really all that expensive.  There are one or
> > two .el files in Emacs (ucs-normalize.el springs to mind) which have
> > very large lists with vectors in them, yet they don't seem noticeably to
> > slow down the Emacs build.

> So maybe we should redefine `eval` as "strip and then eval"?

Isn't `eval' already complicated enough, with lexical-binding as an
argument?  Stripping SWPs is not a part of evaluation.  It is something
else.  eval should "do one thing and do it well".

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-09 17:42                         ` Alan Mackenzie
@ 2022-03-09 18:06                           ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-03-09 20:32                             ` Alan Mackenzie
  0 siblings, 1 reply; 36+ messages in thread
From: Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-03-09 18:06 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

>> I don't understand the scenario you're thinking of.
>> Are you thinking of something like `(eval-when-compile (byte-compile ...))?
> Yes.
>> Does that ever happen in real life?
> Probably exceedingly seldomly.
> What's to be gained by not catering to this unusual case?  What do we
> lose?

We lose making it work right for the 99% other cases that *do* occur?

>> >> And why bother stripping the result of `byte-compile-eval`?
>> > Because it might be the result of evaluating a defun (or defvar or
>> > defconst).
>
>> AFAIK sympos should only appear within the compiler pipeline between the
>> "read" and the "emit resulting bytecode".  They may be passed to various
>> functions and macros along the way, but I can't think of any scenario
>> where they'd end up returned by `(byte-compile-)eval`.
>
>> > This was the situation which gave rise to the bug.
>
>> Could you give some details about how it played out?
>> [ Either here or as a comment in the code.  ]
>
> Michael byte compiled cl-generic.el.  This created cl-generic.elc
> correctly, but also left uncompiled forms in the function cells of the
> symbols defun'd inside an eval-{when,and}-compile.  These forms
> contained symbols with positions.

Hmm... we're talking about stripping the result of `byte-compile-eval`.
This function is only used for `eval-when-compile`, not `eval-and-compile`.
And nothing in your above description indicates that the sympos appeared
in the resulting value of `eval-when-compile` (as opposed to appearing
in the slot of functions and variables that were set during the course
of the evaluation).

>> >> Fundamentally, `eval` should always strip before doing its job.
>> > Except when what it's evaluating is a defun, defmacrro, defsubst, etc.
>> Why?
> Because that evaluated form might later be byte compiled, and the SWPs
> will be needed for that.

I don't understand the scenario you're thinking of.
Are thinking of a case like:

- something causes the execution of (eval '(defun foo ...))
- the user types `M-x byte-compile RET foo RET`

If so, then:
- I don't think we should care about this case because it's extremely
  rare and fundamentally broken (the symbol's function cell contains
  a function *value* (i.e. a closure) and not a function's source code,
  so in general we need `byte-compile--reify-function` which implements
  a heuristic to go back to something like a source form, which can
  break in various ways in corner cases).
- If we don't strip before calling the `M-x byte-compile` then the code
  may/will bisbehave because of the presence of the sympos.


        Stefan






^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-09 18:06                           ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-03-09 20:32                             ` Alan Mackenzie
  2022-03-09 22:04                               ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  0 siblings, 1 reply; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-09 20:32 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

Hello, Stefan.

On Wed, Mar 09, 2022 at 13:06:11 -0500, Stefan Monnier wrote:
> >> I don't understand the scenario you're thinking of.
> >> Are you thinking of something like `(eval-when-compile (byte-compile ...))?
> > Yes.
> >> Does that ever happen in real life?
> > Probably exceedingly seldomly.
> > What's to be gained by not catering to this unusual case?  What do we
> > lose?

> We lose making it work right for the 99% other cases that *do* occur?

How would it not work right for such a case?  Can you give an example?

> >> >> And why bother stripping the result of `byte-compile-eval`?
> >> > Because it might be the result of evaluating a defun (or defvar or
> >> > defconst).

> >> AFAIK sympos should only appear within the compiler pipeline between the
> >> "read" and the "emit resulting bytecode".  They may be passed to various
> >> functions and macros along the way, but I can't think of any scenario
> >> where they'd end up returned by `(byte-compile-)eval`.

> >> > This was the situation which gave rise to the bug.

> >> Could you give some details about how it played out?
> >> [ Either here or as a comment in the code.  ]

> > Michael byte compiled cl-generic.el.  This created cl-generic.elc
> > correctly, but also left uncompiled forms in the function cells of the
> > symbols defun'd inside an eval-{when,and}-compile.  These forms
> > contained symbols with positions.

> Hmm... we're talking about stripping the result of `byte-compile-eval`.
> This function is only used for `eval-when-compile`, not `eval-and-compile`.
> And nothing in your above description indicates that the sympos appeared
> in the resulting value of `eval-when-compile` (as opposed to appearing
> in the slot of functions and variables that were set during the course
> of the evaluation).

OK, sorry, I was mistaken.  These forms with SWPs arose from
evan-AND-compile, which doesn't use byte-compile-eval.

> >> >> Fundamentally, `eval` should always strip before doing its job.
> >> > Except when what it's evaluating is a defun, defmacrro, defsubst, etc.
> >> Why?
> > Because that evaluated form might later be byte compiled, and the SWPs
> > will be needed for that.

> I don't understand the scenario you're thinking of.
> Are thinking of a case like:

> - something causes the execution of (eval '(defun foo ...))
> - the user types `M-x byte-compile RET foo RET`

Sorry again, I've lost the thread here.  Weren't we talking about
eval-{when,and}-compile, not eval?  Inside these two special forms, we
should preserve the SWPs as long as possible (i.e. as long as they won't
cause any errors).

> If so, then:
> - I don't think we should care about this case because it's extremely
>   rare and fundamentally broken (the symbol's function cell contains
>   a function *value* (i.e. a closure) and not a function's source code,
>   so in general we need `byte-compile--reify-function` which implements
>   a heuristic to go back to something like a source form, which can
>   break in various ways in corner cases).


Really?  After evaluating a defun, etc., we have a lisp form in the
function cell, which may be a closure.  The function byte-compile
compiles an arbitrary form, doesn't it?

> - If we don't strip before calling the `M-x byte-compile` then the code
>   may/will bisbehave because of the presence of the sympos.

How?  byte-compile is designed to use SWPs.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-09 20:32                             ` Alan Mackenzie
@ 2022-03-09 22:04                               ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-03-11 22:01                                 ` Alan Mackenzie
  0 siblings, 1 reply; 36+ messages in thread
From: Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-03-09 22:04 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

Alan Mackenzie [2022-03-09 20:32:59] wrote:
> On Wed, Mar 09, 2022 at 13:06:11 -0500, Stefan Monnier wrote:
>> >> I don't understand the scenario you're thinking of.
>> >> Are you thinking of something like `(eval-when-compile (byte-compile ...))?
>> > Yes.
>> >> Does that ever happen in real life?
>> > Probably exceedingly seldomly.
>> > What's to be gained by not catering to this unusual case?  What do we
>> > lose?
>
>> We lose making it work right for the 99% other cases that *do* occur?
> How would it not work right for such a case?  Can you give an example?

I'm not sure what "such a case" you're thinking of.  But in general,
evaluation of code doesn't expect symbols to have positions: it may test
`eq` between symbols and it may be run without having the magical
`symbols-with-pos-enabled`.

So as a general rule we should strip symbols before we pass it to `eval`.

>> >> >> And why bother stripping the result of `byte-compile-eval`?
>> >> > Because it might be the result of evaluating a defun (or defvar or
>> >> > defconst).
>
>> >> AFAIK sympos should only appear within the compiler pipeline between the
>> >> "read" and the "emit resulting bytecode".  They may be passed to various
>> >> functions and macros along the way, but I can't think of any scenario
>> >> where they'd end up returned by `(byte-compile-)eval`.
>
>> >> > This was the situation which gave rise to the bug.
>
>> >> Could you give some details about how it played out?
>> >> [ Either here or as a comment in the code.  ]
>
>> > Michael byte compiled cl-generic.el.  This created cl-generic.elc
>> > correctly, but also left uncompiled forms in the function cells of the
>> > symbols defun'd inside an eval-{when,and}-compile.  These forms
>> > contained symbols with positions.
>
>> Hmm... we're talking about stripping the result of `byte-compile-eval`.
>> This function is only used for `eval-when-compile`, not `eval-and-compile`.
>> And nothing in your above description indicates that the sympos appeared
>> in the resulting value of `eval-when-compile` (as opposed to appearing
>> in the slot of functions and variables that were set during the course
>> of the evaluation).
>
> OK, sorry, I was mistaken.  These forms with SWPs arose from
> evan-AND-compile, which doesn't use byte-compile-eval.

OK, now can we get back to the question:

    And why bother stripping the result of `byte-compile-eval`?

>> >> >> Fundamentally, `eval` should always strip before doing its job.
>> >> > Except when what it's evaluating is a defun, defmacrro, defsubst, etc.
>> >> Why?
>> > Because that evaluated form might later be byte compiled, and the SWPs
>> > will be needed for that.
>
>> I don't understand the scenario you're thinking of.
>> Are thinking of a case like:
>
>> - something causes the execution of (eval '(defun foo ...))
>> - the user types `M-x byte-compile RET foo RET`
>
> Sorry again, I've lost the thread here.  Weren't we talking about
> eval-{when,and}-compile, not eval?

No, the text you cited even included my original statement:

    Fundamentally, `eval` should always strip before doing its job.

> Inside these two special forms, we should preserve the SWPs as long as
> possible (i.e. as long as they won't cause any errors).

We should preserve them while macroexpanding and compiling their
contents, yes, but then we should strip the symbols before we pass the
result to `eval`.

>> If so, then:
>> - I don't think we should care about this case because it's extremely
>>   rare and fundamentally broken (the symbol's function cell contains
>>   a function *value* (i.e. a closure) and not a function's source code,
>>   so in general we need `byte-compile--reify-function` which implements
>>   a heuristic to go back to something like a source form, which can
>>   break in various ways in corner cases).
>
> Really?  After evaluating a defun, etc., we have a lisp form in the
> function cell, which may be a closure.

A closure is not "a Lisp form".  In general passing a closure to `eval`
may signal an error because it may very well be an invalid form.

The body of a closure is a Lips form, yes.  But that's not what's inside
a symbol's function cell.

> The function byte-compile compiles an arbitrary form, doesn't it?

Try the following:

    ;; -*- lexical-binding: t -*-

    (let ((x 0))
      (defun my-inc1 ()
        (interactive)
        (message "x=%S" (setq x (1+ x))))
      (defun my-inc2 ()
        (interactive)
        (message "x=%S" (setq x (1+ x)))))

load that file.  Then try `M-x my-inc1` and `M-x my-inc2` and you'll see
that they correctly increment the same counter.
Now do `M-: (byte-compile 'my-inc1)`.
And try `M-x my-inc1` and `M-x my-inc2` and you'll see that suddenly
they each have their own counter.

That's because it's one of the corner cases that
`byte-compile--reify-function` can't handle correctly.

>> - If we don't strip before calling the `M-x byte-compile` then the code
>>   may/will bisbehave because of the presence of the sympos.
> How?  byte-compile is designed to use SWPs.

The misbehavior I'm referring to is what happens when you call the
function before you byte-compile it (or, more likely, when you never end
up byte-compiling it), because the presence of sympos in the function
will mess up its semantics (because `symbols-with-pos-enabled` won't be
set any more when it's called).


        Stefan






^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-09 22:04                               ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-03-11 22:01                                 ` Alan Mackenzie
  2022-03-12  4:23                                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  0 siblings, 1 reply; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-11 22:01 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Michael Heerdegen, acm, Lars Ingebrigtsen, 54079

Hello, Stefan.

On Wed, Mar 09, 2022 at 17:04:00 -0500, Stefan Monnier wrote:
> Alan Mackenzie [2022-03-09 20:32:59] wrote:
> > On Wed, Mar 09, 2022 at 13:06:11 -0500, Stefan Monnier wrote:
> >> >> I don't understand the scenario you're thinking of.
> >> >> Are you thinking of something like `(eval-when-compile (byte-compile ...))?
> >> > Yes.
> >> >> Does that ever happen in real life?
> >> > Probably exceedingly seldomly.
> >> > What's to be gained by not catering to this unusual case?  What do we
> >> > lose?

> >> We lose making it work right for the 99% other cases that *do* occur?
> > How would it not work right for such a case?  Can you give an example?

> I'm not sure what "such a case" you're thinking of.

One of the "99% other cases that *do* occur" that you referred to in your
previous paragraph.  You say that these wouldn't "work right".  I'm
asking you for an example of such a "not working right".

> But in general, evaluation of code doesn't expect symbols to have
> positions:

The evaluation is completely indifferent to the SWPs, when
symbols-with-pos-enabled is t.  This is the case whilst
eval-{when,and}-compile is being evaluated during a compilation.

> ... it may test `eq` between symbols and it may be run without having
> the magical `symbols-with-pos-enabled`.

I've lost the thread here.  What scenario are you considering?  I thought
we were talking only about the `eval' within eval-{when,and}-compile.

> So as a general rule we should strip symbols before we pass it to `eval`.

I don't see this, due to all the confusion we're experiencing.  As a
general rule, within byte compilation, symbol stripping should be
postponed as long as possible whilst a compilation of the form is still
possible.

But, looking at the code, I don't think byte-compile binds
symbols-with-pos-enabled to t.  This could be a bug.

> >> >> >> And why bother stripping the result of `byte-compile-eval`?
> >> >> > Because it might be the result of evaluating a defun (or defvar or
> >> >> > defconst).

> >> >> AFAIK sympos should only appear within the compiler pipeline between the
> >> >> "read" and the "emit resulting bytecode".  They may be passed to various
> >> >> functions and macros along the way, but I can't think of any scenario
> >> >> where they'd end up returned by `(byte-compile-)eval`.

> >> >> > This was the situation which gave rise to the bug.

> >> >> Could you give some details about how it played out?
> >> >> [ Either here or as a comment in the code.  ]

> >> > Michael byte compiled cl-generic.el.  This created cl-generic.elc
> >> > correctly, but also left uncompiled forms in the function cells of the
> >> > symbols defun'd inside an eval-{when,and}-compile.  These forms
> >> > contained symbols with positions.

> >> Hmm... we're talking about stripping the result of `byte-compile-eval`.
> >> This function is only used for `eval-when-compile`, not `eval-and-compile`.
> >> And nothing in your above description indicates that the sympos appeared
> >> in the resulting value of `eval-when-compile` (as opposed to appearing
> >> in the slot of functions and variables that were set during the course
> >> of the evaluation).

> > OK, sorry, I was mistaken.  These forms with SWPs arose from
> > evan-AND-compile, which doesn't use byte-compile-eval.

> OK, now can we get back to the question:

>     And why bother stripping the result of `byte-compile-eval`?

(This is in eval-when-compile only.)  That result may contain SWPs.  For
example:

    (eval-when-compile (cons 'foo 'bar))

..  I don't think the stripping happens anywhere else, though I might be
mistaken, here.

> >> >> >> Fundamentally, `eval` should always strip before doing its job.
> >> >> > Except when what it's evaluating is a defun, defmacrro, defsubst, etc.
> >> >> Why?
> >> > Because that evaluated form might later be byte compiled, and the SWPs
> >> > will be needed for that.

> >> I don't understand the scenario you're thinking of.
> >> Are thinking of a case like:

> >> - something causes the execution of (eval '(defun foo ...))
> >> - the user types `M-x byte-compile RET foo RET`

> > Sorry again, I've lost the thread here.  Weren't we talking about
> > eval-{when,and}-compile, not eval?

> No, the text you cited even included my original statement:

>     Fundamentally, `eval` should always strip before doing its job.

You mean, by "always", you meant ALWAYS???   I understood you to mean
"always, when in the context of eval-{when,and}-compile".  If we're not
inside a compilation, and thus there're no SWPs hanging around, stripping
symbols from an expression will just mean a loss of time for a null
operation.

> > Inside these two special forms, we should preserve the SWPs as long as
> > possible (i.e. as long as they won't cause any errors).

> We should preserve them while macroexpanding and compiling their
> contents, yes, but then we should strip the symbols before we pass the
> result to `eval`.

At the moment, I disagree with you.  I don't think you have given an
example of a form FOO which will only work if stripping is done before
evaluation in

    (eval-when-compile FOO)

or

    (eval-and-compile FOO)

..  At the moment, I still think it is better to strip the positions after
the evaluation.

> >> If so, then:
> >> - I don't think we should care about this case because it's extremely
> >>   rare and fundamentally broken (the symbol's function cell contains
> >>   a function *value* (i.e. a closure) and not a function's source code,
> >>   so in general we need `byte-compile--reify-function` which implements
> >>   a heuristic to go back to something like a source form, which can
> >>   break in various ways in corner cases).

> > Really?  After evaluating a defun, etc., we have a lisp form in the
> > function cell, which may be a closure.

> A closure is not "a Lisp form".  In general passing a closure to `eval`
> may signal an error because it may very well be an invalid form.

> The body of a closure is a Lisp form, yes.  But that's not what's inside
> a symbol's function cell.

> > The function byte-compile compiles an arbitrary form, doesn't it?

> Try the following:

>     ;; -*- lexical-binding: t -*-

>     (let ((x 0))
>       (defun my-inc1 ()
>         (interactive)
>         (message "x=%S" (setq x (1+ x))))
>       (defun my-inc2 ()
>         (interactive)
>         (message "x=%S" (setq x (1+ x)))))

> load that file.  Then try `M-x my-inc1` and `M-x my-inc2` and you'll see
> that they correctly increment the same counter.
> Now do `M-: (byte-compile 'my-inc1)`.
> And try `M-x my-inc1` and `M-x my-inc2` and you'll see that suddenly
> they each have their own counter.

> That's because it's one of the corner cases that
> `byte-compile--reify-function` can't handle correctly.

> >> - If we don't strip before calling the `M-x byte-compile` then the code
> >>   may/will bisbehave because of the presence of the sympos.
> > How?  byte-compile is designed to use SWPs.

> The misbehavior I'm referring to is what happens when you call the
> function before you byte-compile it (or, more likely, when you never end
> up byte-compiling it), because the presence of sympos in the function
> will mess up its semantics (because `symbols-with-pos-enabled` won't be
> set any more when it's called).

I'm puzzled.  Are we still talking about eval-{when,and}-compile, here?

If so, how can a form with SWPs get into a symbol's function cell?  The
positions are stripped inside the e-w/a-compile.

If not, the only function which produces SWPs is read-positioning-symbols
(in lread.c).  (There are other functions which _can_ produce SWPs, but
they're used only by the compiler, I think.)  read-positioning-symbols is
called only by the byte compiler, so how can an uncompiled form with SWPs
get into a symbol's function cell?

I think the only answer to either of the above two paragraphs is a bug.
Ah, maybe not.  I think you're thinking of something like

    (eval-when-compile
      (fset 'foo '(defun bar ....)))

, where SWPs will escape into foo.  Again, this is presumably a
vanishingly unlikely thing to write, except for the writing of test
suites.  ;-)

Maybe we will need to make strip-symbol-positions available to (advanced)
users, after all (having renamed it from
byte-run-strip-symbol-positions).

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-11 22:01                                 ` Alan Mackenzie
@ 2022-03-12  4:23                                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-03-13 16:49                                     ` Alan Mackenzie
  0 siblings, 1 reply; 36+ messages in thread
From: Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-03-12  4:23 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

>> I'm not sure what "such a case" you're thinking of.
>
> One of the "99% other cases that *do* occur" that you referred to in your
> previous paragraph.  You say that these wouldn't "work right".  I'm
> asking you for an example of such a "not working right".

(defalias 'FOO BAR)

>> But in general, evaluation of code doesn't expect symbols to have
>> positions:
>
> The evaluation is completely indifferent to the SWPs, when
> symbols-with-pos-enabled is t.

While evaluating (defalias 'FOO BAR) everything will be dandy.
But when you later call FOO while `symbols-with-pos-enabled` is not
t anymore, then things go haywire.

Same with

    (put FOO BAR (lambda ...))

or

    (add-hook FOO (lambda ...))

or

    (setcdr FOO (lambda ...)))

or

    ...

Of course, similar thing will go wrong without `lambda` because for example

    (setq FOO 'bar)

will set FOO to `bar`-with-pos, so when you later compare FOO to
`bar` you will find that `eq` will say nil.
Etc... the list goes on and on.

>> ... it may test `eq` between symbols and it may be run without having
>> the magical `symbols-with-pos-enabled`.
> I've lost the thread here.  What scenario are you considering?  I thought
> we were talking only about the `eval' within eval-{when,and}-compile.

Whatever appears in the code passed through `eval-{when,and}-compile`
can survive that code, so the fact that `symbols-with-pos-enabled` is
non-nil while evaluating it doesn't make it correct.

>> So as a general rule we should strip symbols before we pass it to `eval`.
> I don't see this, due to all the confusion we're experiencing.  As a
> general rule, within byte compilation, symbol stripping should be
> postponed as long as possible whilst a compilation of the form is still
> possible.

`eval` is exactly this "as long as possible" point where we know the
code won't be compiled because it's about to be interpreted.

> But, looking at the code, I don't think byte-compile binds
> symbols-with-pos-enabled to t.  This could be a bug.

Maybe not: it has no reason to presume that its argument has positions.
If it does, then it must be because the caller explicitly arranged for
it to happen and so the caller could be in charge of binding
that variable.

>>     And why bother stripping the result of `byte-compile-eval`?
>
> (This is in eval-when-compile only.)  That result may contain SWPs.  For
> example:
>
>     (eval-when-compile (cons 'foo 'bar))

But the result is then passed to the compiler anyway, so this is the
same as writing

      '(foo bar)

which will come "with pos", so I still don't see why we'd need to strip
pos from the result here.  Even more so because the code passed to
`byte-compile-eval` is compiled so it presumably doesn't have any sympos
anyway, so there shouldn't be any sympos in its output either anyway.

>>     Fundamentally, `eval` should always strip before doing its job.
> You mean, by "always", you meant ALWAYS???

Yes.

> I understood you to mean "always, when in the context of
> eval-{when,and}-compile".  If we're not inside a compilation, and thus
> there're no SWPs hanging around, stripping symbols from an expression
> will just mean a loss of time for a null operation.

That's right: the only cases where not stripping the arg of `eval` is OK
is when we know that stripping would do nothing.
IOW it's an optimization.

> At the moment, I disagree with you.  I don't think you have given an
> example of a form FOO which will only work if stripping is done before
> evaluation in
>
>     (eval-when-compile FOO)
>
> or
>
>     (eval-and-compile FOO)
>
> ..  At the moment, I still think it is better to strip the positions after
> the evaluation.

And I still haven't heard any good reason why stripping them after the
evaluation would be of any use since that result is passed to the
compiler which is perfectly happy to accept code-with-pos.

>> The misbehavior I'm referring to is what happens when you call the
>> function before you byte-compile it (or, more likely, when you never end
>> up byte-compiling it), because the presence of sympos in the function
>> will mess up its semantics (because `symbols-with-pos-enabled` won't be
>> set any more when it's called).
>
> I'm puzzled.  Are we still talking about eval-{when,and}-compile, here?

No.  We're talking about a hypothetical case where `(eval '(defun foo ...))`
is executed somehow and where the `(defun foo ...)` part somehow
contains symposes.

I used it as an example of why `eval` should conceptually always strip
its argument (or if `eval` doesn't do it, then its caller should make
sure that the code passed to it doesn't contains symposes).

From what I understand you suggested that in that cases `eval` should
preserve the symposes in the hope that the user later calls
`(byte-compile 'foo)` which can then benefit from those symposes to give
better warnings.  And I pointed out that this still means that (until
that `byte-compile` call comes) the code of `foo` will likely be broken
so any uses of `foo` will likely misbehave.

> If so, how can a form with SWPs get into a symbol's function cell?
> The positions are stripped inside the e-w/a-compile.

Hmm... I agree that the positions *should* be stripped inside
e-w/a-compile before passing the code to `eval`, but the code I see
right now in bytecomp.el says:

    (eval-and-compile . ,(lambda (&rest body)
                           (byte-compile-recurse-toplevel
                            (macroexp-progn body)
                            (lambda (form)
                              ;; Don't compile here, since we don't know
                              ;; whether to compile as byte-compile-form
                              ;; or byte-compile-file-form.
                              (let* ((print-symbols-bare t) ; Possibly redundant binding.
                                     (expanded
                                      (macroexpand--all-toplevel
                                       form
                                       macroexpand-all-environment)))
                                (eval expanded lexical-binding)
                                expanded)))))

I don't see where the symbols are stripped before passing the code to `eval`.
So if `form` is of the form `(defun foo ...)` I think we have a problem.

>     (eval-when-compile
>       (fset 'foo '(defun bar ....)))
>
> , where SWPs will escape into foo.

Why do you need the `put` in there?  Doesn't the same problem show up with

     (eval-and-compile
       (defun bar ....))

which does not seem "vanishingly unlikely" at all.


        Stefan






^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-12  4:23                                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-03-13 16:49                                     ` Alan Mackenzie
  2022-03-14 12:10                                       ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  0 siblings, 1 reply; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-13 16:49 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

Hello, Stefan.

On Fri, Mar 11, 2022 at 23:23:50 -0500, Stefan Monnier wrote:

[ .... ]

> Whatever appears in the code passed through `eval-{when,and}-compile`
> can survive that code, so the fact that `symbols-with-pos-enabled` is

> `eval` is exactly this "as long as possible" point where we know the
> code won't be compiled because it's about to be interpreted.
> non-nil while evaluating it doesn't make it correct.

OK, you've persuaded me.  ;-)  Lisp forms need to be stripped of SWPs
before being passed into `eval'.

> > But, looking at the code, I don't think byte-compile binds
> > symbols-with-pos-enabled to t.  This could be a bug.

> Maybe not: it has no reason to presume that its argument has positions.

More to the point, I don't think it can assume that there aren't SWPs in
the form.

> If it does, then it must be because the caller explicitly arranged for
> it to happen and so the caller could be in charge of binding
> that variable.

The caller could be any lisp function.  It's just that binding
symbols-with-pos-enabled in byte-compile is cheap and harmless, and there
might well be legitimate cases that need it.  I can't think of any at the
moment, though.


> >>     And why bother stripping the result of `byte-compile-eval`?

I think this point is moot if the form has been stripped before going
into byte-compile-eval.  In that case, there will be no need to strip the
result.

[ .... ]

> >>     Fundamentally, `eval` should always strip before doing its job.
> > You mean, by "always", you meant ALWAYS???

> Yes.

> > I understood you to mean "always, when in the context of
> > eval-{when,and}-compile".  If we're not inside a compilation, and thus
> > there're no SWPs hanging around, stripping symbols from an expression
> > will just mean a loss of time for a null operation.

> That's right: the only cases where not stripping the arg of `eval` is OK
> is when we know that stripping would do nothing.
> IOW it's an optimization.

It's NOT an optimisation.  It's a straightforward implementation of the
design.  To add unnecessary stripping of non-existent symbol positions
would be a pessimisation, a possibly measurable slowing down of Emacs.

The whole essence of the SWP design was to minimise the influence of the
SWPs on code which isn't the compiler.  Always stripping before calling
`eval' would not be in accordance with that design.

There are a lot of places where `eval' is called.  Inserting
byte-run-strip-symbol-positions before each of them would be tedious in
the extreme.  Instead damaging `eval', to make it do two disparate things
which just happen to follow on from eachother sometimes, would be worse;
it would likely involve adding an extra &optional argument meaning
"strip".

[ .... ]

> >> The misbehavior I'm referring to is what happens when you call the
> >> function before you byte-compile it (or, more likely, when you never end
> >> up byte-compiling it), because the presence of sympos in the function
> >> will mess up its semantics (because `symbols-with-pos-enabled` won't be
> >> set any more when it's called).

> > I'm puzzled.  Are we still talking about eval-{when,and}-compile, here?

> No.  We're talking about a hypothetical case where `(eval '(defun foo ...))`
> is executed somehow and where the `(defun foo ...)` part somehow
> contains symposes.

Modulo bugs, I think that could only happen in a macro, where the (defun
foo ...) bit was an argument, and the macro takes steps to preserve that
argument beyond its natural scope.

> I used it as an example of why `eval` should conceptually always strip
> its argument (or if `eval` doesn't do it, then its caller should make
> sure that the code passed to it doesn't contains symposes).

I think it will be rare.

[ .... ]

I will prepare another patch which strips the forms of SWPs before the
`eval' in eval-{when,and}-compile.

Apologies if I've inadvertently failed to answer any open points.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-13 16:49                                     ` Alan Mackenzie
@ 2022-03-14 12:10                                       ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
  0 siblings, 0 replies; 36+ messages in thread
From: Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-03-14 12:10 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Michael Heerdegen, Lars Ingebrigtsen, 54079

>> If it does, then it must be because the caller explicitly arranged for
>> it to happen and so the caller could be in charge of binding
>> that variable.
> The caller could be any lisp function.

Yes, but if there are symposes, it can only be because somewhere up the
call chain, someone decided to use sympos, and that someone should be
the one responsible to bind `symbols-with-pos-enabled`, I think.

> It's just that binding symbols-with-pos-enabled in byte-compile is
> cheap and harmless,

Agreed.

> and there might well be legitimate cases that need it.  I can't think
> of any at the moment, though.

Fair enough.

>> That's right: the only cases where not stripping the arg of `eval` is OK
>> is when we know that stripping would do nothing.
>> IOW it's an optimization.
> It's NOT an optimisation.  It's a straightforward implementation of the
> design.  To add unnecessary stripping of non-existent symbol positions
> would be a pessimisation, a possibly measurable slowing down of Emacs.

We violently agree.


        Stefan






^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-09 17:29                     ` Alan Mackenzie
@ 2022-03-16  1:44                       ` Michael Heerdegen
  2022-03-16 11:52                         ` Alan Mackenzie
  2022-03-16 19:29                         ` Alan Mackenzie
  0 siblings, 2 replies; 36+ messages in thread
From: Michael Heerdegen @ 2022-03-16  1:44 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: John Wiegley, Lars Ingebrigtsen, 54079, Thierry Volpiatto

Alan Mackenzie <acm@muc.de> writes:

> OK.  So the longer I hear nothing from you, the better.  ;-)  Thanks!

Hope is was long enough :-)

I now tried to debug the other thing that is still happening for me -
failing compilation with async-bytecomp.el when updating packages with
M-x list-packages (I CC the maintainers).

In my currently running session I get

#+begin_src emacs-lisp
(async-inject-variables "\\`\\(?:load-path\\'\\|byte-\\)")
  ==>
  (setq byte-set-marker 147 byte-optimize--dynamic-vars
        '(#<symbol magit-log-margin at 854429>
          #<symbol magit-status-mode-map at 854375> ...)
        ...)
#+end_src

These symbols with position in that sexp seem to be the, or one, cause
of the trouble.  The symbols directly come from a `mapatoms' call, and
AFAIK this sexp is then `print'ed and sent to another emacs process,
where it causes a `read' error.

What has to be done to fix this one?

TIA,

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-16  1:44                       ` Michael Heerdegen
@ 2022-03-16 11:52                         ` Alan Mackenzie
  2022-03-16 19:22                           ` Michael Heerdegen
  2022-03-17  2:58                           ` Michael Heerdegen
  2022-03-16 19:29                         ` Alan Mackenzie
  1 sibling, 2 replies; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-16 11:52 UTC (permalink / raw)
  To: Michael Heerdegen
  Cc: John Wiegley, Lars Ingebrigtsen, Thierry Volpiatto, 54079, acm

On Wed, Mar 16, 2022 at 02:44:07 +0100, Michael Heerdegen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > OK.  So the longer I hear nothing from you, the better.  ;-)  Thanks!

> Hope it was long enough :-)

Indeed it was!  Thanks!

As you'll have gathered, I've been debating the final form of the patch
with Stefan M. for the last week.  The patch I'll actually be committing
will be slightly different from the one from last week, but not in any
way which should affect the success of the problem fix.

> I now tried to debug the other thing that is still happening for me -
> failing compilation with async-bytecomp.el when updating packages with
> M-x list-packages (I CC the maintainers).

> In my currently running session I get

> #+begin_src emacs-lisp
> (async-inject-variables "\\`\\(?:load-path\\'\\|byte-\\)")
>   ==>
>   (setq byte-set-marker 147 byte-optimize--dynamic-vars
>         '(#<symbol magit-log-margin at 854429>
>           #<symbol magit-status-mode-map at 854375> ...)
>         ...)
> #+end_src

> These symbols with position in that sexp seem to be the, or one, cause
> of the trouble.  The symbols directly come from a `mapatoms' call, and
> AFAIK this sexp is then `print'ed and sent to another emacs process,
> where it causes a `read' error.

> What has to be done to fix this one?

I don't know, as yet.  Almost certainly, the symbols with position are
somehow escaping from the compilation context.  It looks like something
in magit.  Possibly, I'll need to ask the magit maintainer to call
byte-run-strip-symbol-positions (which _surely_ ought to be renamed to
strip_symbol_positions) where these SWPs escape.  (Possibly in a macro).

In the meantime, could you possibly raise a fresh bug report for this
one, please, and give a complete recipe for reproducing the bug.
Thanks!

> TIA,

> Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-16 11:52                         ` Alan Mackenzie
@ 2022-03-16 19:22                           ` Michael Heerdegen
  2022-03-17  2:58                           ` Michael Heerdegen
  1 sibling, 0 replies; 36+ messages in thread
From: Michael Heerdegen @ 2022-03-16 19:22 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: John Wiegley, Lars Ingebrigtsen, 54079, Thierry Volpiatto

Alan Mackenzie <acm@muc.de> writes:

> I don't know, as yet.  Almost certainly, the symbols with position are
> somehow escaping from the compilation context.  It looks like something
> in magit.  Possibly, I'll need to ask the magit maintainer to call
> byte-run-strip-symbol-positions (which _surely_ ought to be renamed to
> strip_symbol_positions) where these SWPs escape.  (Possibly in a macro).

Evaluating the following expression gives me a result containing symbols
with positions.  Is this expected?  Does not happen in emacs -Q, but in
my session.

#+begin_src emacs-lisp
(let ((include-regexp "\\`\\(?:load-path\\'\\|byte-\\)")
      (bindings))
  (mapatoms
   (lambda (sym)
     (let* ((sname (and (boundp sym) (symbol-name sym)))
            (value (and sname (symbol-value sym))))
       (when (and sname
                  (or (null include-regexp)
                      (string-match include-regexp sname))
                  (not (string-match "-syntax-table\\'" sname)))
         (unless (or (stringp value)
                     (memq value '(nil t))
                     (numberp value)
                     (vectorp value))
           (setq value `(quote ,value)))
         (setq bindings (cons value bindings)
               bindings (cons sym bindings))))))
  bindings)
#+end_src

That gives me:

(#<symbol magit-log-margin at 854429>
 #<symbol magit-status-mode-map at 854375>
 #<symbol helm-source-global-mark-ring at 785224>
 #<symbol my-magit-log-standard-args at 799360>
 #<symbol my-helm-source-ediff-current-file at 798535>
 #<symbol helm-source-my-external-command-history at 786886>
 #<symbol helm-source-my-external-command-not-found at 786729>
 #<symbol helm-source-my-external-commands at 786482>
 #<symbol my-helm-do-asynchronous-shell-command-history at 782092>
 #<symbol my-helm-elisp-completion-at-point--current-pattern at 767706>
 #<symbol debbugs-gnu-mode-map at 753120>
 #<symbol debbugs-gnu-phrase-prompt at 752550>
 #<symbol meta-mode-map at 732486>
 #<symbol LaTeX-mode-map at 732050>
 #<symbol meta-mode-map at 731547>
 #<symbol slime-volleyball-mode-map at 726893>
 #<symbol proced-format-alist at 677873>
 #<symbol proced-grammar-alist at 677841>
 #<symbol bbdb-records at 665737>
 #<symbol org-todo-state-map at 663167>
 #<symbol calendar-mode-map at 655782>
 #<symbol mines-mode-map at 653366>
 #<symbol message-mode-map at 652702>
 #<symbol mail-mode-map at 652419>
 #<symbol locate-command at 648823>
 #<symbol locate-prompt-for-command at 648696>
 #<symbol locate-make-command-line at 648660>
 #<symbol scheme-mode-map at 648523>
 #<symbol lisp-mode-map at 648394>
 #<symbol conf-mode-map at 596162>
 #<symbol ispell-local-dictionary at 591900> my-helm-eww-input-history
 #<symbol eww-history at 569895>
 #<symbol eshell-rm-removes-directories at 558353>
 #<symbol eshell-visual-commands at 558318>
 #<symbol edebug-backtrace-buffer at 508499>
 #<symbol edebug-trace-buffer at 507825>
 #<symbol wdired-mode-map at 503468>
 #<symbol warning-suppress-log-types at 476716>
 #<symbol find-cmd-widget-regex-media-file at 364490> dg-date
 #<symbol calc-group-digits at 306798>
 #<symbol calc-full-float-format at 306746>
 #<symbol calc-float-format at 306699>
 #<symbol calc-prefer-frac at 306653>
 #<symbol calc-language at 304105>
 #<symbol calc-help-map at 297613>
 #<symbol calc-mode-map at 297474>
 #<symbol calc-group-char at 294322>
 #<symbol calc-group-digits at 294226>
 #<symbol calc-simplify-mode at 294098>
 #<symbol calc-prefer-frac at 294042>
 #<symbol calc-symbolic-mode at 293982>
 #<symbol calc-angle-mode at 293925>
 #<symbol var-EvalRules at 291659>
 #<symbol calc-highlight-selections-with-faces at 291239>
 #<symbol apropos-mode-map at 287698>
 #<symbol gnus-message-buffer at 238830>
 #<symbol gnus-message-buffer at 238673>
 #<symbol gnus-summary-buffer at 238492>
 #<symbol gnus-summary-buffer at 238331>
 #<symbol gnus-article-buffer at 238167>
 #<symbol gnus-article-buffer at 238006>
 #<symbol enable-temp-map at 232360>
 #<symbol my-toggle-map at 227323> cl-struct-my-command-register-tags
 #<symbol ps-header-lines at 160592>
 #<symbol ps-right-header at 160565>
 #<symbol ps-left-header at 160539>
 #<symbol ps-header-font-size at 160508>
 #<symbol ps-header-title-font-size at 160471>
 #<symbol ps-spool-buffer at 160444>
 #<symbol ps-font-size at 160420>
 #<symbol frameset--reuse-list at 100801>
 #<symbol frameset--action-map at 100763>
 #<symbol frameset--target-display at 100721> uncached mark default-low default-high default score)

Doesn't look magit specific.

> In the meantime, could you possibly raise a fresh bug report for this
> one, please, and give a complete recipe for reproducing the bug.

Still appropriate with the new information?

TIA,

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-16  1:44                       ` Michael Heerdegen
  2022-03-16 11:52                         ` Alan Mackenzie
@ 2022-03-16 19:29                         ` Alan Mackenzie
  1 sibling, 0 replies; 36+ messages in thread
From: Alan Mackenzie @ 2022-03-16 19:29 UTC (permalink / raw)
  To: Michael Heerdegen
  Cc: John Wiegley, Lars Ingebrigtsen, Thierry Volpiatto, 54079-done,
	acm

Hello, Michael.

On Wed, Mar 16, 2022 at 02:44:07 +0100, Michael Heerdegen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > OK.  So the longer I hear nothing from you, the better.  ;-)  Thanks!

> Hope it was long enough :-)

Yes, thanks!  I've committed a patch (slightly different from the one
you tested ;-), and I'm closing the bug with this post.

[ .... ]

> Michael.

-- 
Alan Mackenzie (Nuremberg, Germany).





^ permalink raw reply	[flat|nested] 36+ messages in thread

* bug#54079: 29.0.50; Method dispatching eratically fails
  2022-03-16 11:52                         ` Alan Mackenzie
  2022-03-16 19:22                           ` Michael Heerdegen
@ 2022-03-17  2:58                           ` Michael Heerdegen
  1 sibling, 0 replies; 36+ messages in thread
From: Michael Heerdegen @ 2022-03-17  2:58 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: John Wiegley, Lars Ingebrigtsen, 54079, Thierry Volpiatto

Alan Mackenzie <acm@muc.de> writes:

> In the meantime, could you possibly raise a fresh bug report for this
> one, please, and give a complete recipe for reproducing the bug.
> Thanks!

Done now - see bug#54433: 29.0.50; Invalid read syntax: "#<" with
async-bytecomp.

And thanks for fixing this one!

Regards,

Michael.





^ permalink raw reply	[flat|nested] 36+ messages in thread

end of thread, other threads:[~2022-03-17  2:58 UTC | newest]

Thread overview: 36+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-02-21  0:12 bug#54079: 29.0.50; Method dispatching eratically fails Michael Heerdegen
2022-02-21  1:14 ` Michael Heerdegen
2022-03-04 16:12   ` Lars Ingebrigtsen
2022-02-21  3:16 ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-02-21  4:21   ` Michael Heerdegen
2022-02-22 23:55     ` Michael Heerdegen
2022-02-23  0:27       ` Michael Heerdegen
2022-02-23  0:47         ` Michael Heerdegen
2022-03-04  2:08           ` Michael Heerdegen
2022-03-04 19:11             ` Alan Mackenzie
2022-03-05 16:37             ` Alan Mackenzie
2022-03-05 17:57               ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-03-05 19:00                 ` Alan Mackenzie
2022-03-05 23:01                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-03-05 18:19               ` Eli Zaretskii
2022-03-05 19:07                 ` Alan Mackenzie
2022-03-06  2:19               ` Michael Heerdegen
2022-03-08 19:28                 ` Alan Mackenzie
2022-03-08 19:53                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-03-08 20:48                     ` Alan Mackenzie
2022-03-08 21:03                       ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-03-09 17:42                         ` Alan Mackenzie
2022-03-09 18:06                           ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-03-09 20:32                             ` Alan Mackenzie
2022-03-09 22:04                               ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-03-11 22:01                                 ` Alan Mackenzie
2022-03-12  4:23                                   ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-03-13 16:49                                     ` Alan Mackenzie
2022-03-14 12:10                                       ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-03-09  4:10                   ` Michael Heerdegen
2022-03-09 17:29                     ` Alan Mackenzie
2022-03-16  1:44                       ` Michael Heerdegen
2022-03-16 11:52                         ` Alan Mackenzie
2022-03-16 19:22                           ` Michael Heerdegen
2022-03-17  2:58                           ` Michael Heerdegen
2022-03-16 19:29                         ` Alan Mackenzie

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.