bug#54079: 29.0.50; Method dispatching eratically fails

[Alan Mackenzie <acm@muc.de>]

Hello, Stefan.

On Wed, Mar 09, 2022 at 17:04:00 -0500, Stefan Monnier wrote:
> Alan Mackenzie [2022-03-09 20:32:59] wrote:
> > On Wed, Mar 09, 2022 at 13:06:11 -0500, Stefan Monnier wrote:
> >> >> I don't understand the scenario you're thinking of.
> >> >> Are you thinking of something like `(eval-when-compile (byte-compile ...))?
> >> > Yes.
> >> >> Does that ever happen in real life?
> >> > Probably exceedingly seldomly.
> >> > What's to be gained by not catering to this unusual case?  What do we
> >> > lose?

> >> We lose making it work right for the 99% other cases that *do* occur?
> > How would it not work right for such a case?  Can you give an example?

> I'm not sure what "such a case" you're thinking of.

One of the "99% other cases that *do* occur" that you referred to in your
previous paragraph.  You say that these wouldn't "work right".  I'm
asking you for an example of such a "not working right".

> But in general, evaluation of code doesn't expect symbols to have
> positions:

The evaluation is completely indifferent to the SWPs, when
symbols-with-pos-enabled is t.  This is the case whilst
eval-{when,and}-compile is being evaluated during a compilation.

> ... it may test `eq` between symbols and it may be run without having
> the magical `symbols-with-pos-enabled`.

I've lost the thread here.  What scenario are you considering?  I thought
we were talking only about the `eval' within eval-{when,and}-compile.

> So as a general rule we should strip symbols before we pass it to `eval`.

I don't see this, due to all the confusion we're experiencing.  As a
general rule, within byte compilation, symbol stripping should be
postponed as long as possible whilst a compilation of the form is still
possible.

But, looking at the code, I don't think byte-compile binds
symbols-with-pos-enabled to t.  This could be a bug.

> >> >> >> And why bother stripping the result of `byte-compile-eval`?
> >> >> > Because it might be the result of evaluating a defun (or defvar or
> >> >> > defconst).

> >> >> AFAIK sympos should only appear within the compiler pipeline between the
> >> >> "read" and the "emit resulting bytecode".  They may be passed to various
> >> >> functions and macros along the way, but I can't think of any scenario
> >> >> where they'd end up returned by `(byte-compile-)eval`.

> >> >> > This was the situation which gave rise to the bug.

> >> >> Could you give some details about how it played out?
> >> >> [ Either here or as a comment in the code.  ]

> >> > Michael byte compiled cl-generic.el.  This created cl-generic.elc
> >> > correctly, but also left uncompiled forms in the function cells of the
> >> > symbols defun'd inside an eval-{when,and}-compile.  These forms
> >> > contained symbols with positions.

> >> Hmm... we're talking about stripping the result of `byte-compile-eval`.
> >> This function is only used for `eval-when-compile`, not `eval-and-compile`.
> >> And nothing in your above description indicates that the sympos appeared
> >> in the resulting value of `eval-when-compile` (as opposed to appearing
> >> in the slot of functions and variables that were set during the course
> >> of the evaluation).

> > OK, sorry, I was mistaken.  These forms with SWPs arose from
> > evan-AND-compile, which doesn't use byte-compile-eval.

> OK, now can we get back to the question:

>     And why bother stripping the result of `byte-compile-eval`?

(This is in eval-when-compile only.)  That result may contain SWPs.  For
example:

    (eval-when-compile (cons 'foo 'bar))

..  I don't think the stripping happens anywhere else, though I might be
mistaken, here.

> >> >> >> Fundamentally, `eval` should always strip before doing its job.
> >> >> > Except when what it's evaluating is a defun, defmacrro, defsubst, etc.
> >> >> Why?
> >> > Because that evaluated form might later be byte compiled, and the SWPs
> >> > will be needed for that.

> >> I don't understand the scenario you're thinking of.
> >> Are thinking of a case like:

> >> - something causes the execution of (eval '(defun foo ...))
> >> - the user types `M-x byte-compile RET foo RET`

> > Sorry again, I've lost the thread here.  Weren't we talking about
> > eval-{when,and}-compile, not eval?

> No, the text you cited even included my original statement:

>     Fundamentally, `eval` should always strip before doing its job.

You mean, by "always", you meant ALWAYS???   I understood you to mean
"always, when in the context of eval-{when,and}-compile".  If we're not
inside a compilation, and thus there're no SWPs hanging around, stripping
symbols from an expression will just mean a loss of time for a null
operation.

> > Inside these two special forms, we should preserve the SWPs as long as
> > possible (i.e. as long as they won't cause any errors).

> We should preserve them while macroexpanding and compiling their
> contents, yes, but then we should strip the symbols before we pass the
> result to `eval`.

At the moment, I disagree with you.  I don't think you have given an
example of a form FOO which will only work if stripping is done before
evaluation in

    (eval-when-compile FOO)

or

    (eval-and-compile FOO)

..  At the moment, I still think it is better to strip the positions after
the evaluation.

> >> If so, then:
> >> - I don't think we should care about this case because it's extremely
> >>   rare and fundamentally broken (the symbol's function cell contains
> >>   a function *value* (i.e. a closure) and not a function's source code,
> >>   so in general we need `byte-compile--reify-function` which implements
> >>   a heuristic to go back to something like a source form, which can
> >>   break in various ways in corner cases).

> > Really?  After evaluating a defun, etc., we have a lisp form in the
> > function cell, which may be a closure.

> A closure is not "a Lisp form".  In general passing a closure to `eval`
> may signal an error because it may very well be an invalid form.

> The body of a closure is a Lisp form, yes.  But that's not what's inside
> a symbol's function cell.

> > The function byte-compile compiles an arbitrary form, doesn't it?

> Try the following:

>     ;; -*- lexical-binding: t -*-

>     (let ((x 0))
>       (defun my-inc1 ()
>         (interactive)
>         (message "x=%S" (setq x (1+ x))))
>       (defun my-inc2 ()
>         (interactive)
>         (message "x=%S" (setq x (1+ x)))))

> load that file.  Then try `M-x my-inc1` and `M-x my-inc2` and you'll see
> that they correctly increment the same counter.
> Now do `M-: (byte-compile 'my-inc1)`.
> And try `M-x my-inc1` and `M-x my-inc2` and you'll see that suddenly
> they each have their own counter.

> That's because it's one of the corner cases that
> `byte-compile--reify-function` can't handle correctly.

> >> - If we don't strip before calling the `M-x byte-compile` then the code
> >>   may/will bisbehave because of the presence of the sympos.
> > How?  byte-compile is designed to use SWPs.

> The misbehavior I'm referring to is what happens when you call the
> function before you byte-compile it (or, more likely, when you never end
> up byte-compiling it), because the presence of sympos in the function
> will mess up its semantics (because `symbols-with-pos-enabled` won't be
> set any more when it's called).

I'm puzzled.  Are we still talking about eval-{when,and}-compile, here?

If so, how can a form with SWPs get into a symbol's function cell?  The
positions are stripped inside the e-w/a-compile.

If not, the only function which produces SWPs is read-positioning-symbols
(in lread.c).  (There are other functions which _can_ produce SWPs, but
they're used only by the compiler, I think.)  read-positioning-symbols is
called only by the byte compiler, so how can an uncompiled form with SWPs
get into a symbol's function cell?

I think the only answer to either of the above two paragraphs is a bug.
Ah, maybe not.  I think you're thinking of something like

    (eval-when-compile
      (fset 'foo '(defun bar ....)))

, where SWPs will escape into foo.  Again, this is presumably a
vanishingly unlikely thing to write, except for the writing of test
suites.  ;-)

Maybe we will need to make strip-symbol-positions available to (advanced)
users, after all (having renamed it from
byte-run-strip-symbol-positions).

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).