bug#65017: 29.1; Byte compiler interaction with cl-lib function objects, removes symbol-function

[Alan Mackenzie <acm@muc.de>]

Hello, Stefan.

On Thu, Aug 03, 2023 at 10:43:16 -0400, Stefan Monnier wrote:
> > We can simplify your nice little test case to

> > ------- first file -----------
> > (require 'cl-macs)
> > (defun zeta () (cl-flet () #'equal))
> > ------- second file ---------
> > (defun eta () (cl-flet () (funcall #'equal 12 34)))
> > ------------------------------

> > and indeed, the leak is in cl--labels-convert-cache which will contain
> > `equal` as a symbol-with-pos after byte-compiling the first file, and this
> > causes trouble in the second file.

> > cl--labels-convert-cache contains

> >   (#<symbol equal at 49> function #<symbol equal at 49>)

> > and the function `eta` is consequently defined as

> >   (closure (t) nil (progn (#<symbol equal at 49> 12 34)))

> > where 49 is the position of `equal` in the first file.

> Thanks Mattias.

> AFAICT the problem is that OT1H `symbols-with-pos-enabled` is
> non-nil while loading the second file, but OTOH positions aren't
> stripped from the resulting code.

As I suggested in my other post, it might be a "simple" problem of cache
invalidation.  Why is the value from the first compilation hanging around
until the second one?

> So in `cl--labels-convert` when we check

>     (eq f (car cl--labels-convert-cache))

> we get when `f` is just `equal` whereas the cache contains
> the sympos, but that sympos is not stripped later on.

> I don't know why `symbols-with-pos-enabled` is non-nil at that point (I
> thought we only enabled it wile byte-compiling), ....

I believe that is indeed the case.

> .... but assuming there's a good reason for it, I tried to work around
> the problem with the patch below which is conceptually correct (indeed
> we should only return (cdr cl--labels-convert-cache) only in the case
> where `f` is exactly the very same object as (car
> cl--labels-convert-cache)).

> Sadly, it doesn't seem to help for a reason that escapes me.

Setting symbols-with-pos-enabled to nil does nothing other than disable
the mechanisms which handle symbols with position.  Any such symbols
which exist will continue to be swp, but will no longer be EQ to the base
symbol, or other swp's with the same base symbol.

> The *Messages* buffer says:

>     For information about GNU Emacs and the GNU system, type C-h C-a.
>     Compiling .../tmp/foo1.el...
>     Self-rewrite #<symbol equal at 82> to #'#<symbol equal at 82> (t)
>     Compiling .../tmp/foo1.el...done
>     Wrote .../tmp/foo1.elc
>     Self-rewrite equal to #'#<symbol equal at 82> (t)
>     Self-rewrite #<symbol equal at 82> to #'#<symbol equal at 82> (t)

> where the middle "self-rewrite" is the culprit.
> Apparently

>     (let ((symbols-with-pos-enabled nil))
>       (eq f (car cl--labels-convert-cache)))

> returned non-nil when `f` was the bare `equal` and the `car` returned
> the sympos.  How can that be?  I thought `eq` should really be the
> good old "pointer equality" when `symbols-with-pos-enabled` is nil.

> What am I missing?


>         Stefan


> diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
> index 0a3181561bd..bc71f565f3b 100644
> --- a/lisp/emacs-lisp/cl-macs.el
> +++ b/lisp/emacs-lisp/cl-macs.el
> @@ -2037,7 +2039,12 @@
>     ;; *after* handling `function', but we want to stop macroexpansion from
>     ;; being applied infinitely, so we use a cache to return the exact `form'
>     ;; being expanded even though we don't receive it.
> -   ((eq f (car cl--labels-convert-cache)) (cdr cl--labels-convert-cache))
> +   ((let ((symbols-with-pos-enabled nil))
> +      (eq f (car cl--labels-convert-cache)))
> +    (let ((print-symbols-bare nil))
> +      (message "Self-rewrite %S to %S (%S)" f (cdr cl--labels-convert-cache)
> +               symbols-with-pos-enabled))
> +    (cdr cl--labels-convert-cache))
>     (t
>      (let* ((found (assq f macroexpand-all-environment))
>             (replacement (and found

-- 
Alan Mackenzie (Nuremberg, Germany).