bug#46573: 28.0.50; Error when edebugging setting unbound place

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Michael Heerdegen]

Hello,

when edebugging something like

  (setf PLACE VALUE)

Edebug stops after PLACE and displays its "binding".  While that would
make some sense for e.g. `cl-callf', in the above situation the behavior
is confusing because when the code is run the binding of PLACE is never
referenced, so this suggests that the code uses the binding of PLACE in
some way which is not the case.

But when PLACE is even unbound (e.g. and unbound variable, or an unbound
eieio-object slot [a totally legitimate case!]), Edebug kicks me out of
the session and raises an error.  Edebug can't be used for such code.
That needs fixing.

TIA,

Michael.

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Lars Ingebrigtsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> when edebugging something like
>
>   (setf PLACE VALUE)
>
> Edebug stops after PLACE and displays its "binding". 

This is due to this definition:

;; Autoload this `put' since a user might use C-u C-M-x on an expression
;; containing a non-trivial `push' even before gv.el was loaded.
;;;###autoload
(put 'gv-place 'edebug-form-spec '(form)) ;So-called "indirect spec".

That's certainly not correct for the simplest forms like

(setf foo 'bar)

And it's not really correct for things like

(setf (gethash 1 table) 'bar)

either, is it?  (Although it is amusing to see the results of
(gethash 1 table), it's not really helpful.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Stefan Monnier]

>> when edebugging something like
>>
>>   (setf PLACE VALUE)
>>
>> Edebug stops after PLACE and displays its "binding". 

Hmm... looks like we have a bug.

> This is due to this definition:
>
> ;; Autoload this `put' since a user might use C-u C-M-x on an expression
> ;; containing a non-trivial `push' even before gv.el was loaded.
> ;;;###autoload
> (put 'gv-place 'edebug-form-spec '(form)) ;So-called "indirect spec".
>
> That's certainly not correct for the simplest forms like

This has been there since the introduction of `gv`, so I think it
*is* correct.  The problem is elsewhere (likely introduced by some of
my recent changes to Edebug).


        Stefan

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> (put 'gv-place 'edebug-form-spec '(form)) ;So-called "indirect spec".
>>
>> That's certainly not correct for the simplest forms like
>
> This has been there since the introduction of `gv`, so I think it
> *is* correct.  The problem is elsewhere (likely introduced by some of
> my recent changes to Edebug).

Darn!  I thought I had finally learned how to read edebug specs.  :-/  I
though `form' meant that it's going to be instrumented?  Hm...  but it's
`(form)' which means, er, uhm.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Darn!  I thought I had finally learned how to read edebug specs.  :-/  I
> though `form' meant that it's going to be instrumented?  Hm...  but it's
> `(form)' which means, er, uhm.

Oh, it still means `form':

If the symbol has an Edebug specification, this @dfn{indirect
specification} should be either a list specification that is used in
place of the symbol

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Lars Ingebrigtsen]

I checked older Emacs versions -- (setf foo 'bar) works in Emacs 26.1,
but not in Emacs 27.1.  So I suspect it's not the recent changes that
broke this?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Stefan Monnier]

>>> (put 'gv-place 'edebug-form-spec '(form)) ;So-called "indirect spec".
>>>
>>> That's certainly not correct for the simplest forms like
>>
>> This has been there since the introduction of `gv`, so I think it
>> *is* correct.  The problem is elsewhere (likely introduced by some of
>> my recent changes to Edebug).
>
> Darn!  I thought I had finally learned how to read edebug specs.  :-/  I
> though `form' meant that it's going to be instrumented?  Hm...  but it's
> `(form)' which means, er, uhm.

Yes, it means Edebug rewrites (setf x 5) to something like:

    (edebug-after (edebug-before 1) 3 (setf (edebug-after 0 2 x) 5))

Whose behavior then depends on the definition of (edebug-after N1 N2 EXP)
as a "place", which is here:

    (put 'edebug-after 'gv-expander
         (lambda (do before index place)
           (gv-letplace (getter setter) place
             (funcall do `(edebug-after ,before ,index ,getter)
                      (lambda (store)
                        `(progn (edebug-after ,before ,index ,getter)
                                ,(funcall setter store)))))))

and indeed, there's the bug, introduced by Gemini's commit
d79cf638f278e50c22feb53d6ba556f5ce9d7853 which does (among various other
things):

    [...]
    * lisp/emacs-lisp/gv.el: Modify edebug-after's gv-expander to
    instrument in the setter as well as the getter.
    [...]

    diff --git a/lisp/emacs-lisp/gv.el b/lisp/emacs-lisp/gv.el
    --- a/lisp/emacs-lisp/gv.el
    +++ b/lisp/emacs-lisp/gv.el
    @@ -302,5 +302,7 @@
     (put 'edebug-after 'gv-expander
          (lambda (do before index place)
            (gv-letplace (getter setter) place
              (funcall do `(edebug-after ,before ,index ,getter)
    -                  setter))))
    +                  (lambda (store)
    +                    `(progn (edebug-after ,before ,index ,getter)
    +                            ,(funcall setter store)))))))

Gemini, how important is it to instrument the setter?  It is definitely
undesirable for Edebug, which you end up seeing the result of
computations which don't take place at all during
un-instrumented execution.  How 'bout using something like

    `(edebug-after ,before ,index ,(funcall setter store))

instead?


        Stefan

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 17 Feb 2021 00:21:17 +0100
> Cc: Michael Heerdegen <michael_heerdegen@web.de>, 46573@debbugs.gnu.org
> 
> I checked older Emacs versions -- (setf foo 'bar) works in Emacs 26.1,
> but not in Emacs 27.1.

Then we should try fixing it in Emacs 27.2, if that's possible without
risky changes.

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Gemini Lasswell]

Stefan Monnier writes:

> Gemini, how important is it to instrument the setter?  It is definitely
> undesirable for Edebug, which you end up seeing the result of
> computations which don't take place at all during
> un-instrumented execution.  How 'bout using something like
>
>     `(edebug-after ,before ,index ,(funcall setter store))
>
> instead?

Hi Stefan,

Sorry to be slow responding to this.  I don't think I would have added
this complication without a good reason, but it's not apparent to me now
what that reason was.  I hope to find time to look at it more this
weekend.

Best,

Gemini

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Stefan Monnier]

>> I checked older Emacs versions -- (setf foo 'bar) works in Emacs 26.1,
>> but not in Emacs 27.1.
> Then we should try fixing it in Emacs 27.2, if that's possible without
> risky changes.

I plan to install the patch below into `master` to fix this problem, but
I believe this change is sufficiently safe for the `emacs-27` branch, so
you agree I'll install it into `emacs-27` instead.

For code like

    (setf (car x) (cdr x))

The instrumented in Emacs-27.1 looked like:

    (let* ((v (edebug-after 0 2 x)))
      (progn
        (edebug-after (edebug-before 1) 3 (car v))
        (setcar v (edebug-after (edebug-before 4) 6
                                (cdr (edebug-after 0 5 x))))))

with the new code it will look like:

    (let* ((v (edebug-after 0 2 x)))
      (edebug-after
       (edebug-before 1) 3
       (setcar v (edebug-after (edebug-before 4) 6
                               (cdr (edebug-after 0 5 x))))))

For reference in Emacs-26, it looked like:

    (let* ((v (edebug-after 0 2 x)))
      (setcar v (edebug-after (edebug-before 4) 6
                              (cdr (edebug-after 0 5 x)))))

IIUC Gemini is too busy with real life to look at this now, but my
understanding of the motivation for his change was that the Emacs-26
code left instrumentation points 1 and 3 above completely unused (they
are generated by Edebug and then thrown away by macro expansion), giving
the mistaken impression (to the code-coverage checker) that these
execution paths are not exercised.  The new form should preserve the
benefit of Gemini's change in this respect while improving the behavior
during Edebug.


        Stefan


* lisp/emacs-lisp/gv.el (edebug-after): Don't run getter in the setter

This fixes bug#46573 which was introduced by commit
d79cf638f278e50c22feb53d6ba556f5ce9d7853.
The new code is a middle ground, which makes sure the instrumentation
point is used (so the coverage checker won't have ghost unreachable
instrumentation points) yet without artificially running the getter
when we only need to run the setter.


diff --git a/lisp/emacs-lisp/gv.el b/lisp/emacs-lisp/gv.el
index 3d8054950c..ce48e578e0 100644
--- a/lisp/emacs-lisp/gv.el
+++ b/lisp/emacs-lisp/gv.el
@@ -324,8 +324,7 @@ 'gv-place
        (gv-letplace (getter setter) place
          (funcall do `(edebug-after ,before ,index ,getter)
                   (lambda (store)
-                    `(progn (edebug-after ,before ,index ,getter)
-                            ,(funcall setter store)))))))
+                    `(edebug-after ,before ,index ,(funcall setter store)))))))
 
 ;;; The common generalized variables.
 

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Gemini Lasswell <gazally@runbox.com>, michael_heerdegen@web.de,
>   46573@debbugs.gnu.org
> Date: Mon, 01 Mar 2021 16:17:40 -0500
> 
> >> I checked older Emacs versions -- (setf foo 'bar) works in Emacs 26.1,
> >> but not in Emacs 27.1.
> > Then we should try fixing it in Emacs 27.2, if that's possible without
> > risky changes.
> 
> I plan to install the patch below into `master` to fix this problem, but
> I believe this change is sufficiently safe for the `emacs-27` branch, so
> you agree I'll install it into `emacs-27` instead.

I'm okay with installing this on the emacs-27 branch, but please wait
for a day to let others comment.

Thanks.

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> >> I checked older Emacs versions -- (setf foo 'bar) works in Emacs 26.1,
> >> but not in Emacs 27.1.
> > Then we should try fixing it in Emacs 27.2, if that's possible without
> > risky changes.
>
> I plan to install the patch below into `master` to fix this problem, but
> I believe this change is sufficiently safe for the `emacs-27` branch, so
> you agree I'll install it into `emacs-27` instead.

I tried your patch (thanks for working on it) now.  The error is gone,
but in my example

  (setf y 3)

when I step with edebug the thing still stops after the "y" to say the
value is "3", not after the number "3" - that position is omitted.  Is
this intentional?

Thanks,

Michael.

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Stefan Monnier]

>> >> I checked older Emacs versions -- (setf foo 'bar) works in Emacs 26.1,
>> >> but not in Emacs 27.1.
>> > Then we should try fixing it in Emacs 27.2, if that's possible without
>> > risky changes.
>>
>> I plan to install the patch below into `master` to fix this problem, but
>> I believe this change is sufficiently safe for the `emacs-27` branch, so
>> you agree I'll install it into `emacs-27` instead.
>
> I tried your patch (thanks for working on it) now.  The error is gone,
> but in my example
>
>   (setf y 3)
>
> when I step with edebug the thing still stops after the "y" to say the
> value is "3", not after the number "3" - that position is omitted.  Is
> this intentional?

Intentional, maybe not, but expected yes: the instrumentation adds
a "before" step right before the "GV assignment" itself and an "after"
step right after it.

Maybe we should add a special case for when `setf` is used for
a simple variable such that the variable is not instrumented at all,
just as is the case for `setq`?


        Stefan

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Maybe we should add a special case for when `setf` is used for
> a simple variable such that the variable is not instrumented at all,
> just as is the case for `setq`?

I don't know.  Maybe not.

When the value expression is a list, i.e. not something self-evaluating
like an number, the behavior doesn't seem so strange anymore to me, it
makes sense.  And my gut feeling is that it would be more confusing to
handle different place expressions differently than setq and setf.

So the fix is ok from my side.  Thanks for the clarification.

Regards,

Michael.

bug#46573: 28.0.50; Error when edebugging setting unbound place

[Stefan Monnier]

> I'm okay with installing this on the emacs-27 branch, but please wait
> for a day to let others comment.

Pushed,


        Stefan