bug#66940: Dynamic scoping is all weird now?

bug#66940: Dynamic scoping is all weird now?

[Dave Goel]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]

I was playing around, checking my sanity re : dynamic scoping.  I'm
probably rusty, but I could swear that on older emacsen, this sort of code
would print 33 ten times.


On emacs -Q with debian stable (28.2).
I try this


(progn
  (setq lexical-binding nil)

  (dotimes (ii 10)
    (defmacro mac ()
      `(message "%S" ,ii)
      (sit-for 0.1))
    (let ((old_ii ii))
      (setq ii 33)
      (mac)
      (setq ii old_ii)
      )))

Exits without error the first two times, although, strangely, i don't see
any messages the second time. (I expected to see ten 33's).


*(a) As in, the second time, it doesn't print 33 ten times, as we would
expect.*
*(b) The third time is stranger. *
Let's evaluate it a third time! The third time is even weirder.
The third time, it complains
Debugger entered--Lisp error: (void-variable ii)
  (list 'message "%S" ii)
  (lambda nil (list 'message "%S" ii) (list 'print ii) (sit-for 0.1))()
  macroexpand((mac) nil)
  macroexp-macroexpand((mac) nil)
  macroexp--expand-all((mac))
  macroexp--all-forms(((setq ii 33) (mac) (setq ii old_ii)))
  macroexp--expand-all((let ((old_ii ii)) (setq ii 33) (mac) (setq ii
old_ii)))

and so on.


The bug(?) was reproduced on 29.1 as well - by e1f on #emacs

*(c) It gets even funnier. e1f changed ii to var, but the third time, it
still complained about ii, with the above error message, and not about var.*
I reproduced that as well.

Dave

[-- Attachment #2: Type: text/html, Size: 2672 bytes --]

bug#66940: Dynamic scoping is all weird now?

[Dave Goel]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

I apologize.I meant to say it should print 0 to 10 the first time around
because ,ii should hardcode the value at the time of macro definition.
But, the second and third time are so absolutely weird?



On Sun, Nov 5, 2023 at 12:06 AM Dave Goel <deego3@gmail.com> wrote:

> I was playing around, checking my sanity re : dynamic scoping.  I'm
> probably rusty, but I could swear that on older emacsen, this sort of code
> would print 33 ten times.
>
>
> On emacs -Q with debian stable (28.2).
> I try this
>
>
> (progn
>   (setq lexical-binding nil)
>
>   (dotimes (ii 10)
>     (defmacro mac ()
>       `(message "%S" ,ii)
>       (sit-for 0.1))
>     (let ((old_ii ii))
>       (setq ii 33)
>       (mac)
>       (setq ii old_ii)
>       )))
>
> Exits without error the first two times, although, strangely, i don't see
> any messages the second time. (I expected to see ten 33's).
>
>
> *(a) As in, the second time, it doesn't print 33 ten times, as we would
> expect.*
> *(b) The third time is stranger. *
> Let's evaluate it a third time! The third time is even weirder.
> The third time, it complains
> Debugger entered--Lisp error: (void-variable ii)
>   (list 'message "%S" ii)
>   (lambda nil (list 'message "%S" ii) (list 'print ii) (sit-for 0.1))()
>   macroexpand((mac) nil)
>   macroexp-macroexpand((mac) nil)
>   macroexp--expand-all((mac))
>   macroexp--all-forms(((setq ii 33) (mac) (setq ii old_ii)))
>   macroexp--expand-all((let ((old_ii ii)) (setq ii 33) (mac) (setq ii
> old_ii)))
>
> and so on.
>
>
> The bug(?) was reproduced on 29.1 as well - by e1f on #emacs
>
> *(c) It gets even funnier. e1f changed ii to var, but the third time, it
> still complained about ii, with the above error message, and not about var.*
> I reproduced that as well.
>
> Dave
>
>
>

[-- Attachment #2: Type: text/html, Size: 3782 bytes --]

bug#66940: Dynamic scoping is all weird now?

[Gerd Möllmann]

Dave Goel <deego3@gmail.com> writes:

> On Sun, Nov 5, 2023 at 12:06 AM Dave Goel <deego3@gmail.com> wrote:
>
>> (progn
>>   (setq lexical-binding nil)
>>
>>   (dotimes (ii 10)
>>     (defmacro mac ()
>>       `(message "%S" ,ii)
>>       (sit-for 0.1))

The macro expansion of mac is what it returns, which is what sit-for
returns...

bug#66940: Dynamic scoping is all weird now?

[Dave Goel]

[-- Attachment #1: Type: text/plain, Size: 740 bytes --]

got you.
My macro should have been


(defmacro mac()
  `(progn
     (message "%S" ,ii)
     (sit-for 0.1)))

So, the original macro had a brain-fart basically amounted to returning nil
unconditionally.
In that case, why were we getting that strange error?



On Sun, Nov 5, 2023 at 1:04 AM Gerd Möllmann <gerd.moellmann@gmail.com>
wrote:

> Dave Goel <deego3@gmail.com> writes:
>
> > On Sun, Nov 5, 2023 at 12:06 AM Dave Goel <deego3@gmail.com> wrote:
> >
> >> (progn
> >>   (setq lexical-binding nil)
> >>
> >>   (dotimes (ii 10)
> >>     (defmacro mac ()
> >>       `(message "%S" ,ii)
> >>       (sit-for 0.1))
>
> The macro expansion of mac is what it returns, which is what sit-for
> returns...
>

[-- Attachment #2: Type: text/html, Size: 2091 bytes --]

bug#66940: Dynamic scoping is all weird now?

[Gerd Möllmann]

Dave Goel <deego3@gmail.com> writes:

> (defmacro mac()
>   `(progn
>      (message "%S" ,ii)
>      (sit-for 0.1)))
>
> So, the original macro had a brain-fart basically amounted to returning nil
> unconditionally.
> In that case, why were we getting that strange error?

Don't know exactly what the "strage error" refers to, but maybe a (pp
(macrexpand-all ...)) helps? In this case, the result is

"(progn
  (setq lexical-binding nil)
  (let ((upper-bound 10) (counter 0))
    (while (< counter upper-bound)
      (let ((ii counter))
	(defalias 'mac
	  (cons 'macro
		#'(lambda nil (list 'message \"%S\" ii) (sit-for 0.1))))
	(let ((old_ii ii)) (setq ii 33) (mac) (setq ii old_ii)))
      (setq counter (1+ counter)))))

One things that gets kind of more obvious is that defmacro is something
"global" in the sense that it sets mac's symbol function. You could
moved the defmacro out of the progn somewhere else.

Does that help?

bug#66940: Dynamic scoping is all weird now?

[Dave Goel]

[-- Attachment #1: Type: text/plain, Size: 359 bytes --]

On Sun, Nov 5, 2023 at 2:25 AM Gerd Möllmann <gerd.moellmann@gmail.com>
wrote:

>
> Don't know exactly what the "strage error" refers to
>

My original code may not make logical sense, but it was valid code?

It shouldn't have produced the errors described in sections (b) and (c) of
the bug report?

Those are both emacs bugs in (b) and (c)?

[-- Attachment #2: Type: text/html, Size: 1251 bytes --]

bug#66940: Dynamic scoping is all weird now?

[Gerd Möllmann]

Dave Goel <deego3@gmail.com> writes:

>>
>> Don't know exactly what the "strage error" refers to
>>
>
> My original code may not make logical sense, but it was valid code?
>
> It shouldn't have produced the errors described in sections (b) and (c) of
> the bug report?
>
> Those are both emacs bugs in (b) and (c)?

I must admit that I couldn't follow (c). In case (b) is don't see a bug.
Maybe you could write some easier to understand code which demonstates
what you mean?

bug#66940: Dynamic scoping is all weird now?

[Dave Goel]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

Gerd,

Ok.

I hope this bugreport is clearer -

Here's some code.

The variable ii is clearly set every time mac was defined OR called.

backquote should expand, and mac should have read its content as (list
'message "%S" ii).




(progn
  (setq lexical-binding nil)

  (dotimes (ii 10)
    (defmacro mac ()
      `(message "%S" ,ii)
      )
    (let
        ((old_ii ii))
      (setq ii 33)
      (mac)
      (setq ii old_ii)
      )))


You eval this code once. It works.
You eval this again. It works.
The third time, though, it lands you in the debugger. The very same code.
Why the third time? And, why the debugger?  ii is well set every time it is
used.

That was bug (b). That's clearly an emacs bug?


Here's (c) -

The third time, change every "ii" to "var" ,and change every "old_ii" to
"old_var".
And, evaluate it.
We still get the very same error, but the error involves ii, not var. The
code we are defining and eval'ing has no "ii" whatsoever. It should have
forgotten the old mac that involved ii. We defined a new mac involving
"var."


>
>

[-- Attachment #2: Type: text/html, Size: 1642 bytes --]

bug#66940: Dynamic scoping is all weird now?

[Dave Goel]

[-- Attachment #1: Type: text/plain, Size: 1666 bytes --]

Here's the simplest MRE I have so far -

(progn
  (setq lexical-binding nil)

  (let ((ii 1))
    (defmacro mac ()
      `(message "%S" ,ii)
      )
    (mac)))


Eval it three times, using C-x C-e.
Error on the third time.

(Following the prior response, I figured I will discuss on devel first to
confirm before officially reporting it, but was kinda directed back here.)



On Sun, Nov 5, 2023 at 2:07 PM Dave Goel <deego3@gmail.com> wrote:

> Gerd,
>
> Ok.
>
> I hope this bugreport is clearer -
>
> Here's some code.
>
> The variable ii is clearly set every time mac was defined OR called.
>
> backquote should expand, and mac should have read its content as (list
> 'message "%S" ii).
>
>
>
>
> (progn
>   (setq lexical-binding nil)
>
>   (dotimes (ii 10)
>     (defmacro mac ()
>       `(message "%S" ,ii)
>       )
>     (let
>         ((old_ii ii))
>       (setq ii 33)
>       (mac)
>       (setq ii old_ii)
>       )))
>
>
> You eval this code once. It works.
> You eval this again. It works.
> The third time, though, it lands you in the debugger. The very same code.
> Why the third time? And, why the debugger?  ii is well set every time it is
> used.
>
> That was bug (b). That's clearly an emacs bug?
>
>
> Here's (c) -
>
> The third time, change every "ii" to "var" ,and change every "old_ii" to
> "old_var".
> And, evaluate it.
> We still get the very same error, but the error involves ii, not var. The
> code we are defining and eval'ing has no "ii" whatsoever. It should have
> forgotten the old mac that involved ii. We defined a new mac involving
> "var."
>
>
>>
>>

[-- Attachment #2: Type: text/html, Size: 2895 bytes --]

bug#66940: Dynamic scoping is all weird now?

[Michael Heerdegen]

Dave Goel <deego3@gmail.com> writes:

> (progn
>   (setq lexical-binding nil)
>
>   (dotimes (ii 10)
>     (defmacro mac ()
>       `(message "%S" ,ii)
>       )
>     (let
>         ((old_ii ii))
>       (setq ii 33)
>       (mac)
>       (setq ii old_ii)
>       )))

This code has two problems:

(1) Setting `lexical-binding' in the middle of an evaluation does not
work as you think.  Either you eval en expression (like the above) using
lexical binding or dynamical binding.  If you change `lexical-binding'
in the middle you might get unexpected behavior.  Please set the binding
mode only in the file header, not in the code.  In the extremely rare
cases where you really must evaluate an expression using the other
binding mode use `eval' with an appropriate second argument.

(2) Macro expansion does not work as you think.  Most of the time macros
are expanded _once_ in the complete expression and then the result is
evaluated.  When you redefine a macro in the middle of evaluating code
using it, most of the time this will not have an effect because the
macro had already been expanded in the following code.


> You eval this code once. It works.
> You eval this again. It works.
> The third time, though, it lands you in the debugger. The very same
> code. Why the third time? And, why the debugger?  ii is well set every
> time it is used.

It's not surprising.

The first time you start evaluating the expression using lexical
binding.  Because your code sets lexical-binding to nil, the second time
you actually use dynamical binding completely.

The third run fails because when `mac' is expanded a variable `ii' is
undefined on top-level, so macroexpansion fails.

The second run is different: because the first run had used
lexical-binding (more or less completely, since you started the
interpreter using lexical-binding mode), the macroexpander of `mac' is
actually a closure that inherited the value of `ii' from the last
iteration of the first run.  Because of that the second run succeeds.


The second run redefines `mac' so that its reference to `ii' now means
the dynamical global variable, which doesn't exist.

So, I think everything indeed perfectly works as expected here.


Michael.

bug#66940: Dynamic scoping is all weird now?

[Gerd Möllmann]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> So, I think everything indeed perfectly works as expected here.

Thanks for explaining this much better than I could have done it.

bug#66940: Dynamic scoping is all weird now?

[Drew Adams]

> > So, I think everything indeed perfectly works as expected here.
> 
> Thanks for explaining this much better than I could have done it.

Ditto.  And for helping me _understand_ it.

bug#66940: Dynamic scoping is all weird now?

[Dave Goel]

[-- Attachment #1: Type: text/plain, Size: 3181 bytes --]

Michael,

Many thanks for that excellent explanation.

One more question -

With this, first, let's simplify our example as follows.

Let's eval this at the start:

(setq lexical-binding  nil)


Eval it, so we are operating in the old world of dynamic scoping.


(let ((ii 1))
    (defmacro mac ()
      `(print ,ii)
      )
    (mac))

As you said, with lexical-binding nil, the ii = 1 had no effect above. It
will use the ii from the runtime environment.


So, let's try this -

(setq ii 3)
(let ((ii 4))
  (mac))

It evals to 3, not 4. Per our understanding of dynamic scoping, we would
have expected it use the innermost ii?  Why does it use the global ii?



On Sun, Nov 5, 2023 at 8:52 PM Michael Heerdegen <michael_heerdegen@web.de>
wrote:

> Dave Goel <deego3@gmail.com> writes:
>
> > (progn
> >   (setq lexical-binding nil)
> >
> >   (dotimes (ii 10)
> >     (defmacro mac ()
> >       `(message "%S" ,ii)
> >       )
> >     (let
> >         ((old_ii ii))
> >       (setq ii 33)
> >       (mac)
> >       (setq ii old_ii)
> >       )))
>
> This code has two problems:
>
> (1) Setting `lexical-binding' in the middle of an evaluation does not
> work as you think.  Either you eval en expression (like the above) using
> lexical binding or dynamical binding.  If you change `lexical-binding'
> in the middle you might get unexpected behavior.  Please set the binding
> mode only in the file header, not in the code.  In the extremely rare
> cases where you really must evaluate an expression using the other
> binding mode use `eval' with an appropriate second argument.
>
> (2) Macro expansion does not work as you think.  Most of the time macros
> are expanded _once_ in the complete expression and then the result is
> evaluated.  When you redefine a macro in the middle of evaluating code
> using it, most of the time this will not have an effect because the
> macro had already been expanded in the following code.
>
>
> > You eval this code once. It works.
> > You eval this again. It works.
> > The third time, though, it lands you in the debugger. The very same
> > code. Why the third time? And, why the debugger?  ii is well set every
> > time it is used.
>
> It's not surprising.
>
> The first time you start evaluating the expression using lexical
> binding.  Because your code sets lexical-binding to nil, the second time
> you actually use dynamical binding completely.
>
> The third run fails because when `mac' is expanded a variable `ii' is
> undefined on top-level, so macroexpansion fails.
>
> The second run is different: because the first run had used
> lexical-binding (more or less completely, since you started the
> interpreter using lexical-binding mode), the macroexpander of `mac' is
> actually a closure that inherited the value of `ii' from the last
> iteration of the first run.  Because of that the second run succeeds.
>
>
> The second run redefines `mac' so that its reference to `ii' now means
> the dynamical global variable, which doesn't exist.
>
> So, I think everything indeed perfectly works as expected here.
>
>
> Michael.
>

[-- Attachment #2: Type: text/html, Size: 4004 bytes --]

bug#66940: Dynamic scoping is all weird now?

[Michael Heerdegen]

Dave Goel <deego3@gmail.com> writes:

> Let's eval this at the start:
>
> (setq lexical-binding nil)
>
> Eval it, so we are operating in the old world of dynamic scoping.

Ok, let's do this.

> (let ((ii 1))
>     (defmacro mac ()
>       `(print ,ii)
>       )
>     (mac))

So now:

  (symbol-function 'mac)
    ==> (macro . (lambda () (list 'print ii)))

> [...]
> So, let's try this -
>
> (setq ii 3)
> (let ((ii 4))
>   (mac))
>
> It evals to 3, not 4. Per our understanding of dynamic scoping, we
> would have expected it use the innermost ii?  Why does it use the
> global ii?

Remember that macros are not expanded on the fly - they are expanded
once for the complete expression in a separate step _before_ the actual
evaluation.  So:

(setq ii 3)
; ii --> 3

Expansion of the second expression:

(let ((ii 4))
  (mac))

  ~~> (let ((ii 4)) (print 3))
                    ^^^^^^^^^
                 expansion of (mac)

Evaluation:

(let ((ii 4)) (print 3))
; prints "3"


A different thing would be to define

     (defmacro mac ()
       '(print ii))

Then the expansion would still reference the variable.  But you could as
well just write (print ii) then.

To sum up: you are using macros wrong in this case.  Macros are not
expanded at run-time.  It is important to remember that.

Note that not even `cl-macrolet' works like this: the definition and
scope of the defined macros is local, but they are still expanded before
evaluation, so the expansion can't refer to run-time bindings as well.


Michael.

bug#66940: Dynamic scoping is all weird now?

[Dave Goel]

[-- Attachment #1: Type: text/plain, Size: 10 bytes --]

Thank you

[-- Attachment #2: Type: text/html, Size: 273 bytes --]

bug#66940: Dynamic scoping is all weird now?

[Michael Heerdegen]

Dave Goel <deego3@gmail.com> writes:

> Thank you

Then let's close this one.

What I still wanted to say (to you): no need to complicate things using
macros here in your example.  Use functions, then variable references
are simple and clear.

If a macro (expansion) needs to refer to run-time values, use macro
arguments to pass an appropriate expression (like a variable).

But this is overkill in your case, no macros needed.  The purpose of
Lisp macros is to extend the language and/or rewrite code (at compile
time).  More or less, only that.  Always think twice if you really need
macros, try to avoid using macros when possible.  This will make the
code easier to understand and maintain.  Only if a macro introduces an
abstraction that makes the code actually easier to understand justifies
using one.

Michael.

bug#66940: Dynamic scoping is all weird now?

[Dave Goel]

[-- Attachment #1: Type: text/plain, Size: 1136 bytes --]

Most absolutely, and of course.
I was just trying macros here to improve my understanding.
Which it happened, thanks to all y'all's excellent explanations.


On Tue, Nov 7, 2023 at 10:07 PM Michael Heerdegen <michael_heerdegen@web.de>
wrote:

> Dave Goel <deego3@gmail.com> writes:
>
> > Thank you
>
> Then let's close this one.
>
> What I still wanted to say (to you): no need to complicate things using
> macros here in your example.  Use functions, then variable references
> are simple and clear.
>
> If a macro (expansion) needs to refer to run-time values, use macro
> arguments to pass an appropriate expression (like a variable).
>
> But this is overkill in your case, no macros needed.  The purpose of
> Lisp macros is to extend the language and/or rewrite code (at compile
> time).  More or less, only that.  Always think twice if you really need
> macros, try to avoid using macros when possible.  This will make the
> code easier to understand and maintain.  Only if a macro introduces an
> abstraction that makes the code actually easier to understand justifies
> using one.
>
> Michael.
>

[-- Attachment #2: Type: text/html, Size: 1911 bytes --]