bug#65209: 30.0.50; Unexpected behaviour of setq-local

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Gerd Möllmann]

Evaluate the following

(progn
   (defvar my-var :default-value)
   (make-variable-buffer-local 'my-var)
   (let ((my-var :let-value))
     (setq-local my-var :buffer-local-new))
   my-var)
=> :default-value

(progn
   (defvar my-var1 :default-value)
   (make-variable-buffer-local 'my-var1)
   (setq my-var1 :buffer-local)
   (let ((my-var1 :let-value))
     (setq-local my-var1 :buffer-local-new))
   my-var1)
=> :buffer-local

In both cases, setq-local has no effect.

(I stumbled over this because lexical-binding is set to nil in *scratch*
when using persistent-scratch (MELPA).  It's quite convolutetd, but in
short: Custom loads persistent-scratch, which loads the contents of
*scratch*, and calls lisp-interaction-mode.  Lisp-interaction-mode
setq-local's lexical-binding to t, but this doesn't work because all
this happens during an eval-buffer that has a specbind of
lexical-binding to nil.)

In GNU Emacs 30.0.50 (build 1, aarch64-apple-darwin22.5.0, NS
  appkit-2299.60 Version 13.4.1 (Build 22F82)) of 2023-07-13 built on
  Mini.fritz.box
Repository revision: e0244f58042740c8fe914e7abe5b02611b176bc2
Repository branch: master
Windowing system distributor 'Apple', version 10.3.2299
System Description:  macOS 13.5

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Eli Zaretskii]

> Date: Thu, 10 Aug 2023 15:50:26 +0200
> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> 
> Evaluate the following
> 
> (progn
>    (defvar my-var :default-value)
>    (make-variable-buffer-local 'my-var)
>    (let ((my-var :let-value))
>      (setq-local my-var :buffer-local-new))
>    my-var)
> => :default-value
> 
> (progn
>    (defvar my-var1 :default-value)
>    (make-variable-buffer-local 'my-var1)
>    (setq my-var1 :buffer-local)
>    (let ((my-var1 :let-value))
>      (setq-local my-var1 :buffer-local-new))
>    my-var1)
> => :buffer-local
> 
> In both cases, setq-local has no effect.
> 
> (I stumbled over this because lexical-binding is set to nil in *scratch*
> when using persistent-scratch (MELPA).  It's quite convolutetd, but in
> short: Custom loads persistent-scratch, which loads the contents of
> *scratch*, and calls lisp-interaction-mode.  Lisp-interaction-mode
> setq-local's lexical-binding to t, but this doesn't work because all
> this happens during an eval-buffer that has a specbind of
> lexical-binding to nil.)

Adding Stefan.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Evaluate the following
>
> (progn
>   (defvar my-var :default-value)
>   (make-variable-buffer-local 'my-var)
>   (let ((my-var :let-value))
>     (setq-local my-var :buffer-local-new))
>   my-var)
> => :default-value
>
> (progn
>   (defvar my-var1 :default-value)
>   (make-variable-buffer-local 'my-var1)
>   (setq my-var1 :buffer-local)
>   (let ((my-var1 :let-value))
>     (setq-local my-var1 :buffer-local-new))
>   my-var1)
> => :buffer-local
>
> In both cases, setq-local has no effect.

But isn't it expected that `let' restores the value that was present
before entering `let'?

Michael.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Gerd Möllmann]

On 11.08.23 02:17, Michael Heerdegen wrote:
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
>> Evaluate the following
>>
>> (progn
>>    (defvar my-var :default-value)
>>    (make-variable-buffer-local 'my-var)
>>    (let ((my-var :let-value))
>>      (setq-local my-var :buffer-local-new))
>>    my-var)
>> => :default-value
>>
>> (progn
>>    (defvar my-var1 :default-value)
>>    (make-variable-buffer-local 'my-var1)
>>    (setq my-var1 :buffer-local)
>>    (let ((my-var1 :let-value))
>>      (setq-local my-var1 :buffer-local-new))
>>    my-var1)
>> => :buffer-local
>>
>> In both cases, setq-local has no effect.
> 
> But isn't it expected that `let' restores the value that was present
> before entering `let'?
That's right, but here's what the docs say about setq-local:

  -- Macro: setq-local &rest pairs
      PAIRS is a list of variable and value pairs.  This macro creates a
      buffer-local binding in the current buffer for each of the
      variables, and gives them a buffer-local value.  It is equivalent
      to calling ‘make-local-variable’ followed by ‘setq’ for each of the
      variables.

In my first example, setq-local doesn't create a buffer-local binding. 
In the second example, it doesn't set a buffer-local value.  So 
setq-local doesn't what the first sentence of the doc says it does.

The second sentence of the doc says something quite different than the 
first sentence because setq in the presence of a let-binding of course 
sets the value of the let-binding.

What's the right thing to do here I find hard to tell.

I think I'd personally prefer if setq-local did what the first sentence 
says.  Just to make things in general easier to understand, I guess.   I 
mean, the let-binding which changes the behaviour of 
lisp-interaction-mode in my case could be anywhere.  It's kind of like a 
spooky action at a distance.  Good luck debugging something like that.

But that's just my 2 cents.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> That's right, but here's what the docs say about setq-local:
>
>  -- Macro: setq-local &rest pairs
>      PAIRS is a list of variable and value pairs.  This macro creates a
>      buffer-local binding in the current buffer for each of the
>      variables, and gives them a buffer-local value.  It is equivalent
>      to calling ‘make-local-variable’ followed by ‘setq’ for each of the
>      variables.
>
> In my first example, setq-local doesn't create a buffer-local
> binding. In the second example, it doesn't set a buffer-local value.
> So setq-local doesn't what the first sentence of the doc says it does.

I think it does - it's only that `let' restores the old value.
>
> The second sentence of the doc says something quite different than the
> first sentence because setq in the presence of a let-binding of course
> sets the value of the let-binding.

I don't think this is what is happening.  E.g.

#+begin_src emacs-lisp
(progn
  (defvar my-var :default-value)
  (let ((my-var :let-value))
    (make-local-variable 'my-var)
    (setq my-var :buffer-local-new))
  (list my-var (local-variable-p 'my-var))) ;; (:buffer-local-new t)
#+end_src

`setq' sets the buffer local binding, not the global value the `let'
binding refers to.

> I mean, the let-binding which changes the behaviour of
> lisp-interaction-mode in my case could be anywhere.  It's kind of like
> a spooky action at a distance.

A different perspective on your recipe could also be: it changes the
value of a variable that controls evaluation of a buffer while
evaluating the contents of the buffer.  Probably not a good idea.

> Good luck debugging something like that.

The same can happen if you `setq' a global binding that happens to be
undone by a surrounding `let'.  Or that is `setq'd again by other code.

Michael.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Gerd Möllmann]

:close

On 11.08.23 07:53, Michael Heerdegen wrote:
> I don't think this is what is happening.  E.g.
> 
> #+begin_src emacs-lisp
> (progn
>    (defvar my-var :default-value)
>    (let ((my-var :let-value))
>      (make-local-variable 'my-var)
>      (setq my-var :buffer-local-new))
>    (list my-var (local-variable-p 'my-var))) ;; (:buffer-local-new t)
> #+end_src
> 
> `setq' sets the buffer local binding, not the global value the `let'
> binding refers to.

Indeed!  I stand corrected, and rest my case.  I guess I'm getting old.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Eli Zaretskii]

> Cc: 65209@debbugs.gnu.org
> Date: Fri, 11 Aug 2023 10:17:15 +0200
> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> 
> :close
> 
> On 11.08.23 07:53, Michael Heerdegen wrote:
> > I don't think this is what is happening.  E.g.
> > 
> > #+begin_src emacs-lisp
> > (progn
> >    (defvar my-var :default-value)
> >    (let ((my-var :let-value))
> >      (make-local-variable 'my-var)
> >      (setq my-var :buffer-local-new))
> >    (list my-var (local-variable-p 'my-var))) ;; (:buffer-local-new t)
> > #+end_src
> > 
> > `setq' sets the buffer local binding, not the global value the `let'
> > binding refers to.
> 
> Indeed!  I stand corrected, and rest my case.  I guess I'm getting old.

Don't we all?

I guess this bug can be closed, then?  Or is there anything left to do?

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Gerd Möllmann]

Yes please close it. 

Sent from my iPhone

> On Aug 11, 2023, at 13:09, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> 
>> 
>> Cc: 65209@debbugs.gnu.org
>> Date: Fri, 11 Aug 2023 10:17:15 +0200
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> 
>> :close
>> 
>>> On 11.08.23 07:53, Michael Heerdegen wrote:
>>> I don't think this is what is happening.  E.g.
>>> 
>>> #+begin_src emacs-lisp
>>> (progn
>>>   (defvar my-var :default-value)
>>>   (let ((my-var :let-value))
>>>     (make-local-variable 'my-var)
>>>     (setq my-var :buffer-local-new))
>>>   (list my-var (local-variable-p 'my-var))) ;; (:buffer-local-new t)
>>> #+end_src
>>> 
>>> `setq' sets the buffer local binding, not the global value the `let'
>>> binding refers to.
>> 
>> Indeed!  I stand corrected, and rest my case.  I guess I'm getting old.
> 
> Don't we all?
> 
> I guess this bug can be closed, then?  Or is there anything left to do?

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Fri, 11 Aug 2023 13:34:02 +0200
> Cc: michael_heerdegen@web.de, 65209@debbugs.gnu.org
> 
> Yes please close it. 

Done, thanks.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Drew Adams]

> I guess I'm getting old.

Such is life.  Anyone who's not getting old isn't alive.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > > #+begin_src emacs-lisp
> > > (progn
> > >    (defvar my-var :default-value)
> > >    (let ((my-var :let-value))
> > >      (make-local-variable 'my-var)
> > >      (setq my-var :buffer-local-new))
> > >    (list my-var (local-variable-p 'my-var))) ;; (:buffer-local-new t)
> > > #+end_src
> > >
> > > `setq' sets the buffer local binding, not the global value the `let'
> > > binding refers to.

> I guess this bug can be closed, then?  Or is there anything left to
> do?

After thinking more about it:

I could explain the behavior of Gerd's recipes using the manual, but it
doesn't explain the state in my above example _after_ the `let' has
been left.

The manual has a *Warning* section about mixing `let', buffer-local
variables, and changing the current buffer.  But it doesn't explain what
happens in a single buffer when you `let'-bind a (declared) variable,
make it local inside the `let', and finish the `let'.  Is the variable
still buffer-local afterwards, or does `let' remove the
buffer-localness?  With other words, is leaving a `let' the same as a
`setq' to the remembered value, or is it more?

Do we answer that question somewhere?  If not, I suggest to add that,
maybe as an example before the *Warning* section.

Michael.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>   65209@debbugs.gnu.org
> Date: Sun, 13 Aug 2023 06:16:42 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > > > `setq' sets the buffer local binding, not the global value the `let'
> > > > binding refers to.
> 
> > I guess this bug can be closed, then?  Or is there anything left to
> > do?
> 
> After thinking more about it:
> 
> I could explain the behavior of Gerd's recipes using the manual, but it
> doesn't explain the state in my above example _after_ the `let' has
> been left.
> 
> The manual has a *Warning* section about mixing `let', buffer-local
> variables, and changing the current buffer.  But it doesn't explain what
> happens in a single buffer when you `let'-bind a (declared) variable,
> make it local inside the `let', and finish the `let'.  Is the variable
> still buffer-local afterwards, or does `let' remove the
> buffer-localness?  With other words, is leaving a `let' the same as a
> `setq' to the remembered value, or is it more?
> 
> Do we answer that question somewhere?  If not, I suggest to add that,
> maybe as an example before the *Warning* section.

I added Stefan to this discussion near its beginning, but you've
removed his address from the CC list.  I've now re-added him back in
the hope that he can shed some light on the convoluted case you
describe.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Evaluate the following
>
> (progn
>   (defvar my-var :default-value)
>   (make-variable-buffer-local 'my-var)
>   (let ((my-var :let-value))
>     (setq-local my-var :buffer-local-new))
>   my-var)
> => :default-value
>
> (progn
>   (defvar my-var1 :default-value)
>   (make-variable-buffer-local 'my-var1)
>   (setq my-var1 :buffer-local)
>   (let ((my-var1 :let-value))
>     (setq-local my-var1 :buffer-local-new))
>   my-var1)
> => :buffer-local
>
> In both cases, setq-local has no effect.

Hmm... this is a bug in `make-local-variable` where we do:

  if (blv ? blv->local_if_set
      : (forwarded && BUFFER_OBJFWDP (valcontents.fwd)))
    {
      tem = Fboundp (variable);
      /* Make sure the symbol has a local value in this particular buffer,
	 by setting it to the same value it already has.  */
      Fset (variable, (EQ (tem, Qt) ? Fsymbol_value (variable) : Qunbound));
      return variable;
    }

I.e. we assume that if a var is `make-variable-buffer-local` then `set`
will make sure it has a buffer-local value, but this is not true if the
var is currently let-bound.

40 years after buffer-local and let-bindings were brought together and
we're still finding bugs :-(


        Stefan

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Hmm... this is a bug in `make-local-variable` where we do:
>
>   if (blv ? blv->local_if_set
>       : (forwarded && BUFFER_OBJFWDP (valcontents.fwd)))
>     {
>       tem = Fboundp (variable);
>       /* Make sure the symbol has a local value in this particular buffer,
> 	 by setting it to the same value it already has.  */
>       Fset (variable, (EQ (tem, Qt) ? Fsymbol_value (variable) : Qunbound));
>       return variable;
>     }
>
> I.e. we assume that if a var is `make-variable-buffer-local` then `set`
> will make sure it has a buffer-local value, but this is not true if the
> var is currently let-bound.

The patch below seems to work (for some reason, I'm getting several
errors in `make check` today, but they seems unaffected by this patch
(i.e. I get the same errors in a vanilla build)).


        Stefan


diff --git a/src/data.c b/src/data.c
index 619ab8fde64..6dc3a686aa8 100644
--- a/src/data.c
+++ b/src/data.c
@@ -2213,8 +2213,7 @@ DEFUN ("make-local-variable", Fmake_local_variable, Smake_local_variable,
   if (sym->u.s.trapped_write == SYMBOL_NOWRITE)
     xsignal1 (Qsetting_constant, variable);
 
-  if (blv ? blv->local_if_set
-      : (forwarded && BUFFER_OBJFWDP (valcontents.fwd)))
+  if (!blv && (forwarded && BUFFER_OBJFWDP (valcontents.fwd)))
     {
       tem = Fboundp (variable);
       /* Make sure the symbol has a local value in this particular buffer,
diff --git a/test/src/data-tests.el b/test/src/data-tests.el
index 680fdd57d71..d93bfec3015 100644
--- a/test/src/data-tests.el
+++ b/test/src/data-tests.el
@@ -768,6 +768,24 @@ data-tests-make-local-forwarded-var
                          (default-value 'last-coding-system-used))
                    '(no-conversion bug34318)))))
 
+(defvar-local data-tests--bug65209 :default-value)
+
+(ert-deftest data-tests-make-local-bug65209 ()
+  (let (vli vlo vgi vgo)
+    (with-temp-buffer
+      (let ((data-tests--bug65209 :let-bound-value))
+        ;; While `setq' would not make the var buffer-local
+        ;; (because we'd be setq-ing the let-binding instead),
+        ;; `setq-local' definitely should.
+        (setq-local data-tests--bug65209 :buffer-local-value)
+        (setq vgi (with-temp-buffer data-tests--bug65209))
+        (setq vli data-tests--bug65209))
+      (setq vgo (with-temp-buffer data-tests--bug65209))
+      (setq vlo data-tests--bug65209))
+    (should (equal (list vli vlo vgi vgo)
+                   '(:buffer-local-value :buffer-local-value
+                     :let-bound-value :default-value)))))
+
 (ert-deftest data-tests-make_symbol_constant ()
   "Can't set variable marked with 'make_symbol_constant'."
   (should-error (setq most-positive-fixnum 1) :type 'setting-constant))

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> The patch below seems to work (for some reason, I'm getting several
> errors in `make check` today, but they seems unaffected by this patch
> (i.e. I get the same errors in a vanilla build)).

Thanks that you looked closer.  I've installed the patch locally and
will report back when I find any problem.

> +(defvar-local data-tests--bug65209 :default-value)
> +
> +(ert-deftest data-tests-make-local-bug65209 ()
> +  (let (vli vlo vgi vgo)
> +    (with-temp-buffer
> +      (let ((data-tests--bug65209 :let-bound-value))
> +        ;; While `setq' would not make the var buffer-local
> +        ;; (because we'd be setq-ing the let-binding instead),
> +        ;; `setq-local' definitely should.
> +        (setq-local data-tests--bug65209 :buffer-local-value)
> +        (setq vgi (with-temp-buffer data-tests--bug65209))
> +        (setq vli data-tests--bug65209))
> +      (setq vgo (with-temp-buffer data-tests--bug65209))
> +      (setq vlo data-tests--bug65209))
> +    (should (equal (list vli vlo vgi vgo)
> +                   '(:buffer-local-value :buffer-local-value
> +                     :let-bound-value :default-value)))))
> +

That would actually be a very good example for
(info "(elisp) Intro to Buffer-Local") if you replace vli, vlo, vgi, vgo
with non-abbreviating names.

Michael.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Gerd Möllmann]

Sent from my iPhone

> On Aug 14, 2023, at 05:24, Michael Heerdegen <michael_heerdegen@web.de> wrote:
> 
> Thanks that you looked closer.

Yeah. Hats off to Stefan! 

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

My patch failed to account for DEFVAR_PER_BUFFER variables, so
I reworked it to handle that case as well and pushed the result to
`master`.


        Stefan

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> My patch failed to account for DEFVAR_PER_BUFFER variables, so
> I reworked it to handle that case as well and pushed the result to
> `master`.

Thank you.  I did not experience any problems so far.

Ehm - does somebody care about the manual?

Regards,

Michael.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Eli Zaretskii]

> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>  65209-done@debbugs.gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>
> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Sun, 20 Aug 2023 06:43:02 +0200
> 
> Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
> text editors" <bug-gnu-emacs@gnu.org> writes:
> 
> > My patch failed to account for DEFVAR_PER_BUFFER variables, so
> > I reworked it to handle that case as well and pushed the result to
> > `master`.
> 
> Thank you.  I did not experience any problems so far.
> 
> Ehm - does somebody care about the manual?

What do you mean?

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > Ehm - does somebody care about the manual?
>
> What do you mean?

My wish would be that the manual says a bit more about how let-binding
and buffer-local play together.  We have this

|    *Warning:* When a variable has buffer-local bindings in one or more
| buffers, ‘let’ rebinds the binding that’s currently in effect.  For
| instance, if the current buffer has a buffer-local value, ‘let’
| temporarily rebinds that.  If no buffer-local bindings are in effect,
| ‘let’ rebinds the default value.  If inside the ‘let’ you then change to
| a different current buffer in which a different binding is in effect,
| you won’t see the ‘let’ binding any more.  And if you exit the ‘let’
| while still in the other buffer, you won’t see the unbinding occur
| (though it will occur properly).

in (info "(elisp) Intro to Buffer-Local") [good] and also some words
about `set-default' being prevented from setting the default value
inside `let' in (info "(elisp) Default Value").

But the manual leaves open some questions (after reading the manual I
was not able to tell that the bug in this report was a bug):

How are bindings working when not changing the current buffer inside the
let?  If I create a buffer-local binding and the `let' is left, what
binding is left in the buffer - is the variable in that buffer still
buffer local, or does leaving the let maybe also remove the local
variable?

This example derived from Stefan's test clarifies most questions:

#+begin_src emacs-lisp
;; -*- lexical-binding: t -*-

(defvar-local my-var :default)

(let (inside-let local-inside-let inside-let-other-buffer local-outer outer)
  (with-temp-buffer
    (let ((my-var :let-bound))
      (setq inside-let my-var)
      (setq-local my-var :buffer-local)
      (setq local-inside-let my-var)
      (setq inside-let-other-buffer (with-temp-buffer my-var)))
    (setq local-outer my-var)
    (setq outer (with-temp-buffer my-var)))
  (list inside-let local-inside-let inside-let-other-buffer
        local-outer outer))

;; --> (:let-bound :buffer-local :let-bound
;;      :buffer-local :default)
#+end_src


Then, sometimes one sees this warning (e.g. when evaluating the above
example):

| Making my-var buffer-local while locally let-bound!

People will want to know what it means and whether (or when) they need
to care about it or can ignore it.


Finally, should we also talk about lexical variables vs. buffer-local?
They are different again:

#+begin_src emacs-lisp
(with-temp-buffer
  (let ((var :let-bound))
    (set (make-local-variable 'var) :buffer-local)
    var)) ;;    --> :let-bound, not :buffer-local!
#+end_src


Michael.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: 65209@debbugs.gnu.org,  gerd.moellmann@gmail.com,  monnier@iro.umontreal.ca
> Date: Tue, 22 Aug 2023 05:09:44 +0200
> 
> But the manual leaves open some questions (after reading the manual I
> was not able to tell that the bug in this report was a bug):
> 
> How are bindings working when not changing the current buffer inside the
> let?  If I create a buffer-local binding and the `let' is left, what
> binding is left in the buffer - is the variable in that buffer still
> buffer local, or does leaving the let maybe also remove the local
> variable?

If someone knows how to describe what happens in this situation in a
way that won't confuse the reader, I'm okay with adding such a
description.

> Finally, should we also talk about lexical variables vs. buffer-local?
> They are different again:

Same thing here.

IOW, it is impossible for me to judge whether some text is good enough
for the manual without seeing the proposed text.  IME, some very
complicated issues are better left unsaid, especially if they
sufficiently rare and obscure, to avoid making the more common
scenarios potentially less clear.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> IOW, it is impossible for me to judge whether some text is good enough
> for the manual without seeing the proposed text.  IME, some very
> complicated issues are better left unsaid, especially if they
> sufficiently rare and obscure, to avoid making the more common
> scenarios potentially less clear.

True, I know what you mean.  OTOH, a good understanding of the underlying
language (Elisp) makes the number of things that seem confusing or
obscure smaller.

Ok - I'm not good at writing documentation.  As a start, do you think
that it's possible to create a good addition to the manual out of this
stub?


| The following example demonstrates which binding is "seen" when
| let-binding and buffer-local variables are combined:
|
| (defvar var :default)
|
| (with-temp-buffer
|   (let ((var :let-bound))
|     var                            ; --> :let-bound
|     (setq-local var :buffer-local)
|     var                            ; --> :buffer-local
|     (with-temp-buffer var))        ; --> :let-bound
|   var                              ; --> :buffer-local
|   (with-temp-buffer var))          ; --> :default
|
|
| A lexical variable always refers to its lexical binding even when
| a buffer-local variable of the same name exists:
|
| (let ((lvar :lexical))
|   (setq-local lvar :local)
|   lvar                           ; --> :lexical
|   (symbol-value 'lvar))          ; --> :local


I would not suggest such an addition if I would not have missed that
kind of information in the manual.  In some situations the above stuff
gets important, and let-binding variables that are made buffer-local is
daily business.  Maybe this stuff is clear to people with a different
background - I don't know.


Michael.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: 65209@debbugs.gnu.org,  gerd.moellmann@gmail.com,  monnier@iro.umontreal.ca
> Date: Wed, 23 Aug 2023 05:47:18 +0200
> 
> Ok - I'm not good at writing documentation.  As a start, do you think
> that it's possible to create a good addition to the manual out of this
> stub?
> 
> 
> | The following example demonstrates which binding is "seen" when
> | let-binding and buffer-local variables are combined:
> |
> | (defvar var :default)
> |
> | (with-temp-buffer
> |   (let ((var :let-bound))
> |     var                            ; --> :let-bound
> |     (setq-local var :buffer-local)
> |     var                            ; --> :buffer-local
> |     (with-temp-buffer var))        ; --> :let-bound
> |   var                              ; --> :buffer-local
> |   (with-temp-buffer var))          ; --> :default
> |
> |
> | A lexical variable always refers to its lexical binding even when
> | a buffer-local variable of the same name exists:
> |
> | (let ((lvar :lexical))
> |   (setq-local lvar :local)
> |   lvar                           ; --> :lexical
> |   (symbol-value 'lvar))          ; --> :local

Yes, but it has to be described in English, not (only) in Lisp.

(TBH, the first one is almost self-evident, and I find nothing
surprising or unexpected in it.)

> I would not suggest such an addition if I would not have missed that
> kind of information in the manual.

Which part(s) did you miss?  AFAIU, the behavior with dynamic
variables is described in very clear terms which make the first
example completely described.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> (TBH, the first one is almost self-evident, and I find nothing
> surprising or unexpected in it.)

A more corner case is:

    (defvar-local my-foo :default)
    (with-temp-buffer
      (let ((my-foo :global-let))
        (with-temp-buffer
          (setq my-foo :set)
          (list my-foo (with-temp-buffer my-foo)))))

vs

    (defvar-local my-foo :default)
    (with-temp-buffer
      (let ((my-foo :global-let))
        (setq my-foo :set)
        (list my-foo (with-temp-buffer my-foo))))

IIRC this is done purposefully (the code has to work harder to get this
semantics), but not really documented, and I can't offhand give you
a good reason for that semantics.


        Stefan

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> A more corner case is:
>
>     (defvar-local my-foo :default)
>     (with-temp-buffer
>       (let ((my-foo :global-let))
>         (with-temp-buffer
>           (setq my-foo :set)
>           (list my-foo (with-temp-buffer my-foo)))))
>
> vs
>
>     (defvar-local my-foo :default)
>     (with-temp-buffer
>       (let ((my-foo :global-let))
>         (setq my-foo :set)
>         (list my-foo (with-temp-buffer my-foo))))
>
> IIRC this is done purposefully (the code has to work harder to get this
> semantics), but not really documented, and I can't offhand give you
> a good reason for that semantics.

Thanks for the example.  (info "(elisp) Creating Buffer-Local") says
about this case

| Making a variable buffer-local within a ‘let’-binding for that
| variable does not work reliably, unless the buffer in which you do
| this is not current either on entry to or exit from the ‘let’.
| This is because ‘let’ does not distinguish between different kinds
| of bindings; it knows only which variable the binding was made for.

I would really like to know a bit more than "doesn't work reliable".
Such situations happen in real Emacs life.  It's not always under the
control of the user or the programmer which variables are currently
let-bound when a buffer is created and buffer local variables might get
set.  "Anything can happen" leaves me with a very uneasy feeling.


Michael.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes, but it has to be described in English, not (only) in Lisp.

I'm not able to come with acceptable English anyway.

> (TBH, the first one is almost self-evident, and I find nothing
> surprising or unexpected in it.)

Hmm - ok, then maybe this is really not necessary to explain.  I don't
know.

> > I would not suggest such an addition if I would not have missed that
> > kind of information in the manual.
>
> Which part(s) did you miss?  AFAIU, the behavior with dynamic
> variables is described in very clear terms which make the first
> example completely described.

I mean the aspect that Stefan had fixed here: that a local variable
created inside a let-binding of the "same" variable survives the
termination of the `let' expression.  Which means in particular: `let'
seems to not restore the value of the variable because the former
binding is "shadowed" by the buffer local binding.  This is similar to
what is described in the "Warning" paragraph in the manual I cited, but
happening without changing the current buffer.

Could we maybe melt this case into this "Warning" paragraph?

Michael.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Stefan Monnier
>  <monnier@iro.umontreal.ca>,  65209@debbugs.gnu.org,
>   gerd.moellmann@gmail.com
> Date: Thu, 24 Aug 2023 03:06:32 +0200
> 
> > IIRC this is done purposefully (the code has to work harder to get this
> > semantics), but not really documented, and I can't offhand give you
> > a good reason for that semantics.
> 
> Thanks for the example.  (info "(elisp) Creating Buffer-Local") says
> about this case
> 
> | Making a variable buffer-local within a ‘let’-binding for that
> | variable does not work reliably, unless the buffer in which you do
> | this is not current either on entry to or exit from the ‘let’.
> | This is because ‘let’ does not distinguish between different kinds
> | of bindings; it knows only which variable the binding was made for.
> 
> I would really like to know a bit more than "doesn't work reliable".

I don't think we should tell more about it in the manual.  When the
manual says "doesn't work reliably", the meaning is clear: stay away
from doing that unless you know very well what you are doing.  So if
someone wants to use this regardless of the warning, and they don't
yet "know what they are doing well enough", they should either study
the code or ask some expert.  Such subtleties have no place in the
manual, since that would make it very large and hard to read for the
majority who will indeed "stay away".

> Such situations happen in real Emacs life.  It's not always under the
> control of the user or the programmer which variables are currently
> let-bound when a buffer is created and buffer local variables might get
> set.  "Anything can happen" leaves me with a very uneasy feeling.

Please describe such situations, where the escape (i.e. how to avoid
bumping into this subtlety, by either rewriting the code or using
auxiliary variables) is not clear.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: 65209@debbugs.gnu.org,  gerd.moellmann@gmail.com,  monnier@iro.umontreal.ca
> Date: Thu, 24 Aug 2023 05:31:20 +0200
> 
> > > I would not suggest such an addition if I would not have missed that
> > > kind of information in the manual.
> >
> > Which part(s) did you miss?  AFAIU, the behavior with dynamic
> > variables is described in very clear terms which make the first
> > example completely described.
> 
> I mean the aspect that Stefan had fixed here: that a local variable
> created inside a let-binding of the "same" variable survives the
> termination of the `let' expression.  Which means in particular: `let'
> seems to not restore the value of the variable because the former
> binding is "shadowed" by the buffer local binding.  This is similar to
> what is described in the "Warning" paragraph in the manual I cited, but
> happening without changing the current buffer.
> 
> Could we maybe melt this case into this "Warning" paragraph?

Feel free to suggest such a change, even if it isn't perfect, so that
we could have some real basis for discussing it.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Please describe such situations, where the escape (i.e. how to avoid
> bumping into this subtlety, by either rewriting the code or using
> auxiliary variables) is not clear.

I don't know.  The problem is that when setting buffer-local variables
it is unknown which bindings are in effect unless you explicitly look.

Recently there was a question, I tried to find it again but failed,
where a user wanted to set, or expected a variable to be set - was it
`lexical-binding'? - and that failed because there was a binding of
`lexical-binding' in the call chain that prevented this.  Too bad I
don't find it anymore, it was not long ago.  Maybe the example has even
been fixed now, but maybe it was also the case that Stefan mention, that
case where the right semantics were not trivial.

Anyway, I mean potentially any operation where a new buffer is created
and set up, and any bindings of variables that may be made buffer local
are also bound when creating a buffer.  Dunno, like, creating a new
dired buffer from dired, and you need to bind the same variable to
create the buffer that you also want to make buffer local.

I don't know whether something like this can happen at all.  Stefan's
examples use `setq', not `setq-local', so maybe everything is just fine
after his fix of this bug.  I can't tell for sure because I can't read C
very efficiently and we don't have doc describing this, so I am not
really a good person to ask.  Since Stefan is quiet, let's assume the
situation is good enough now.


Michael.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: monnier@iro.umontreal.ca,  65209@debbugs.gnu.org,  gerd.moellmann@gmail.com
> Date: Sat, 26 Aug 2023 04:09:11 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Please describe such situations, where the escape (i.e. how to avoid
> > bumping into this subtlety, by either rewriting the code or using
> > auxiliary variables) is not clear.
> 
> I don't know.  The problem is that when setting buffer-local variables
> it is unknown which bindings are in effect unless you explicitly look.

Only if you happen to write the convoluted code such as the one you
show, right?  Because in simple cases the results are predictable and
known in advance, right?

> I don't know whether something like this can happen at all.  Stefan's
> examples use `setq', not `setq-local', so maybe everything is just fine
> after his fix of this bug.  I can't tell for sure because I can't read C
> very efficiently and we don't have doc describing this, so I am not
> really a good person to ask.  Since Stefan is quiet, let's assume the
> situation is good enough now.

From where I stand, we cannot possibly include all of this in the
manual.  Emacs Lisp is a large and complex language, especially when
the code mixes some of the advanced features in complex ways.  Trying
to describe all of them and all of their combinations will just make
the manual harder to read.

There are times when I bump into situations like that, even though I
try to avoid such complex code as mush as possible (because it also
increases the probability of making mistakes due to complexity).  When
that happens, and the blunder is not clear upon examination and
debugging, I simply rewrite the code to use simpler patterns.  This is
my advice to anyone who bumps into these situations.

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Recently there was a question, I tried to find it again but failed,
> where a user wanted to set, or expected a variable to be set - was it
> `lexical-binding'? - and that failed because there was a binding of
> `lexical-binding' in the call chain that prevented this.

AFAIU you're describing the problem that lead to the current bug-report :-)


        Stefan

bug#65209: 30.0.50; Unexpected behaviour of setq-local

[Michael Heerdegen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > Recently there was a question, I tried to find it again but failed,
> > where a user wanted to set, or expected a variable to be set - was it
> > `lexical-binding'? - and that failed because there was a binding of
> > `lexical-binding' in the call chain that prevented this.
>
> AFAIU you're describing the problem that lead to the current bug-report :-)

Likely.  Ok, then I think we are done (the report is already closed).

Thx,

Michael.