Why is not end-of-defun-function buffer local?

Why is not end-of-defun-function buffer local?

[Lennart Borgman (gmail)]

It surprises me that beginning/end-of-defun-function are not always 
buffer local.

Re: Why is not end-of-defun-function buffer local?

[Stefan Monnier]

> It surprises me that beginning/end-of-defun-function are not always
> buffer local.

Why force them to be buffer-local?  When you need to set them in
a buffer, just do

  (set (make-local-variable 'end-of-defun-function) 'foo)

If you think that's too long, then you may want to request a new
`setq-local' macro.


        Stefan

Re: Why is not end-of-defun-function buffer local?

[Lennart Borgman (gmail)]

Stefan Monnier wrote:
>> It surprises me that beginning/end-of-defun-function are not always
>> buffer local.
> 
> Why force them to be buffer-local?  When you need to set them in
> a buffer, just do
> 
>   (set (make-local-variable 'end-of-defun-function) 'foo)
> 
> If you think that's too long, then you may want to request a new
> `setq-local' macro.


setq-local would be a good idea.

But I do not understand how you think in this case. Are not the 
variables above always meant to be buffer local?

Looking at some code that is a bit older it looks like some of it uses 
make-local-variable where it is not needed since the variables in 
question are always buffer local. From that I draw the conclusion that 
the code in Emacs uses make-variable-buffer-local more often now. Is not 
that the case?

Re: Why is not end-of-defun-function buffer local?

[Stefan Monnier]

>>> It surprises me that beginning/end-of-defun-function are not always
>>> buffer local.
>> 
>> Why force them to be buffer-local?  When you need to set them in
>> a buffer, just do
>> 
>> (set (make-local-variable 'end-of-defun-function) 'foo)
>> 
>> If you think that's too long, then you may want to request a new
>> `setq-local' macro.

> setq-local would be a good idea.

> But I do not understand how you think in this case.  Are not the
> variables above always meant to be buffer local?

I don't see why they should always be buffer-local.  It is quite
meaningful to set this variable globally so as to get a new improved
default behavior.

> Looking at some code that is a bit older it looks like some of it uses
> make-local-variable where it is not needed since the variables in question
> are always buffer local. From that I draw the conclusion that the code in
> Emacs uses make-variable-buffer-local more often now. Is not that the case?

make-variable-buffer-local has the following downsides:
1 - it cannot be reverted.
2 - it may be done too late.
3 - when you see `setq' it's not obvious that the setting is buffer-local
    unless you remember seeing the call to make-variable-buffer-local.
The second problem may also explain what you're seeing: some code may
set a variable before the make-variable-buffer-local gets run.
It's actually "common" to introduce bugs this way, because people see
"this is automatically buffer-local" in the C-h v info, so they just use
`setq' without realizing that the setq may occur before the package
gets loaded.
make-variable-buffer-local is not evil, but make-local-variable is much
tamer and more explicit, and it works just as well in most cases.


        Stefan

Re: Why is not end-of-defun-function buffer local?

[Lennart Borgman (gmail)]

Stefan Monnier wrote:
>> Looking at some code that is a bit older it looks like some of it uses
>> make-local-variable where it is not needed since the variables in question
>> are always buffer local. From that I draw the conclusion that the code in
>> Emacs uses make-variable-buffer-local more often now. Is not that the case?
> 
> make-variable-buffer-local has the following downsides:
> 1 - it cannot be reverted.
> 2 - it may be done too late.
> 3 - when you see `setq' it's not obvious that the setting is buffer-local
>     unless you remember seeing the call to make-variable-buffer-local.
> The second problem may also explain what you're seeing: some code may
> set a variable before the make-variable-buffer-local gets run.
> It's actually "common" to introduce bugs this way, because people see
> "this is automatically buffer-local" in the C-h v info, so they just use
> `setq' without realizing that the setq may occur before the package
> gets loaded.
> make-variable-buffer-local is not evil, but make-local-variable is much
> tamer and more explicit, and it works just as well in most cases.


Thanks, that was a good explanation. Why not add this to the doc string 
of make-variable-buffer-local?

Re: Why is not end-of-defun-function buffer local?

[Stefan Monnier]

>>> Looking at some code that is a bit older it looks like some of it uses
>>> make-local-variable where it is not needed since the variables in question
>>> are always buffer local. From that I draw the conclusion that the code in
>>> Emacs uses make-variable-buffer-local more often now. Is not that the case?
>> 
>> make-variable-buffer-local has the following downsides:
>> 1 - it cannot be reverted.
>> 2 - it may be done too late.
>> 3 - when you see `setq' it's not obvious that the setting is buffer-local
>> unless you remember seeing the call to make-variable-buffer-local.
>> The second problem may also explain what you're seeing: some code may
>> set a variable before the make-variable-buffer-local gets run.
>> It's actually "common" to introduce bugs this way, because people see
>> "this is automatically buffer-local" in the C-h v info, so they just use
>> `setq' without realizing that the setq may occur before the package
>> gets loaded.
>> make-variable-buffer-local is not evil, but make-local-variable is much
>> tamer and more explicit, and it works just as well in most cases.


> Thanks, that was a good explanation. Why not add this to the doc string of
> make-variable-buffer-local?

Oh, and since I've been looking at the low-level code that handles
variable lookup and things like that, there's another reason:
make-variable-buffer-local has a very subtle semantics which requires
pretty ugly and debatable C code.
More specifically, the problem is to decide *when* to make a variable
buffer-local.  I.e. Setting the variable via `setq' should make it
buffer-local, but setting it with `let' shouldn't.  But

   (let ((var 1))
     (setq var 2))

should not make `var' buffer-local either, because the `setq' is
"protected" within a let.  OTOH

   (let ((var 1))
     (with-current-buffer <otherbuf>
       (setq var 2)))

should make `var' buffer-local in <otherbuf> unless the code is itself
run within a `let' which was itself done in <otherbuf>.  Yuck!

So every `setq' on a variable that has been make-variable-buffer-local
may require walking up the current list of `let' bindings to decide
whether to make the variable buffer-local.  Yup, that's right:
the (setq var 2) will take time proportional to the stack depth :-(

And in order to be able to walk up the stack and decide which let
binding might be relevant, the runtime representation of some
let-bindings requires an extra cons-cell, which is not used for
anything else.


        Stefan

Re: Why is not end-of-defun-function buffer local?

[Lennart Borgman (gmail)]

Stefan Monnier wrote:
>>>> Looking at some code that is a bit older it looks like some of it uses
>>>> make-local-variable where it is not needed since the variables in question
>>>> are always buffer local. From that I draw the conclusion that the code in
>>>> Emacs uses make-variable-buffer-local more often now. Is not that the case?
>>> make-variable-buffer-local has the following downsides:
>>> 1 - it cannot be reverted.
>>> 2 - it may be done too late.
>>> 3 - when you see `setq' it's not obvious that the setting is buffer-local
>>> unless you remember seeing the call to make-variable-buffer-local.
>>> The second problem may also explain what you're seeing: some code may
>>> set a variable before the make-variable-buffer-local gets run.
>>> It's actually "common" to introduce bugs this way, because people see
>>> "this is automatically buffer-local" in the C-h v info, so they just use
>>> `setq' without realizing that the setq may occur before the package
>>> gets loaded.
>>> make-variable-buffer-local is not evil, but make-local-variable is much
>>> tamer and more explicit, and it works just as well in most cases.
> 
> 
>> Thanks, that was a good explanation. Why not add this to the doc string of
>> make-variable-buffer-local?
> 
> Oh, and since I've been looking at the low-level code that handles
> variable lookup and things like that, there's another reason:
> make-variable-buffer-local has a very subtle semantics which requires
> pretty ugly and debatable C code.
> More specifically, the problem is to decide *when* to make a variable
> buffer-local.  I.e. Setting the variable via `setq' should make it
> buffer-local, but setting it with `let' shouldn't.  But
> 
>    (let ((var 1))
>      (setq var 2))
> 
> should not make `var' buffer-local either, because the `setq' is
> "protected" within a let.  OTOH
> 
>    (let ((var 1))
>      (with-current-buffer <otherbuf>
>        (setq var 2)))
> 
> should make `var' buffer-local in <otherbuf> unless the code is itself
> run within a `let' which was itself done in <otherbuf>.  Yuck!
> 
> So every `setq' on a variable that has been make-variable-buffer-local
> may require walking up the current list of `let' bindings to decide
> whether to make the variable buffer-local.  Yup, that's right:
> the (setq var 2) will take time proportional to the stack depth :-(
> 
> And in order to be able to walk up the stack and decide which let
> binding might be relevant, the runtime representation of some
> let-bindings requires an extra cons-cell, which is not used for
> anything else.


Perhaps make-variable-buffer-local var could be treated like this:

1) When entering (let ((var 1)) make a buffer local copy of the variable 
just as if (make-local-variable 'var) was called before let.

2) When leaving (let ((var 1))...) delete the buffer local copy of the 
variable if it has the default value.

That is of course a slightly different semantic, but I wonder if it matters.

The advantage is that var could be treated just as if it was made buffer 
local with make-local-variable.

I might be misunderstanding something, of course, since I do not know 
this code.

Re: Why is not end-of-defun-function buffer local?

[martin rudalics]

 > More specifically, the problem is to decide *when* to make a variable
 > buffer-local.  I.e. Setting the variable via `setq' should make it
 > buffer-local, but setting it with `let' shouldn't.

Has this to be carved in stone?  Why should with

(defvar foo 1)
(make-variable-buffer-local 'foo)

the form below not set the default value of foo

(let ((foo 2))
   (setq-default foo 3))

and the following form

(progn
   (setq foo 4)
   (let ((foo 2))	
     (setq-default foo 3)))

set it?  Isn't code depending on one or the other behavior silly anyway?

Re: Why is not end-of-defun-function buffer local?

[Stefan Monnier]

>> More specifically, the problem is to decide *when* to make a variable
>> buffer-local.  I.e. Setting the variable via `setq' should make it
>> buffer-local, but setting it with `let' shouldn't.

> Has this to be carved in stone?  Why should with

> (defvar foo 1)
> (make-variable-buffer-local 'foo)

> the form below not set the default value of foo

> (let ((foo 2))
>   (setq-default foo 3))

> and the following form

> (progn
>   (setq foo 4)
>   (let ((foo 2))	
>     (setq-default foo 3)))

> set it?  Isn't code depending on one or the other behavior silly anyway?

Yes, the interaction between buffer-local bindings and let bindings
is messy.  There is no generally right answer, AFAICT.  So we just
developped some heuristic over time about what the semantics should look
like, so as to work OK in most cases.
The "automatically local when set" is of course making it yet
a bit messier.  Not sure what we should do about that.


        Stefan

Re: Why is not end-of-defun-function buffer local?

[martin rudalics]

 > The "automatically local when set" is of course making it yet
 > a bit messier.  Not sure what we should do about that.

I'm still missing you.  Consider

(defvar foo 0)

(let ((this-buffer (current-buffer)))
   (set (make-local-variable 'foo) 1)
   (let ((foo 2))
     (with-temp-buffer "*foo*"
       (let ((foo 3))
	(with-current-buffer this-buffer
	  foo)))))

Doesn't this exhibit the same "deep binding" behavior as
`make-variable-buffer-local'?

Re: Why is not end-of-defun-function buffer local?

[Stefan Monnier]

>> The "automatically local when set" is of course making it yet
>> a bit messier.  Not sure what we should do about that.

> I'm still missing you.  Consider

> (defvar foo 0)

> (let ((this-buffer (current-buffer)))
>   (set (make-local-variable 'foo) 1)
>   (let ((foo 2))
>     (with-temp-buffer "*foo*"
>       (let ((foo 3))
> 	(with-current-buffer this-buffer
> 	  foo)))))

> Doesn't this exhibit the same "deep binding" behavior as
> `make-variable-buffer-local'?

Of course.  It's just a bit less surprising because you then only see it
when you explicitly ask to make-local-variable.


        Stefan