Need Advice: Rebinding in the Face of Buffer-Local Variables; `let-default'

[hokomo <hokomo@disroot.org>]

Hello,

I'm looking for a bit of Emacs Lisp advice regarding the behavior of buffer-local variables. Assume `lexical-binding' is t in all of the examples that follow.

First, some context. Buffer-local variables have the curious property that their let-bindings temporarily "disappear" when the buffer is changed. This is indeed documented in the manual [1] (see "Warning: ..."), but here's a slightly bigger example:

  (defvar local 'default)

  (let ((a (generate-new-buffer "a"))
        (b (generate-new-buffer "b")))
    (unwind-protect
        (progn
          ;; Make `local' buffer-local in `a'.
          (with-current-buffer a
            (make-local-variable 'local)
            (setq local 123))
          (message "(1) `local': %s" local)
          (with-current-buffer a
            (message "(2) `local': %s" local)
            ;; Bind `local' within `a'.
            (let ((local 456))
              (message "(3) `local': %s" local)
              ;; The binding of `local' "disappears".
              (with-current-buffer b
                (message "(4) `local': %s" local)
                ;; The binding of `local' is back.
                (with-current-buffer a
                  (message "(5) `local': %s" local))))))
      (kill-buffer a)
      (kill-buffer b)))

  ;; (1) ‘local’: default
  ;; (2) ‘local’: 123
  ;; (3) ‘local’: 456
  ;; (4) ‘local’: default
  ;; (5) ‘local’: 456

Note the outputs of (4) and (5). Conceptually, it's as if a buffer-local variable has a separate "stack" of dynamic bindings within each buffer where it is marked buffer-local. This is in addition to the variable's "default stack" of dynamic bindings that make up its "default" and "top-level default" values. The default value of a variable is therefore the value of its innermost non-buffer-local binding, or the global/top-level value if it is not let-bound. [2]

Now, here's a problem I ran into. Assume there's some library that provides a variable `var' and a function `f', and that `f' uses `var' somehow. I want to temporarily rebind `var' and invoke the function `f' to get some desired effect:

  (let ((var 123))
    (f))

However, assume also that (a) `var' just happens to be buffer-local within the current buffer B1 (e.g. because the user made it so), and (b) before doing the work that depends on `var', `f' first switches to buffer B2, does some bookkeeping, then does whatever depends on `var', and finally switches back to B1.

This unfortunate combination of events causes my binding of `var' to never be seen, as shown in the example at the beginning. Essentially, I was expecting to rebind the value of `var' so that it would be seen for the duration of the call to `f', but because `var' happened to be buffer-local (perhaps by user's choice or accident) and `f' changed the buffer, the binding vanished during the part of `f' where it mattered.

Note that it doesn't really matter whether `var' is buffer-local within B2 or not, as the effect will be the same. If it is, then B2's buffer-local binding of `var' will be seen. If it isn't, then `var's default value will be seen. Both of these are different from the binding that was established just before invoking `f' (which was conceptually added to the stack of B1's buffer-local bindings). For the purposes of my problem here, let's say that we are guaranteed `var' won't be buffer-local within B2 (e.g. because it's just a temporary one-off buffer).

Does this mean that the "simple task" of let-binding a variable around a call is inherently "unreliable" in Emacs Lisp due to the fact `var' might be buffer-local in B1? Is there a way out of this dilemma or is this "just the way it is"?

With the above guarantee about B2, one solution could be to explicitly rebind the *default* value of `var'. This can be done by temporarily switching from B1 to some buffer where `var' is not buffer-local, creating the binding, and switching back before invoking `f'. For example, here's how a `let-default' might look like:

  (defmacro let-default (bindings &rest body)
    (declare (indent 1))
    (macroexp-let2 nil buffer '(current-buffer)
      `(with-current-buffer (generate-new-buffer " *let-default*" t)
         (let ,bindings
           (with-current-buffer ,buffer
             ,@body)))))

The usage would then be:

  (let-default ((var 123))
    (f))

A small wrinkle is the existence of "built-in buffer-local variables" [3] (see `buffer-local-variables') such as `default-directory'. These variables are buffer-local within all newly created buffers so rebinding their default value would make no sense. The implementation of `let-default' could check for these and throw an error at macroexpansion-time to avoid programmer suprises. However, these variables don't fit the use case anyway, as they automatically violate our guarantee about B2.

While `let-default' helps, it feels very overkill, as does having to worry about this issue in the first place. But clearly situations like these do pop up so it seems worth talking about. Is there any other way out?

Perhaps it's not so much about binding but about switching buffers? Maybe the lesson here is that switching the current buffer should be done carefully and `f' should restore the original buffer before proceeding with the work that depends on `var', in order to anticipate `var' possibly being buffer-local and not disrupt any bindings established by the user or code? This pushes the responsibility onto library authors and sadly makes buffer-local variables "less transparent" to work with than one would hope for (for the programmer at least).

Or maybe if B2 is a temporary buffer it should first inherit all of B1's buffer-locals and their current bindings? Is there an idiomatic way to do this?

Thoughts?

[1] <https://www.gnu.org/software/emacs/manual/html_node/elisp/Intro-to-Buffer_002dLocal.html>
[2] <https://www.gnu.org/software/emacs/manual/html_node/elisp/Default-Value.html>
[3] <https://www.gnu.org/software/emacs/manual/html_node/elisp/Creating-Buffer_002dLocal.html#index-buffer_002dlocal_002dvariables>

Thanks,
hokomo