Propagating local variables?

Propagating local variables?

[John Wiegley]

I'd like to create a module, propagate.el, which defines a new class of buffer
local variables: buffer local variables that propagate to any buffers or
processes (in the case of exec-path and process-environment) created on behalf
of the parent.

For example, in certain language modes I have `exec-path' customized to pick
up specific versions of the compiler and debugger. If I run M-x gdb, I'd like
the debugger used to be the one associated with _that_ buffer -- without
making that debugger global to all other buffers (or even available at all).

Right now this idea lies at the heart of Shea Levy's nix-buffer package,
allowing you to have the equivalent of a "nix-shell" within Emacs, so that a
given buffer (and its related child buffers and processes) all see a
consistent set of dependencies as defined by some dir-locals.nix file.

A next step for this project is to abstract out the concept of "propagating
locals" to its own module. However, some of the aspects of implementing it
have proven very hacky, so I'm wondering if we need more low-level support for
this idea. For example, at the moment we do it with advice around
generate-new-buffer. What would be better is to have a general hook for
altering what happens when new buffers and processes are created, something
that could "provide an environment" upon creation of these.

All thoughts welcome,
-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Propagating local variables?

[Roland Winkler]

On Tue, May 15 2018, John Wiegley wrote:
> I'd like to create a module, propagate.el, which defines a new class
> of buffer local variables: buffer local variables that propagate to
> any buffers or processes (in the case of exec-path and
> process-environment) created on behalf of the parent.

This is an interesting idea.  I am not sure whether the following is
related to your thoughts.  I'll mention it anyway:

In GNU Elpa is djvu.el for editing djvu documents.  These documents come
with multiple layers for text, annotations, bookmarks, etc. (kind of
similar to pdf documents).  So when you visit such a document with
djvu.el it creates a buffer for each of these layers (plus a few extra
ones for "viewing these layers").  Conventional buffer-local variables
do not work in this case because the buffer-local variables need to be
shared across the buffers that refer to the same document, but buffers
for different documents should peacefully coexist.  Hence to make this
work djvu.el uses a conventional buffer-local variable whose value is a
vector which is the same object for all buffers belonging to one djvu
document (one vector for each djvu document).  The elements of this
vector are then the "document-local" variables where setting such an
element in one buffer is seen in all other buffers for the same
document, while buffers for different djvu documents do not interfere
with each other.  Another advantage is that temp buffers can easily
inherit these document-local variables.

I have always considered this a hack.  But I could not think of a better
solution.  Suggestions are welcome.

Roland

Re: Propagating local variables?

[Stefan Monnier]

> I have always considered this a hack.  But I could not think of a better
> solution.  Suggestions are welcome.

I think most other "multi-buffer" packages tend to treat one of their
buffers as a kind of "master" and have all others point to it, so they
can lookup shared local variables via something like
(buffer-local-value my-parent-buffer 'varname).


        Stefan

Re: Propagating local variables?

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> I think most other "multi-buffer" packages tend to treat one of their
SM> buffers as a kind of "master" and have all others point to it, so they can
SM> lookup shared local variables via something like (buffer-local-value
SM> my-parent-buffer 'varname).

If you're the author of all the code that relates to that buffer, sure. But I
can't expect shell.el to reference the buffer-local `process-environment' of
my Coq buffer in order to setup the right PATH for that project-specific
sub-shell.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Propagating local variables?

[Stefan Monnier]

> SM> I think most other "multi-buffer" packages tend to treat one of their
> SM> buffers as a kind of "master" and have all others point to it, so they can
> SM> lookup shared local variables via something like (buffer-local-value
> SM> my-parent-buffer 'varname).
> If you're the author of all the code that relates to that buffer, sure. But I
> can't expect shell.el to reference the buffer-local `process-environment' of
> my Coq buffer in order to setup the right PATH for that project-specific
> sub-shell.

I only mentioned it as a mechanism to share a variable among various
buffers.  We could decide that shell.el obeys a `parent-buffer` variable
(presumably set by some other package) when non-nil by fetching the
buffer-local `process-environment' of that buffer.


        Stefan

Re: Propagating local variables?

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

SM> I only mentioned it as a mechanism to share a variable among various
SM> buffers. We could decide that shell.el obeys a `parent-buffer` variable
SM> (presumably set by some other package) when non-nil by fetching the
SM> buffer-local `process-environment' of that buffer.

There are too many other cases where this is needed. Expecting it to happen
"by convention" means it won't happen in many cases where it should, simply
because the idea of referencing a parent buffer was omitted.

Now if we could go deeper down, and say that `process-environment' itself is a
"buffer group-local" variable, instead of just a buffer-local variable...

I guess what I want is object-based scoping for dynamic variables, rather than
just the global/local distinction we have now.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Propagating local variables?

[Stefan Monnier]

> SM> I only mentioned it as a mechanism to share a variable among various
> SM> buffers. We could decide that shell.el obeys a `parent-buffer` variable
> SM> (presumably set by some other package) when non-nil by fetching the
> SM> buffer-local `process-environment' of that buffer.
> There are too many other cases where this is needed.  Expecting it to happen
> "by convention" means it won't happen in many cases where it should, simply
> because the idea of referencing a parent buffer was omitted.

Agreed.

Maybe this is remotely related to the idea of major-mode variables.

We could imagine introducing a new notion of "setting", with operations like:

    (setting-set NAME VALUE CONTEXT)
    (setting-get NAME)

where `setting-get` returns the "nearest" setting of NAME.
CONTEXT could be:
- a window
- a frame
- a major-mode
- a buffer
- a directory
- a file
- a terminal
- ...

and the notion of proximity would impose some ordering between these
different CONTEXT values.  Of course, the current-buffer would be
"nearer" than the current frame, but what about the parent of the
current-buffer?
[ This is relevant to your use-case since in some situations we
  currently look at frame-parameters to decide how to build subprocess's
  environment variables (more specifically for the DISPLAY env-var).
  So a process-environment would actually want to be a combination of
  the environment specified from the frame and that specified by your
  parent's buffer.  ]

Of course, like all other proposals it doesn't solve the problem of
making sure this new mechanism is used everywhere.


        Stefan

Re: Propagating local variables?

[Roland Winkler]

On Wed, May 16 2018, Stefan Monnier wrote:
> I think most other "multi-buffer" packages tend to treat one of their
> buffers as a kind of "master" and have all others point to it, so they
> can lookup shared local variables via something like
> (buffer-local-value my-parent-buffer 'varname).

I see.  Would it make sense to also have a built-in counterpart
set-buffer-local-value for setting varname in my-parent-buffer?
(I have to check more carefully how a master buffer approach may
benefit / hurt djvu.el.)

Roland

Re: Propagating local variables?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'd like to create a module, propagate.el, which defines a new class of buffer
  > local variables: buffer local variables that propagate to any buffers or
  > processes (in the case of exec-path and process-environment) created on behalf
  > of the parent.

I'm not sure what the idea means.  What does "parent" mean in this
context?  Is a parent a buffer?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Propagating local variables?

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> I'm not sure what the idea means. What does "parent" mean in this context?
> Is a parent a buffer?

The parent is the buffer where you create the variables that will extend to
any children spawned "on behalf" of that buffer. So, not every buffer you
create while currently visiting that buffer, but those which match some kind
of predicate.

Thus, when editing a C file, you might want the compilation buffer, M-x shell,
M-x gdb, and a few others, to inherit the process-environment so that they are
all using the same version of the tools for a particular project, independent
of whether a dir-locals.el might be usable or not.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Propagating local variables?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The parent is the buffer where you create the variables that will extend to
  > any children spawned "on behalf" of that buffer. So, not every buffer you
  > create while currently visiting that buffer, but those which match some kind
  > of predicate.

That makes sense to me.  But I see two ways to do it:

* choose which buffers are its children
based on filtering their names.

* specify, when creating a buffer, what parent buffer (if any) it
should have.

I think the second way will be better controlled, because it won't
depend on heuristics.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Propagating local variables?

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> * specify, when creating a buffer, what parent buffer (if any) it should
> have.

> I think the second way will be better controlled, because it won't depend on
> heuristics.

I think so too, except that it won't work well with legacy modules, if they
weren't written with such inheritance in mind.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Propagating local variables?

[Shea Levy]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

Maybe a good transition path is to add an explicit (opt-in) parent
buffer mechanism when creating a buffer, and adding a hook when a buffer
is created *without* explicitly setting the parent (or explicitly
setting it to null) so the user can plug their own heuristics in until
legacy modules catch up.

John Wiegley <johnw@gnu.org> writes:

>>>>>> Richard Stallman <rms@gnu.org> writes:
>
>> * specify, when creating a buffer, what parent buffer (if any) it should
>> have.
>
>> I think the second way will be better controlled, because it won't depend on
>> heuristics.
>
> I think so too, except that it won't work well with legacy modules, if they
> weren't written with such inheritance in mind.
>
> -- 
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Propagating local variables?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think so too, except that it won't work well with legacy modules, if they
  > weren't written with such inheritance in mind.

It won't be terribly hard to make simple fixes in them.

Or we could make a variable to hold list of major modes for which they
want to make heuristic decisions about inherited buffer-locals.  This
way, users could request the heuristic approach for any modules for
which that is desirable.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Propagating local variables?

[John Shahid]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > The parent is the buffer where you create the variables that will extend to
>   > any children spawned "on behalf" of that buffer. So, not every buffer you
>   > create while currently visiting that buffer, but those which match some kind
>   > of predicate.
>
> That makes sense to me.  But I see two ways to do it:
>
> * choose which buffers are its children
> based on filtering their names.
>
> * specify, when creating a buffer, what parent buffer (if any) it
> should have.

Out of curiosity, if we go down this path, are we willing to change
functions such as `with-temp-buffer' to create a child buffer by default
? I recently had a use case for this feature. I wanted to set
`process-environment' as a buffer-local variable for certain buffers. I
achieved that using a mode-hook. This worked great with `call-process'
but didn't work so well when `process-lines' was used. `process-lines'
will create a new temp buffer and the buffer-local value is lost.

Now that I think about it, may be `with-temp-buffer' can take another
argument for the parent buffer that `process-lines' could use.

Anyway, I just wanted to share my use case, which could benefit from the
second option.

cheers,

js

Re: Propagating local variables?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Out of curiosity, if we go down this path, are we willing to change
  > functions such as `with-temp-buffer' to create a child buffer by default
  > ? 

I doubt that is desirable in most cases.  I think most of these temp
buffers should be left alone.



-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Propagating local variables?

[Shea Levy]

[-- Attachment #1: Type: text/plain, Size: 2158 bytes --]

To provide some more context, the hackiness is two-fold:

1. How do we decide which buffers should inherit the relevant variables?
   Currently in most cases I'm using a "if it starts with a space it's
   probably morally a child" heuristic, plus a few extras hard-coded,
   but it's not great.
2. How do we actually ensure the variables are propagated? I wrote the
   inherit-local package to do the propagation, but there's no obvious
   hook point to call it (thus the advice around generate-new-buffer)

"John Wiegley" <johnw@gnu.org> writes:

> I'd like to create a module, propagate.el, which defines a new class of buffer
> local variables: buffer local variables that propagate to any buffers or
> processes (in the case of exec-path and process-environment) created on behalf
> of the parent.
>
> For example, in certain language modes I have `exec-path' customized to pick
> up specific versions of the compiler and debugger. If I run M-x gdb, I'd like
> the debugger used to be the one associated with _that_ buffer -- without
> making that debugger global to all other buffers (or even available at all).
>
> Right now this idea lies at the heart of Shea Levy's nix-buffer package,
> allowing you to have the equivalent of a "nix-shell" within Emacs, so that a
> given buffer (and its related child buffers and processes) all see a
> consistent set of dependencies as defined by some dir-locals.nix file.
>
> A next step for this project is to abstract out the concept of "propagating
> locals" to its own module. However, some of the aspects of implementing it
> have proven very hacky, so I'm wondering if we need more low-level support for
> this idea. For example, at the moment we do it with advice around
> generate-new-buffer. What would be better is to have a general hook for
> altering what happens when new buffers and processes are created, something
> that could "provide an environment" upon creation of these.
>
> All thoughts welcome,
> -- 
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Propagating local variables?

[Clément Pit-Claudel]

On 2018-05-17 08:18, Shea Levy wrote:
> 2. How do we actually ensure the variables are propagated? I wrote the
>    inherit-local package to do the propagation, but there's no obvious
>    hook point to call it (thus the advice around generate-new-buffer)

We now have variable watchers:

> add-variable-watcher is a built-in function in ‘C source code’.
> 
> (add-variable-watcher SYMBOL WATCH-FUNCTION)
> 
> Cause WATCH-FUNCTION to be called when SYMBOL is set.
> 
> It will be called with 4 arguments: (SYMBOL NEWVAL OPERATION WHERE).
> SYMBOL is the variable being changed.
> NEWVAL is the value it will be changed to.
> OPERATION is a symbol representing the kind of change, one of: ‘set’,
> ‘let’, ‘unlet’, ‘makunbound’, and ‘defvaralias’.
> WHERE is a buffer if the buffer-local value of the variable is being
> changed, nil otherwise.
> 
> All writes to aliases of SYMBOL will call WATCH-FUNCTION too.

These could probably be used for that purpose:

- add watchers to the symbols of each shared variable
- create a buffer local list of dependent buffers to the parent buffer
- every time one of the watched variable changes, propagate the changes to the dependent buffers of the buffer it was changed in

It would be fairly understandable, too, since adding a watcher to a variable adds an informative message to that variable's docstring.

Of course, this assumes that the set of variables that you want to propagate is small and known beforehand.  As an alternative, a buffer-local-variable-change-hook might be a useful addition?

Clément.

Re: Propagating local variables?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > We now have variable watchers:

If this watcher facility is exactly right for propagating variables,
we can use it.

However, I tend to think that implementing propagating variables
without new primitive features will lead to kludges and/or doing the
wrong think in some cases.  I think we should implement it at C level
rather than make any compromises in their semantics.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)
Skype: No way! See https://stallman.org/skype.html.

Re: Propagating local variables?

[Stefan Monnier]

>> 2. How do we actually ensure the variables are propagated? I wrote the
>>    inherit-local package to do the propagation, but there's no obvious
>>    hook point to call it (thus the advice around generate-new-buffer)
> We now have variable watchers:

FWIW, I think it's better to avoid the problem of keeping copies in-sync
altogether (by storing the info at a single place).


        Stefan

Re: Propagating local variables?

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1187 bytes --]

Clément Pit-Claudel <cpitclaudel@gmail.com> schrieb am Do., 17. Mai 2018 um
16:28 Uhr:

> On 2018-05-17 08:18, Shea Levy wrote:
> > 2. How do we actually ensure the variables are propagated? I wrote the
> >    inherit-local package to do the propagation, but there's no obvious
> >    hook point to call it (thus the advice around generate-new-buffer)
>
> We now have variable watchers:
>
> > add-variable-watcher is a built-in function in ‘C source code’.
> >
> > (add-variable-watcher SYMBOL WATCH-FUNCTION)
> >
> > Cause WATCH-FUNCTION to be called when SYMBOL is set.
> >
> > It will be called with 4 arguments: (SYMBOL NEWVAL OPERATION WHERE).
> > SYMBOL is the variable being changed.
> > NEWVAL is the value it will be changed to.
> > OPERATION is a symbol representing the kind of change, one of: ‘set’,
> > ‘let’, ‘unlet’, ‘makunbound’, and ‘defvaralias’.
> > WHERE is a buffer if the buffer-local value of the variable is being
> > changed, nil otherwise.
> >
> > All writes to aliases of SYMBOL will call WATCH-FUNCTION too.
>
> These could probably be used for that purpose:
>

No, they are only for debugging purposes.

[-- Attachment #2: Type: text/html, Size: 1572 bytes --]

Re: Propagating local variables?

[Clément Pit-Claudel]

On 2018-05-19 08:06, Philipp Stephani wrote:
> No, they are only for debugging purposes. 

Why?

Re: Propagating local variables?

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 947 bytes --]

John Wiegley <johnw@gnu.org> schrieb am Di., 15. Mai 2018 um 21:40 Uhr:

> I'd like to create a module, propagate.el, which defines a new class of
> buffer
> local variables: buffer local variables that propagate to any buffers or
> processes (in the case of exec-path and process-environment) created on
> behalf
> of the parent.
>

There is some pior art to this (namely, `default-directory'), but such a
facility needs to be very carefully designed because it changes fundamental
aspects of ELisp in a subtle way. For example, callers of
`with-temp-buffer' can currently assume that buffer-local variables are
take from their global defaults and not from the "parent" buffer. This
needs to stay that way. So probably such an inheritance facility should be
opt-in and not affect existing code.
Before jumping to implementation discussions, we should analyze the precise
requirements for such a facility and make sure it's not a breaking change.

[-- Attachment #2: Type: text/html, Size: 1257 bytes --]

Re: Propagating local variables?

[Stefan Monnier]

> `with-temp-buffer' can currently assume that buffer-local variables are
> take from their global defaults and not from the "parent" buffer.  This
> needs to stay that way.  So probably such an inheritance facility should be
> opt-in and not affect existing code.

For that same reason I'd recommend we don't try and tweak the way
variables work, but instead build something else (maybe using
buffer-local variables internally).


        Stefan