Buffer-local process environments

Buffer-local process environments

[Augusto Stoffel]

[-- Attachment #1: Type: text/plain, Size: 2503 bytes --]

When developing software, it's often not possible or desirable to
install compilers and related tools globally, so (barring the
availability of fancy build tools) one has to fiddle with environment
variables.

Currently it's quite hard to set up Emacs to interact correctly with
external tools on a per-project basis, because there is no unified way
adjust the environment.  For instance:

- `M-x compile' has the `compilation-environment' variable.
- python-mode uses `python-shell-virtualenv-root' when launching a
  REPL.
- Eglot provides no other mechanism than setting `exec-path' and
  `process-environment' buffer-locally.
- etc.

Of course, the buffer-local variable approach will work with any
command that doesn't switch buffers, even if that command invents its
own method to adjust the environment.

So why not make buffer-local process environments an official thing?
By this I mean:

1. Fix commands that start external processes after switching buffers
   to do what the user means.

2. Provide a convenient way to set the process environment buffer
   locally.

Point 1 is simple.  I've attached a patch for `compile' as an example.
In the Python REPL case, I'm not sure whether it would be better to
just adapt `run-python' or make a change in comint itself.

Proposal for point 2
--------------------

Add two more special keywords to the file local variable mechanism
(besides `mode', `eval', etc):

- `env' to add an entry to `process-environment'.
- `path' to add an entry to `exec-path' and modify the PATH variable at
  the same time.

Hence, the local variables section of a Python file might look like
this:

# Local Variables:
# path: "~/path/to/some/virtualenv/bin"
# env: "VIRTUAL_ENV=$HOME/path/to/some/virtualenv"
# env: "LANG=C"
# End:

Of course, in practice it's more useful to include this in
.dir-locals.el, but the mechanism is the same.

Closing remarks
---------------

As an indication that this is a real problem, let me point out a list
of MELPA packages just to deal with Python virtualenvs:

- auto-virtualenv
- auto-virtualenvwrapper
- conda
- pungi
- python-environment
- pyvenv
- virtualenv
- virtualenvwrapper

I've tried a couple of those and was never satisfied.

My proposal here is inspired by the `inherienv' [1] and `envrc' [2]
MELPA packages.  The former provides an advice to solve point 1, the
latter solves point 2 by means of an external program called direnv.

[1]: https://github.com/purcell/inheritenv/
[2]: https://github.com/purcell/envrc


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-compile-respect-buffer-local-process-environmen.patch --]
[-- Type: text/x-patch, Size: 1877 bytes --]

From ece87c73bcb5f8e2a59769cfe981c1486e19857b Mon Sep 17 00:00:00 2001
From: Augusto Stoffel <arstoffel@gmail.com>
Date: Thu, 29 Apr 2021 12:45:04 +0200
Subject: [PATCH] Make `compile' respect buffer-local process environment

* lisp/progmodes/compile.el (compilation-start): Use
`process-environment' from original buffer in the compilation process.
---
 lisp/progmodes/compile.el | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/compile.el b/lisp/progmodes/compile.el
index 7a02c3a896..ee73f03c93 100644
--- a/lisp/progmodes/compile.el
+++ b/lisp/progmodes/compile.el
@@ -1779,6 +1779,8 @@ compilation-start
 	    (replace-regexp-in-string "-mode\\'" "" (symbol-name mode))))
 	 (thisdir default-directory)
 	 (thisenv compilation-environment)
+         (this-process-environment process-environment)
+         (env-buffer (when (local-variable-p 'process-environment) (buffer-name)))
 	 outwin outbuf)
     (with-current-buffer
 	(setq outbuf
@@ -1856,6 +1858,9 @@ compilation-start
 		"; default-directory: "
                 (prin1-to-string (abbreviate-file-name default-directory))
 		" -*-\n"
+                (if env-buffer
+                    (format "Process environment is local to buffer `%s'\n" env-buffer)
+                  "")
 		(format "%s started at %s\n\n"
 			mode-name
 			(substring (current-time-string) 0 19))
@@ -1875,7 +1880,7 @@ compilation-start
               (and (derived-mode-p 'comint-mode)
                    (comint-term-environment))
 	      (list (format "INSIDE_EMACS=%s,compile" emacs-version))
-	      (copy-sequence process-environment))))
+	      (copy-sequence this-process-environment))))
         (setq-local compilation-arguments
                     (list command mode name-function highlight-regexp))
         (setq-local revert-buffer-function 'compilation-revert-buffer)
-- 
2.30.2

Re: Buffer-local process environments

[Eli Zaretskii]

> From: Augusto Stoffel <arstoffel@gmail.com>
> Date: Thu, 29 Apr 2021 12:56:36 +0200
> 
> When developing software, it's often not possible or desirable to
> install compilers and related tools globally, so (barring the
> availability of fancy build tools) one has to fiddle with environment
> variables.
> 
> Currently it's quite hard to set up Emacs to interact correctly with
> external tools on a per-project basis, because there is no unified way
> adjust the environment.  For instance:
> 
> - `M-x compile' has the `compilation-environment' variable.
> - python-mode uses `python-shell-virtualenv-root' when launching a
>   REPL.
> - Eglot provides no other mechanism than setting `exec-path' and
>   `process-environment' buffer-locally.
> - etc.

Can you explain why Eglot's way does not provide a solution for these
situations?

Thanks.

Re: Buffer-local process environments

[Augusto Stoffel]

On Thu, 29 Apr 2021 at 15:30, Eli Zaretskii <eliz@gnu.org> wrote:

> Can you explain why Eglot's way does not provide a solution for these
> situations?

Unlike, say, M-x compile with its `compilation-environment' variable,
Eglot does not provide its own mechanism to specify the environment in
which a language server is executed.  (For the record, I think this is
the correct thing to do.)

Still, there are several options to make Eglot launch the correct
language server: you can write a little wrapper script for language
server, or launch Emacs from a terminal with the correct environment,
etc, etc.  But setting `process-environment' and `exec-path'
buffer-locally is by far the most pleasant option in my opinion.

>
> Thanks.

Re: Buffer-local process environments

[Eli Zaretskii]

> From: Augusto Stoffel <arstoffel@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 29 Apr 2021 14:40:26 +0200
> 
> On Thu, 29 Apr 2021 at 15:30, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Can you explain why Eglot's way does not provide a solution for these
> > situations?
> 
> Unlike, say, M-x compile with its `compilation-environment' variable,
> Eglot does not provide its own mechanism to specify the environment in
> which a language server is executed.  (For the record, I think this is
> the correct thing to do.)
> 
> Still, there are several options to make Eglot launch the correct
> language server: you can write a little wrapper script for language
> server, or launch Emacs from a terminal with the correct environment,
> etc, etc.  But setting `process-environment' and `exec-path'
> buffer-locally is by far the most pleasant option in my opinion.

That's exactly why I asked: you said that Eglot provides the ability
to do the latter, so it sounds like the correct solution is for all
the other packages to follow Eglot's example, no?

Re: Buffer-local process environments

[Augusto Stoffel]

On Thu, 29 Apr 2021 at 15:52, Eli Zaretskii <eliz@gnu.org> wrote:

> That's exactly why I asked: you said that Eglot provides the ability
> to do the latter, so it sounds like the correct solution is for all
> the other packages to follow Eglot's example, no?

Yes, I believe it's correct for all other packages and commands to
follow Eglot's example and not break if the environment or exec-path are
set buffer-locally.

In Eglot's case, this is not a big feat, since it calls make-process in
without ever changing buffers.  For other packages and commands, a bit
of extra care may be necessary (cf. my patch to `compile').

Re: Buffer-local process environments

[Stefan Monnier]

> So why not make buffer-local process environments an official thing?

Sounds fine to me.

To fix cases like `compile` or `run-python`, I think patches would be
welcome regardless of what is decided in this discussion: those
behaviors look like plain bugs to me.

> # Local Variables:
> # path: "~/path/to/some/virtualenv/bin"
> # env: "VIRTUAL_ENV=$HOME/path/to/some/virtualenv"
> # env: "LANG=C"
> # End:

A few comments:
- I don't much like "special local vars" (like `eval` and `mode`), so if
  we can find a more general solution (i.e. one that can be useful for
  other settings), that would be better.  Maybe

    # push exec-path: "~/path/to/some/virtualenv/bin"

- both `exec-path` and `process-environment` are "dangerous" variables,
  so encouraging such uses sounds rather risky.

- I'd write `path` above as `exec-path` or `PATH`, or even `$PATH`,
  otherwise it's unclear which "path" is meant.

> --- a/lisp/progmodes/compile.el
> +++ b/lisp/progmodes/compile.el
> @@ -1779,6 +1779,8 @@ compilation-start
>  	    (replace-regexp-in-string "-mode\\'" "" (symbol-name mode))))
>  	 (thisdir default-directory)
>  	 (thisenv compilation-environment)
> +         (this-process-environment process-environment)
> +         (env-buffer (when (local-variable-p 'process-environment) (buffer-name)))
>  	 outwin outbuf)
>      (with-current-buffer
>  	(setq outbuf
> @@ -1856,6 +1858,9 @@ compilation-start
>  		"; default-directory: "
>                  (prin1-to-string (abbreviate-file-name default-directory))
>  		" -*-\n"
> +                (if env-buffer
> +                    (format "Process environment is local to buffer `%s'\n" env-buffer)
> +                  "")
>  		(format "%s started at %s\n\n"
>  			mode-name
>  			(substring (current-time-string) 0 19))
> @@ -1875,7 +1880,7 @@ compilation-start
>                (and (derived-mode-p 'comint-mode)
>                     (comint-term-environment))
>  	      (list (format "INSIDE_EMACS=%s,compile" emacs-version))
> -	      (copy-sequence process-environment))))
> +	      (copy-sequence this-process-environment))))
>          (setq-local compilation-arguments
>                      (list command mode name-function highlight-regexp))
>          (setq-local revert-buffer-function 'compilation-revert-buffer)

Does this work correctly when you `M-x recompile`?


        Stefan

Re: Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

Hi Augusto,

> So why not make buffer-local process environments an official thing?
> By this I mean:
>
> 1. Fix commands that start external processes after switching buffers
>    to do what the user means.
>
> 2. Provide a convenient way to set the process environment buffer
>    locally.
>
> Point 1 is simple.  I've attached a patch for `compile' as an example.
> In the Python REPL case, I'm not sure whether it would be better to
> just adapt `run-python' or make a change in comint itself.

Have you checked, that this works for remote processes? this is already
hairy, and I fear it would be even more complicate. If we are going to
change something in this department, I'd be happy to have also a
rock-solid solution for remote processes.

See also connection-local variables, which are an attempt to make life easier.

Best regards, Michael.

Re: Buffer-local process environments

[Augusto Stoffel]

[-- Attachment #1: Type: text/plain, Size: 3392 bytes --]

On Thu, 29 Apr 2021 at 10:02, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> So why not make buffer-local process environments an official thing?
>
> Sounds fine to me.
>
> To fix cases like `compile` or `run-python`, I think patches would be
> welcome regardless of what is decided in this discussion: those
> behaviors look like plain bugs to me.

Great!

>
>> # Local Variables:
>> # path: "~/path/to/some/virtualenv/bin"
>> # env: "VIRTUAL_ENV=$HOME/path/to/some/virtualenv"
>> # env: "LANG=C"
>> # End:
>
> A few comments:
> - I don't much like "special local vars" (like `eval` and `mode`), so if
>   we can find a more general solution (i.e. one that can be useful for
>   other settings), that would be better.  Maybe
>
>     # push exec-path: "~/path/to/some/virtualenv/bin"

Some notation equivalent to

# eval: (add-to-list (make-local-variable var) value)

might be handy, as well as checkable for safety.  But in the env var
case it would be important to run `substitute-env-vars', so it's a
special case.

>
> - both `exec-path` and `process-environment` are "dangerous" variables,
>   so encouraging such uses sounds rather risky.

True, but setting the environment for buffers in a project is something
one needs to do all the time.  After confirming once, it has to get out
of the way and just happen automatically.  The details of how this works
of course are worth discussing.

>
> - I'd write `path` above as `exec-path` or `PATH`, or even `$PATH`,
>   otherwise it's unclear which "path" is meant.
>
>> --- a/lisp/progmodes/compile.el
>> +++ b/lisp/progmodes/compile.el
>> @@ -1779,6 +1779,8 @@ compilation-start
>>  	    (replace-regexp-in-string "-mode\\'" "" (symbol-name mode))))
>>  	 (thisdir default-directory)
>>  	 (thisenv compilation-environment)
>> +         (this-process-environment process-environment)
>> +         (env-buffer (when (local-variable-p 'process-environment) (buffer-name)))
>>  	 outwin outbuf)
>>      (with-current-buffer
>>  	(setq outbuf
>> @@ -1856,6 +1858,9 @@ compilation-start
>>  		"; default-directory: "
>>                  (prin1-to-string (abbreviate-file-name default-directory))
>>  		" -*-\n"
>> +                (if env-buffer
>> +                    (format "Process environment is local to buffer `%s'\n" env-buffer)
>> +                  "")
>>  		(format "%s started at %s\n\n"
>>  			mode-name
>>  			(substring (current-time-string) 0 19))
>> @@ -1875,7 +1880,7 @@ compilation-start
>>                (and (derived-mode-p 'comint-mode)
>>                     (comint-term-environment))
>>  	      (list (format "INSIDE_EMACS=%s,compile" emacs-version))
>> -	      (copy-sequence process-environment))))
>> +	      (copy-sequence this-process-environment))))
>>          (setq-local compilation-arguments
>>                      (list command mode name-function highlight-regexp))
>>          (setq-local revert-buffer-function 'compilation-revert-buffer)
>
> Does this work correctly when you `M-x recompile`?

I wasn't aware of recompile.  The attached patch works with it.  The
attached version breaks with Tramp, but I think this is a Tramp bug.

In fact, even without the patch, if I make my `process-environment'
buffer local, then `M-x cd' to a ssh path, then do `M-! env', I see a
mixture of my machine's environment and the remote machine's environment.
For instance, PATH has the local machine value.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-compile-respect-buffer-local-process-environmen.patch --]
[-- Type: text/x-patch, Size: 1508 bytes --]

From f803ce025b4161bea68f115ac50ba1174ef863e3 Mon Sep 17 00:00:00 2001
From: Augusto Stoffel <arstoffel@gmail.com>
Date: Thu, 29 Apr 2021 12:45:04 +0200
Subject: [PATCH] Make `compile' respect buffer-local process environment

* lisp/progmodes/compile.el (compilation-start): Use
`process-environment' from original buffer in the compilation process.
---
 lisp/progmodes/compile.el | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lisp/progmodes/compile.el b/lisp/progmodes/compile.el
index 7a02c3a896..52873d2f29 100644
--- a/lisp/progmodes/compile.el
+++ b/lisp/progmodes/compile.el
@@ -1779,6 +1779,9 @@ compilation-start
 	    (replace-regexp-in-string "-mode\\'" "" (symbol-name mode))))
 	 (thisdir default-directory)
 	 (thisenv compilation-environment)
+         (bufferenv (if (local-variable-p 'process-environment)
+                        process-environment
+                      'global))
 	 outwin outbuf)
     (with-current-buffer
 	(setq outbuf
@@ -1846,6 +1849,9 @@ compilation-start
         ;; NB: must be done after (funcall mode) as that resets local variables
         (setq-local compilation-directory thisdir)
         (setq-local compilation-environment thisenv)
+        (if (eq bufferenv 'global)
+            (kill-local-variable 'process-environment)
+          (setq-local process-environment bufferenv))
 	(if highlight-regexp
             (setq-local compilation-highlight-regexp highlight-regexp))
         (if (or compilation-auto-jump-to-first-error
-- 
2.30.2

Re: Buffer-local process environments

[Augusto Stoffel]

On Thu, 29 Apr 2021 at 17:37, Michael Albinus <michael.albinus@gmx.de> wrote:

> Augusto Stoffel <arstoffel@gmail.com> writes:
>
> Hi Augusto,
>

Hi Michael,

>> Point 1 is simple.  I've attached a patch for `compile' as an example.
>> In the Python REPL case, I'm not sure whether it would be better to
>> just adapt `run-python' or make a change in comint itself.
>
> Have you checked, that this works for remote processes? this is already

In fact it breaks with remote processes, see at the end of my previous
message in reply to Stefan.

But I think it's a Tramp bug.  Tramp is supposed to completely ignore
`process-environment', right?  So it shouldn't matter if that variable
is buffer local or not.

> hairy, and I fear it would be even more complicate. If we are going to
> change something in this department, I'd be happy to have also a
> rock-solid solution for remote processes.

True.  "Connection-local variable" sounds a lot like "project-local
variable" to me.  Maybe there could be a unified mechanism.

>
> See also connection-local variables, which are an attempt to make life easier.
>
> Best regards, Michael.

Re: Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

> I wasn't aware of recompile.  The attached patch works with it.  The
> attached version breaks with Tramp, but I think this is a Tramp bug.
>
> In fact, even without the patch, if I make my `process-environment'
> buffer local, then `M-x cd' to a ssh path, then do `M-! env', I see a
> mixture of my machine's environment and the remote machine's environment.
> For instance, PATH has the local machine value.

Tramp uses already a hack. Obviously, it cannot use process-environment
as-it-is on a remote machine, because it is, ahem, a remote machine and
not the local one. Therefore, Tramp forces only environemt variables,
which are not set in the defauzlt value of process-environment, to the
remote. The idea is, that specific remote environment variables are
added to process-environment via a let-binding.

A buffer-local process-environment is another, new game player.

Best regards, Michael.

Re: Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

> Hi Michael,

Hi Augusto,

>> Have you checked, that this works for remote processes? this is already
>
> In fact it breaks with remote processes, see at the end of my previous
> message in reply to Stefan.
>
> But I think it's a Tramp bug.  Tramp is supposed to completely ignore
> `process-environment', right?  So it shouldn't matter if that variable
> is buffer local or not.

No. See my reply the other message.

Granted, it is a hack. I would be happy if we could use something else,
but this is the state of affairs we cannot break reckless.

>> hairy, and I fear it would be even more complicate. If we are going to
>> change something in this department, I'd be happy to have also a
>> rock-solid solution for remote processes.
>
> True.  "Connection-local variable" sounds a lot like "project-local
> variable" to me.  Maybe there could be a unified mechanism.

No, they are not project-local. For example, via connection-local
variables you could specify which shell to use on remote.

About the syntax ... Tramp is backwards compatible down to Emacs
25. Connection-local variables were introduced in Emacs 26. I'm all ears
to new proposals, but we must ensure this compatibility.

Best regards, Michael.

Re: Buffer-local process environments

[Augusto Stoffel]

Hi again,

>> But I think it's a Tramp bug.  Tramp is supposed to completely ignore
>> `process-environment', right?  So it shouldn't matter if that variable
>> is buffer local or not.
>
> No. See my reply the other message.

The following prints the words "dangerous" on emacs -Q.  Also if I
replace "/sudo::/" by any of the ssh hosts I have access to.  This isn't
expected, is it?

(let ((default-directory "/sudo::/"))
  (setq-local process-environment (cons
                                   (concat "PATH=/dangerous:" (getenv "PATH"))
                                   process-environment))
  (shell-command "echo $PATH" t))

>> True.  "Connection-local variable" sounds a lot like "project-local
>> variable" to me.  Maybe there could be a unified mechanism.
>
> No, they are not project-local. For example, via connection-local
> variables you could specify which shell to use on remote.

I was just saying there's perhaps an analogy here.  "Project-local
variables" isn't a thing, as far as I know, but perhaps something like
that could be introduced.  I'm not trying to make a concrete suggestion
right now. :-)

Re: Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

> Hi again,

Hi Augusto,

>>> But I think it's a Tramp bug.  Tramp is supposed to completely ignore
>>> `process-environment', right?  So it shouldn't matter if that variable
>>> is buffer local or not.
>>
>> No. See my reply the other message.
>
> The following prints the words "dangerous" on emacs -Q.  Also if I
> replace "/sudo::/" by any of the ssh hosts I have access to.  This isn't
> expected, is it?
>
> (let ((default-directory "/sudo::/"))
>   (setq-local process-environment (cons
>                                    (concat "PATH=/dangerous:" (getenv "PATH"))
>                                    process-environment))
>   (shell-command "echo $PATH" t))

It is expected. Tramp does not analyze your settings. If you do (concat
"PATH=/dangerous:" (getenv "PATH")), Tramp won't object. Your decision.

>> No, they are not project-local. For example, via connection-local
>> variables you could specify which shell to use on remote.
>
> I was just saying there's perhaps an analogy here.  "Project-local
> variables" isn't a thing, as far as I know, but perhaps something like
> that could be introduced.  I'm not trying to make a concrete suggestion
> right now. :-)

Perhaps. The concepts of connection-local and project-local have been
added to Emacs independently of each other.

Best regards, Michael.

Re: Buffer-local process environments

[Augusto Stoffel]

>> In fact, even without the patch, if I make my `process-environment'
>> buffer local, then `M-x cd' to a ssh path, then do `M-! env', I see a
>> mixture of my machine's environment and the remote machine's environment.
>> For instance, PATH has the local machine value.
>
> Tramp uses already a hack. Obviously, it cannot use process-environment
> as-it-is on a remote machine, because it is, ahem, a remote machine and
> not the local one. Therefore, Tramp forces only environemt variables,
> which are not set in the defauzlt value of process-environment, to the
> remote. The idea is, that specific remote environment variables are
> added to process-environment via a let-binding.

Ah, now I see why this happens.  But I'm not sure I like the idea.  Why
not introduce a specific dynamic var for this purpose, similar to
`compilation-environment' in compile.el?

PS: Is there a single place to tell Tramp not to change the environment
of the remote host at all?  I understand the rationale for all the
automagic stuff, and I would appreciate it if I had to access lots of
remotes.  However, in my case, I only access a very small number of
servers, each of which requires a very specific configuration.

>
> A buffer-local process-environment is another, new game player.
>
> Best regards, Michael.

Re: Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

> Ah, now I see why this happens.  But I'm not sure I like the idea.  Why
> not introduce a specific dynamic var for this purpose, similar to
> `compilation-environment' in compile.el?

Because there's a lot of functions out there, which work for the local
host, and should work also for remote hosts. See for example
vc-git-grep, which has

--8<---------------cut here---------------start------------->8---
(let ((default-directory dir)
      (compilation-environment (cons "PAGER=" compilation-environment)))
--8<---------------cut here---------------end--------------->8---

compilation-environment will be propagated to process-environment later
on. And the setting for PAGER is needed for both local and remote
instances of vc-git-grep. There's no special code in vc-git-grep for the
remote case.

> PS: Is there a single place to tell Tramp not to change the environment
> of the remote host at all?  I understand the rationale for all the
> automagic stuff, and I would appreciate it if I had to access lots of
> remotes.  However, in my case, I only access a very small number of
> servers, each of which requires a very specific configuration.

Hmm, I don't see it immediately. Maybe we could discuss this in a
separate thread, where you specify in more detail what you like?
Perhaps, connection-local variables are the solution for you.

Best regards, Michael.

Re: Buffer-local process environments

[Augusto Stoffel]

> Because there's a lot of functions out there, which work for the local
> host, and should work also for remote hosts. See for example
> vc-git-grep, which has
>
> (let ((default-directory dir)
>       (compilation-environment (cons "PAGER=" compilation-environment)))
>
> compilation-environment will be propagated to process-environment later
> on. And the setting for PAGER is needed for both local and remote
> instances of vc-git-grep. There's no special code in vc-git-grep for the
> remote case.

Some observations:

1. In this case, one could call "git --no-pager" instead of relying on
   an env var.

2. PAGER is overridden by `tramp-remote-process-environment' anyway,
   right?  And unlike something of the likes of PYTHONPATH, I see no
   reason to customize PAGER.

3. If Tramp checked for the buffer-local value of process-environment
   instead of the default value, then the patch for compile.el I
   attached yesterday wouldn't break for remote directories, and the
   let-binding trick in your example would still work.

So I wonder:

A. Are there more compelling examples showing that Lisp code needs
   fine-grained control over variables being exported to a remote host?

B. There is probably a small list of variables that should be preserved
   across machines, while there is an unbounded quantity of variables
   that probably only make sense machine-locally (e.g., any variable
   holding directory names).

In view of 3., one could introduce the convention that the buffer-local
value of `process-environment' is for "project-local" variables, and the
let-bound value is for variables that make sense even on remote machines.

I'm not sure this is a good proposal, though.  It's a subtle rule, and
it could be quite brittle and hard to use.

An alternative proposal is to introduce a variable
`remote-exported-variables', which anyone could set or let-bind or even
override on a connection-local basis.  The value of any variable whose
name appears in this list would be passed though a remote connection.
This proposal would make a lot of sense if assumption B. is true.

Re: Buffer-local process environments

[Augusto Stoffel]

> Because there's a lot of functions out there, which work for the local
> host, and should work also for remote hosts. See for example
> vc-git-grep, which has
>
> (let ((default-directory dir)
>       (compilation-environment (cons "PAGER=" compilation-environment)))
>
> compilation-environment will be propagated to process-environment later
> on. And the setting for PAGER is needed for both local and remote
> instances of vc-git-grep. There's no special code in vc-git-grep for the
> remote case.

In fact, if one launches Emacs via `PAGER= emacs', then Tramp will stop
setting PAGER on the remote as desired.  (Well, it won't, because PAGER
is in `tramp-remote-process-environment', but you see my point.)

Re: Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

>> Because there's a lot of functions out there, which work for the local
>> host, and should work also for remote hosts. See for example
>> vc-git-grep, which has
>>
>> (let ((default-directory dir)
>>       (compilation-environment (cons "PAGER=" compilation-environment)))
>>
>> compilation-environment will be propagated to process-environment later
>> on. And the setting for PAGER is needed for both local and remote
>> instances of vc-git-grep. There's no special code in vc-git-grep for the
>> remote case.
>
> Some observations:
>
> 1. In this case, one could call "git --no-pager" instead of relying on
>    an env var.
>
> 2. PAGER is overridden by `tramp-remote-process-environment' anyway,
>    right?  And unlike something of the likes of PYTHONPATH, I see no
>    reason to customize PAGER.
>
> 3. If Tramp checked for the buffer-local value of process-environment
>    instead of the default value, then the patch for compile.el I
>    attached yesterday wouldn't break for remote directories, and the
>    let-binding trick in your example would still work.

Please, it is just an example. And Tramp settings, like PAGER, have been
added later on.

A simple rgrep for "(process-environment" over Emacs' lisp directory
shows you several other hits, some of them are good for this discussion:

./vc/vc-bzr.el97:      (let ((process-environment (cons (format "BZR_LOG=%s" null-device)
./vc/vc-hg.el1432:             (process-environment (cons "HGPLAIN=1" process-environment))
./vc/vc-hg.el1507:         (process-environment (cons "HGPLAIN=1" process-environment))
./vc/vc-hg.el1522:  (let ((process-environment (cons "HGPLAIN=1" process-environment))
./vc/vc-dispatcher.el328:	      (process-environment (cons "LC_MESSAGES=C" process-environment))

This list is not comprehensive, of course. And again, I have no idea
what is used in the wild.

> So I wonder:
>
> A. Are there more compelling examples showing that Lisp code needs
>    fine-grained control over variables being exported to a remote host?
>
> B. There is probably a small list of variables that should be preserved
>    across machines, while there is an unbounded quantity of variables
>    that probably only make sense machine-locally (e.g., any variable
>    holding directory names).
>
> In view of 3., one could introduce the convention that the buffer-local
> value of `process-environment' is for "project-local" variables, and the
> let-bound value is for variables that make sense even on remote machines.
>
> I'm not sure this is a good proposal, though.  It's a subtle rule, and
> it could be quite brittle and hard to use.

Honestly, the hack we have now is already annoying, at least to me. I
would prefer to have something more solid to apply.

> An alternative proposal is to introduce a variable
> `remote-exported-variables', which anyone could set or let-bind or even
> override on a connection-local basis.  The value of any variable whose
> name appears in this list would be passed though a remote connection.
> This proposal would make a lot of sense if assumption B. is true.

There is already tramp-remote-process-environment, for many years. Not
much appreciated by package authors.

Best regards, Michael.

Re: Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

>> Because there's a lot of functions out there, which work for the local
>> host, and should work also for remote hosts. See for example
>> vc-git-grep, which has
>>
>> (let ((default-directory dir)
>>       (compilation-environment (cons "PAGER=" compilation-environment)))
>>
>> compilation-environment will be propagated to process-environment later
>> on. And the setting for PAGER is needed for both local and remote
>> instances of vc-git-grep. There's no special code in vc-git-grep for the
>> remote case.
>
> In fact, if one launches Emacs via `PAGER= emacs', then Tramp will stop
> setting PAGER on the remote as desired.  (Well, it won't, because PAGER
> is in `tramp-remote-process-environment', but you see my point.)

Sure. But reality is, that a lot of packages are written w/o any care of
Tramp. People call remote processes w/o any care of such
environment. They simply expect that Tramp works (and bash me otherwise :-)

Best regards, Michael.

Re: Buffer-local process environments

[Augusto Stoffel]

> Honestly, the hack we have now is already annoying, at least to me. I
> would prefer to have something more solid to apply.

Hi Michael,

I can only think of two alternatives:

1. Introduce a whitelist of variables that can be exported to a remote.
   Then drop entirely the current hack and just check whether
   `process-environment' contains any exportable variables.  For
   flexibility, you can allow whitelisting a variable name (any value),
   or a specific VAR=VALUE pair.

   The whitelist can be set globally or dynamically.  So, for instance,
   the vc files could just append "BZR_LOG", "HGPLAIN" and whatnot
   globally to this list and never worry again.

   Obviously, this will break third-party code, but the fix is quite
   easy.  Nobody could bash you for that.

2. Introduce a blacklist of variables that are never exported to a
   remote.  This can be done by extending
   `tramp-remote-process-environment' to follow the same convention of
   `process-environment' that an entry of the form VAR, without the
   =VALUE, means removing the variable.

   So, for instance, entries "PATH" and "TERM" should be added to the
   default value of `tramp-remote-process-environment'.  This would
   solve the unintuitive (if not buggy) behavior I pointed out a few
   messages back in this thread.

   As another example, python.el would append "PYTHONPATH" and
   "PYTHONHOME" globally to `tramp-remote-process-environment', since
   these variables hold directory names.

   This option is fully backwards compatible, but keeps the hack you
   said you don't like anyway.

It's also conceivable do to both things, but instead of enforcing 1.,
just give a warning.  Then eventually you could get rid of the hack
without breaking other packages.

Re: Buffer-local process environments

[Stephen Leake]

Augusto Stoffel <arstoffel@gmail.com> writes:

> When developing software, it's often not possible or desirable to
> install compilers and related tools globally, so (barring the
> availability of fancy build tools) one has to fiddle with environment
> variables.

The GNU ELPA package 'wisi' provides that capability.


-- 
-- Stephe

Re: Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

> Hi Michael,

Hi Augusto,

> I can only think of two alternatives:
>
> 1. Introduce a whitelist of variables that can be exported to a remote.
>    Then drop entirely the current hack and just check whether
>    `process-environment' contains any exportable variables.  For
>    flexibility, you can allow whitelisting a variable name (any value),
>    or a specific VAR=VALUE pair.
>
>    The whitelist can be set globally or dynamically.  So, for instance,
>    the vc files could just append "BZR_LOG", "HGPLAIN" and whatnot
>    globally to this list and never worry again.
>
>    Obviously, this will break third-party code, but the fix is quite
>    easy.  Nobody could bash you for that.

We have already a white list, which is tramp-remote-process-environment.
I'm reluctant to add package specific entries there by default, it would
be an endless story. PAGER is included as being a general purpose
environment variable, BZR_LOG and HGPLAIN are not. If packages believe
they shall be added, this could be done in vc-bzr.el or vc-hg.el. Tramp
is a stupid library, it shouldn't try to be clever.

> 2. Introduce a blacklist of variables that are never exported to a
>    remote.  This can be done by extending
>    `tramp-remote-process-environment' to follow the same convention of
>    `process-environment' that an entry of the form VAR, without the
>    =VALUE, means removing the variable.

There are already such variables to be unset. These are the variables
without any value, like "HISTORY="

>    So, for instance, entries "PATH" and "TERM" should be added to the
>    default value of `tramp-remote-process-environment'.  This would
>    solve the unintuitive (if not buggy) behavior I pointed out a few
>    messages back in this thread.

TERM is handled special. All Tramp connections add "TERM=dumb",
hard-coded. Since this shall not be changed by a user, it isn't
configurable here.

PATH is handled by tramp-remote-path. This is a variable on its own,
because just setting a simple string "PATH=/abc:/def" isn't sufficient,
often.

>    As another example, python.el would append "PYTHONPATH" and
>    "PYTHONHOME" globally to `tramp-remote-process-environment', since
>    these variables hold directory names.

Yes, if python.el developers prefer that. However, I doubt, that this
value is always the same for all different remote hosts, the value might
differ depending on the OS the remote host is running.

>    This option is fully backwards compatible, but keeps the hack you
>    said you don't like anyway.

So in fact, this solution is almost the current implementation.

> It's also conceivable do to both things, but instead of enforcing 1.,
> just give a warning.  Then eventually you could get rid of the hack
> without breaking other packages.

As said, variant 1. isn't applicable in my eyes.

Best regards, Michael.

Re: Buffer-local process environments

[Augusto Stoffel]

[-- Attachment #1: Type: text/plain, Size: 2826 bytes --]

On Sat,  8 May 2021 at 19:51, Michael Albinus <michael.albinus@gmx.de> wrote:

Hi Michael,

> We have already a white list, which is tramp-remote-process-environment.
> I'm reluctant to add package specific entries there by default, it would
> be an endless story. PAGER is included as being a general purpose
> environment variable, BZR_LOG and HGPLAIN are not. If packages believe
> they shall be added, this could be done in vc-bzr.el or vc-hg.el. Tramp
> is a stupid library, it shouldn't try to be clever.

Fair enough.  Let's cross out the first idea then.

>> 2. Introduce a blacklist of variables that are never exported to a
>>    remote.  This can be done by extending
>>    `tramp-remote-process-environment' to follow the same convention of
>>    `process-environment' that an entry of the form VAR, without the
>>    =VALUE, means removing the variable.
>
> There are already such variables to be unset. These are the variables
> without any value, like "HISTORY="

This only works for entries in the default toplevel value of
`process-environment'.  Not for things added buffer-locally or
in a let-binding.

The heuristic that every env var which is not in the default toplevel
value of `process-environment' will be exported to the remote works
pretty well.  But it needs a simple way to be overridden, since it can't
read the user's mind.

> TERM is handled special. All Tramp connections add "TERM=dumb",
> hard-coded. Since this shall not be changed by a user, it isn't
> configurable here.

Not all connections add "TERM=dumb".  Here's a counterexample:

    (let ((default-directory "/sudo::/")
          (process-environment `("TERM=dumber"
                                 ,@process-environment)))
      (shell-command "echo $TERM" t))
    => dumber

How do I tell Tramp not to export TERM=dumber to the remote in this
case?

To be clear: this may be a dumb example, but there are plenty of
interesting use-cases involving PATH, PYTHONPATH, and so on, which are
analogous.  How can I set PYTHONPATH buffer-locally, but disallow Tramp
to export this variable to a remote?

>>    As another example, python.el would append "PYTHONPATH" and
>>    "PYTHONHOME" globally to `tramp-remote-process-environment', since
>>    these variables hold directory names.
>
> Yes, if python.el developers prefer that. However, I doubt, that this
> value is always the same for all different remote hosts, the value might
> differ depending on the OS the remote host is running.

That's precisely what I meant: you *do not* want to export those variables
to a remote.  But currently Tramp effectively forbids anyone to set
PYTHONPATH buffer-locally, because this will cause them to be exported.

My suggestion is easier to implement than explain.  So let me repeat it
in the form of a little (possibly incomplete) patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: tramp-env.patch --]
[-- Type: text/x-patch, Size: 1208 bytes --]

diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 015f458a63..3749f84e21 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -3938,21 +3938,23 @@ tramp-handle-make-process
 		       (when (string-match-p "=" elt) elt))
 		     tramp-remote-process-environment))
 	       ;; We use as environment the difference to toplevel
 	       ;; `process-environment'.
 	       (env (dolist (elt process-environment env)
 		      (when
 			  (and
 			   (string-match-p "=" elt)
 			   (not
 			    (member
-			     elt (default-toplevel-value 'process-environment))))
+			     elt (default-toplevel-value 'process-environment)))
+                           (not (member (car (split-string elt "="))
+                                        tramp-remote-process-environment)))
 			(setq env (cons elt env)))))
 	       (env (setenv-internal
 		     env "INSIDE_EMACS" (tramp-inside-emacs) 'keep))
 	       (env (mapcar #'tramp-shell-quote-argument (delq nil env)))
 	       ;; Quote command.
 	       (command (mapconcat #'tramp-shell-quote-argument command " "))
 	       ;; Set cwd and environment variables.
 	       (command
 	        (append `("cd" ,localname "&&" "(" "env") env `(,command ")"))))
 

Re: Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

> Hi Michael,

Hi Augusto,

>>> 2. Introduce a blacklist of variables that are never exported to a
>>>    remote.  This can be done by extending
>>>    `tramp-remote-process-environment' to follow the same convention of
>>>    `process-environment' that an entry of the form VAR, without the
>>>    =VALUE, means removing the variable.
>>
>> There are already such variables to be unset. These are the variables
>> without any value, like "HISTORY="
>
> This only works for entries in the default toplevel value of
> `process-environment'.  Not for things added buffer-locally or
> in a let-binding.

Yes, this is a bug (rather a missing feature) of Tramp. It is
implemented for synchronous processes, but not for asynchronous
processes. I'm aware of this, but I haven't found the mood to implement
it until now.

> The heuristic that every env var which is not in the default toplevel
> value of `process-environment' will be exported to the remote works
> pretty well.  But it needs a simple way to be overridden, since it can't
> read the user's mind.

Sure. Tramp shall not read the user's mind.

(As side remark, my wife expects always that I read her mind. I fail
miserably, usually.)

>> TERM is handled special. All Tramp connections add "TERM=dumb",
>> hard-coded. Since this shall not be changed by a user, it isn't
>> configurable here.
>
> Not all connections add "TERM=dumb".  Here's a counterexample:
>
>     (let ((default-directory "/sudo::/")
>           (process-environment `("TERM=dumber"
>                                  ,@process-environment)))
>       (shell-command "echo $TERM" t))
>     => dumber
>
> How do I tell Tramp not to export TERM=dumber to the remote in this
> case?

You can always shoot yourself in your feed.

> To be clear: this may be a dumb example, but there are plenty of
> interesting use-cases involving PATH, PYTHONPATH, and so on, which are
> analogous.  How can I set PYTHONPATH buffer-locally, but disallow Tramp
> to export this variable to a remote?

Set it only, when the buffer is not remote. Something like

(unless (file-remote-p default-directory) ...)

>>>    As another example, python.el would append "PYTHONPATH" and
>>>    "PYTHONHOME" globally to `tramp-remote-process-environment', since
>>>    these variables hold directory names.
>>
>> Yes, if python.el developers prefer that. However, I doubt, that this
>> value is always the same for all different remote hosts, the value might
>> differ depending on the OS the remote host is running.
>
> That's precisely what I meant: you *do not* want to export those variables
> to a remote.  But currently Tramp effectively forbids anyone to set
> PYTHONPATH buffer-locally, because this will cause them to be exported.

No. See above.

> My suggestion is easier to implement than explain.  So let me repeat it
> in the form of a little (possibly incomplete) patch:
>
> diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
> index 015f458a63..3749f84e21 100644
> --- a/lisp/net/tramp.el
> +++ b/lisp/net/tramp.el
> @@ -3938,21 +3938,23 @@ tramp-handle-make-process
>  		       (when (string-match-p "=" elt) elt))
>  		     tramp-remote-process-environment))
>  	       ;; We use as environment the difference to toplevel
>  	       ;; `process-environment'.
>  	       (env (dolist (elt process-environment env)
>  		      (when
>  			  (and
>  			   (string-match-p "=" elt)
>  			   (not
>  			    (member
> -			     elt (default-toplevel-value 'process-environment))))
> +			     elt (default-toplevel-value 'process-environment)))
> +                           (not (member (car (split-string elt "="))
> +                                        tramp-remote-process-environment)))
>  			(setq env (cons elt env)))))
>  	       (env (setenv-internal
>  		     env "INSIDE_EMACS" (tramp-inside-emacs) 'keep))
>  	       (env (mapcar #'tramp-shell-quote-argument (delq nil env)))
>  	       ;; Quote command.
>  	       (command (mapconcat #'tramp-shell-quote-argument command " "))
>  	       ;; Set cwd and environment variables.
>  	       (command
>  	        (append `("cd" ,localname "&&" "(" "env") env `(,command ")"))))
>

I see. Interesting.

But this still means, that you use something global in
tramp-remote-process-environment for all remote processes.

What, if you want to set PYTHONPATH for hosta, and not for hostb?

Best regards, Michael.

Re: [PATCH] Buffer-local process environments

[Augusto Stoffel]

[-- Attachment #1: Type: text/plain, Size: 2391 bytes --]

After sleeping on this (for quite a while), I'm pretty confident that
the attached patch gives `compile' a sensible behavior in the presence
of a buffer-local `process-environment'.

Specifically, whenever `compile' is called from a buffer where
`process-environment' is local, the *compilation* buffer inherits the
original buffer's `process-environment' and `exec-path'.  When
`process-environment' is not local in the buffer from which `compile'
is called, any local values of those two variables are killed in the
*compilation* buffer as well.  (There's no check for buffer-localness
of `exec-path' because it's usually misguided to keep it out of sync
with PATH.)

Concerning possible interactions with Tramp, I think the conclusion from
what Michael said previously is: if your buffer is remote, then don't
mess with `process-environment' and set
`tramp-remote-process-environment' instead.  If that is a requirement
imposed on the user, then this patch is free of conflicts with Tramp by
decree.

Still, there is one case, which might be quite specific to my personal
workflow, where I get a problematic interaction with Tramp.  I'm often
editing files locally that I can only compile or run on a heavy-duty
remote host.  So I call, from those local buffers

(let ((default-directory "/ssh:remote-host:/path/to/mirror/of/project"))
  (compile "make"))

With the attached patch, a buffer-local PATH value will leak to the
remote machine, which is almost surely the wrong thing to do.  I've
suggested one solution for this on the Tramp side (using
`tramp-process-environment' itself to block certain variables from
being exported to the remote).  I believe Michael wasn't convinced
this is a good idea.  In any case, there's another solution to this
problem, which is to replace the above by

(let ((default-directory "/ssh:remote-host:/path/to/mirror/of/project"))
      (process-environment (default-value 'process-environment))
  (compile "make"))

This is an okay hurdle to introduce, because it is already needed
anyway for functions that starts a process _without_ changing buffers
first, such as `shell-command'.

I hope this makes sense to everybody and can be merged.  I'd also hope
that the present handling of env vars can serve as a model to be
adopted by most commands that switch buffers and then start a process
in this new buffer, such as the `run-<program>` commands.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-compile-respect-buffer-local-process-environmen.patch --]
[-- Type: text/x-patch, Size: 1584 bytes --]

From e46913ec1f0e3f839dd8a36022095587b1962ac5 Mon Sep 17 00:00:00 2001
From: Augusto Stoffel <arstoffel@gmail.com>
Date: Thu, 29 Apr 2021 12:45:04 +0200
Subject: [PATCH] Make 'compile' respect buffer-local process environment

* lisp/progmodes/compile.el (compilation-start): Use
`process-environment' from original buffer in the compilation process.
---
 lisp/progmodes/compile.el | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lisp/progmodes/compile.el b/lisp/progmodes/compile.el
index af7b8292b7..1c11b7707e 100644
--- a/lisp/progmodes/compile.el
+++ b/lisp/progmodes/compile.el
@@ -1783,6 +1783,8 @@ compilation-start
 	    (replace-regexp-in-string "-mode\\'" "" (symbol-name mode))))
 	 (thisdir default-directory)
 	 (thisenv compilation-environment)
+         (bufferenv (when (local-variable-p 'process-environment)
+                      (cons exec-path process-environment)))
 	 outwin outbuf)
     (with-current-buffer
 	(setq outbuf
@@ -1850,6 +1852,11 @@ compilation-start
         ;; NB: must be done after (funcall mode) as that resets local variables
         (setq-local compilation-directory thisdir)
         (setq-local compilation-environment thisenv)
+        (if bufferenv
+            (setq-local exec-path (car bufferenv)
+                        process-environment (cdr bufferenv))
+          (kill-local-variable 'exec-path)
+          (kill-local-variable 'process-environment))
 	(if highlight-regexp
             (setq-local compilation-highlight-regexp highlight-regexp))
         (if (or compilation-auto-jump-to-first-error
-- 
2.31.1

Re: [PATCH] Buffer-local process environments

[Eli Zaretskii]

> From: Augusto Stoffel <arstoffel@gmail.com>
> Date: Sat, 28 Aug 2021 14:28:59 +0200
> 
> Specifically, whenever `compile' is called from a buffer where
> `process-environment' is local, the *compilation* buffer inherits the
> original buffer's `process-environment' and `exec-path'.  When
> `process-environment' is not local in the buffer from which `compile'
> is called, any local values of those two variables are killed in the
> *compilation* buffer as well.  (There's no check for buffer-localness
> of `exec-path' because it's usually misguided to keep it out of sync
> with PATH.)

Thanks.  The patch should be accompanied by a suitable documentation
change, of course.

And I'm not sure I understand the rationale, and you didn't describe
it.  Environment variables and PATH in particular are generally global
on the entire system, so what does this feature correspond to?
separate environment variables in each shell window?  Why would anyone
want to do that?  And why should we make it easier by providing
buffer-local values of those, instead of letting each Lisp program
that needs it let-bind the variables instead?

Re: [PATCH] Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

Hi Augusto,

> diff --git a/lisp/progmodes/compile.el b/lisp/progmodes/compile.el
> index af7b8292b7..1c11b7707e 100644
> --- a/lisp/progmodes/compile.el
> +++ b/lisp/progmodes/compile.el
> @@ -1783,6 +1783,8 @@ compilation-start
>  	    (replace-regexp-in-string "-mode\\'" "" (symbol-name mode))))
>  	 (thisdir default-directory)
>  	 (thisenv compilation-environment)
> +         (bufferenv (when (local-variable-p 'process-environment)
> +                      (cons exec-path process-environment)))

W/o further reading, this cannot be right. Elements of
`process-environment' have always the syntax "VAR=VALUE". You add only a
VALUE (`exec-path') in `bufferenv'. I understand your intention, using
`bufferenv' as transportation vehicle, but I believe this is
mis-designed, and good for trouble. If not now, then later.

(I will review your proposal later, when this is fixed.)

Best regards, Michael.

Re: [PATCH] Buffer-local process environments

[Augusto Stoffel]

On Sat, 28 Aug 2021 at 15:37, Eli Zaretskii <eliz@gnu.org> wrote:

> And I'm not sure I understand the rationale, and you didn't describe
> it.  Environment variables and PATH in particular are generally global
> on the entire system, so what does this feature correspond to?
> separate environment variables in each shell window?  Why would anyone
> want to do that?  And why should we make it easier by providing
> buffer-local values of those, instead of letting each Lisp program
> that needs it let-bind the variables instead?

Yes, sorry for suddenly reviving an old discussion.

Many languages and build tools rely on environment variables, so those
variables are not global, but rather local to a project.  In Python, for
instance, the standard way to choose a particular version of the
interpreter and of various packages is by setting the PATH and perhaps
PYTHONPATH variables.  It's a dumb and simple system, which has its
advantages.

If you are working on two projects with different requirements, you need
to either launch an Emacs instance for each project, which is the
pedestrian option, or have buffer-local environment variables.

There are packages out there to help setting the environment on a
buffer-local basis, e.g. https://melpa.org/#/envrc

Re: [PATCH] Buffer-local process environments

[Augusto Stoffel]

On Sat, 28 Aug 2021 at 14:47, Michael Albinus <michael.albinus@gmx.de> wrote:

> Augusto Stoffel <arstoffel@gmail.com> writes:
>
> Hi Augusto,
>
>> diff --git a/lisp/progmodes/compile.el b/lisp/progmodes/compile.el
>> index af7b8292b7..1c11b7707e 100644
>> --- a/lisp/progmodes/compile.el
>> +++ b/lisp/progmodes/compile.el
>> @@ -1783,6 +1783,8 @@ compilation-start
>>  	    (replace-regexp-in-string "-mode\\'" "" (symbol-name mode))))
>>  	 (thisdir default-directory)
>>  	 (thisenv compilation-environment)
>> +         (bufferenv (when (local-variable-p 'process-environment)
>> +                      (cons exec-path process-environment)))
>
> W/o further reading, this cannot be right. Elements of
> `process-environment' have always the syntax "VAR=VALUE". You add only a
> VALUE (`exec-path') in `bufferenv'. I understand your intention, using
> `bufferenv' as transportation vehicle, but I believe this is
> mis-designed, and good for trouble. If not now, then later.

This is a temporary, lexical variable that is inaccessible to the user
directly.  It's "destructured" again in the next hunk of the patch.

>
> (I will review your proposal later, when this is fixed.)

Thanks!

>
> Best regards, Michael.

Re: [PATCH] Buffer-local process environments

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

>> W/o further reading, this cannot be right. Elements of
>> `process-environment' have always the syntax "VAR=VALUE". You add only a
>> VALUE (`exec-path') in `bufferenv'. I understand your intention, using
>> `bufferenv' as transportation vehicle, but I believe this is
>> mis-designed, and good for trouble. If not now, then later.
>
> This is a temporary, lexical variable that is inaccessible to the user
> directly.  It's "destructured" again in the next hunk of the patch.

I've seen this. But we shall avoid this dirty kind of programming, even
if it is internal only.

Best regards, Michael.

Re: [PATCH] Buffer-local process environments

[Augusto Stoffel]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]

On Sat, 28 Aug 2021 at 15:18, Michael Albinus <michael.albinus@gmx.de> wrote:

> Augusto Stoffel <arstoffel@gmail.com> writes:
>
>>> W/o further reading, this cannot be right. Elements of
>>> `process-environment' have always the syntax "VAR=VALUE". You add only a
>>> VALUE (`exec-path') in `bufferenv'. I understand your intention, using
>>> `bufferenv' as transportation vehicle, but I believe this is
>>> mis-designed, and good for trouble. If not now, then later.
>>
>> This is a temporary, lexical variable that is inaccessible to the user
>> directly.  It's "destructured" again in the next hunk of the patch.
>
> I've seen this. But we shall avoid this dirty kind of programming, even
> if it is internal only.

In this case, here is a variation that handles the two lists separately.

(It will have a surprising effect if some of these lists is empty, but
this should never happen --- and the solution would be to wrap the lists
in a cons, which you don't like...)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-compile-respect-buffer-local-process-environmen.patch --]
[-- Type: text/x-patch, Size: 1630 bytes --]

From a104e1eae100f01585587df30f49866f3b038785 Mon Sep 17 00:00:00 2001
From: Augusto Stoffel <arstoffel@gmail.com>
Date: Thu, 29 Apr 2021 12:45:04 +0200
Subject: [PATCH] Make 'compile' respect buffer-local process environment

* lisp/progmodes/compile.el (compilation-start): Use
`process-environment' from original buffer in the compilation process.
---
 lisp/progmodes/compile.el | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lisp/progmodes/compile.el b/lisp/progmodes/compile.el
index af7b8292b7..bdf20a3f51 100644
--- a/lisp/progmodes/compile.el
+++ b/lisp/progmodes/compile.el
@@ -1783,6 +1783,8 @@ compilation-start
 	    (replace-regexp-in-string "-mode\\'" "" (symbol-name mode))))
 	 (thisdir default-directory)
 	 (thisenv compilation-environment)
+         (bufferpath (when (local-variable-p 'exec-path) exec-path))
+         (bufferenv (when (local-variable-p 'process-environment) process-environment))
 	 outwin outbuf)
     (with-current-buffer
 	(setq outbuf
@@ -1850,6 +1852,12 @@ compilation-start
         ;; NB: must be done after (funcall mode) as that resets local variables
         (setq-local compilation-directory thisdir)
         (setq-local compilation-environment thisenv)
+        (if bufferpath
+            (setq-local exec-path bufferpath)
+          (kill-local-variable 'exec-path))
+        (if bufferenv
+            (setq-local process-environment bufferenv)
+          (kill-local-variable 'process-environment))
 	(if highlight-regexp
             (setq-local compilation-highlight-regexp highlight-regexp))
         (if (or compilation-auto-jump-to-first-error
-- 
2.31.1

Re: [PATCH] Buffer-local process environments

[Stefan Monnier]

Michael Albinus [2021-08-28 15:18:56] wrote:
> Augusto Stoffel <arstoffel@gmail.com> writes:
>
>>> W/o further reading, this cannot be right. Elements of
>>> `process-environment' have always the syntax "VAR=VALUE". You add only a
>>> VALUE (`exec-path') in `bufferenv'. I understand your intention, using
>>> `bufferenv' as transportation vehicle, but I believe this is
>>> mis-designed, and good for trouble. If not now, then later.
>>
>> This is a temporary, lexical variable that is inaccessible to the user
>> directly.  It's "destructured" again in the next hunk of the patch.
>
> I've seen this. But we shall avoid this dirty kind of programming, even
> if it is internal only.

I don't see anything dirty about it.

I think you're just getting confused by the fact that the code uses
`cons` to build a pair of a string and a list and you read it as adding
add a string to the list.

Maybe the confusion would be avoided by switching the two (i.e. use
`(cons process-environment exec-path)`).


        Stefan

Re: [PATCH] Buffer-local process environments

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Augusto Stoffel <arstoffel@gmail.com>
>> Date: Sat, 28 Aug 2021 14:28:59 +0200
>> 
>> Specifically, whenever `compile' is called from a buffer where
>> `process-environment' is local, the *compilation* buffer inherits the
>> original buffer's `process-environment' and `exec-path'.  When
>> `process-environment' is not local in the buffer from which `compile'
>> is called, any local values of those two variables are killed in the
>> *compilation* buffer as well.  (There's no check for buffer-localness
>> of `exec-path' because it's usually misguided to keep it out of sync
>> with PATH.)
>
> Thanks.  The patch should be accompanied by a suitable documentation
> change, of course.
>
> And I'm not sure I understand the rationale, and you didn't describe
> it.  Environment variables and PATH in particular are generally global
> on the entire system, so what does this feature correspond to?
> separate environment variables in each shell window?  Why would anyone
> want to do that?  And why should we make it easier by providing
> buffer-local values of those, instead of letting each Lisp program
> that needs it let-bind the variables instead?

Excuse me if I ask here, it is probably more help than contribution to this
discussion. Is it possible to let-bind environment variables so that a
subprocess inherit those? 

I just wrote today a piece of code where I wanted to start a process with
different environment variables than what Emacs has, for LANG and LC_TIME. I
didn't know of other way to easily pass environment vars, other than to start
new shell process. If it is possible to use some process apis to change process
environment I would happily use them instead of starting an intermediate shell
process. I am really not happy to use shell there.

(defun emacs-vision--get-date ()
  (let ((lang (concat "LANG=" emacs-vision-locale))
        (lctime (concat "LC_TIME=" emacs-vision-locale)))
    (with-temp-buffer
      (insert
       (shell-command-to-string
        (concat lang lctime " date +'%A, %B %d'")))
      (goto-char (- (point-max) 1))
      (buffer-substring-no-properties
       (line-beginning-position) (line-end-position)))))

Do I understand correctly, this patch will let me have separate process
environment, so I could use separate environment variables in subprocess?
Otherwise if I can let-bind environment vars it would be even better.

Re: [PATCH] Buffer-local process environments

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: Augusto Stoffel <arstoffel@gmail.com>,  michael.albinus@gmx.de,
>   monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 28 Aug 2021 16:06:19 +0200
> 
> > And I'm not sure I understand the rationale, and you didn't describe
> > it.  Environment variables and PATH in particular are generally global
> > on the entire system, so what does this feature correspond to?
> > separate environment variables in each shell window?  Why would anyone
> > want to do that?  And why should we make it easier by providing
> > buffer-local values of those, instead of letting each Lisp program
> > that needs it let-bind the variables instead?
> 
> Excuse me if I ask here, it is probably more help than contribution to this
> discussion. Is it possible to let-bind environment variables so that a
> subprocess inherit those? 

Of course.  You let-bind process-environment, and then change the
local value.  We have a number of places where we do that.

> (defun emacs-vision--get-date ()
>   (let ((lang (concat "LANG=" emacs-vision-locale))
>         (lctime (concat "LC_TIME=" emacs-vision-locale)))
>     (with-temp-buffer
>       (insert
>        (shell-command-to-string
>         (concat lang lctime " date +'%A, %B %d'")))
>       (goto-char (- (point-max) 1))
>       (buffer-substring-no-properties
>        (line-beginning-position) (line-end-position)))))
> 
> Do I understand correctly, this patch will let me have separate process
> environment, so I could use separate environment variables in subprocess?

Yes, but that's not what you were asking about, I believe.

Re: [PATCH] Buffer-local process environments

[Augusto Stoffel]

On Sat, 28 Aug 2021 at 10:05, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Michael Albinus [2021-08-28 15:18:56] wrote:
>> Augusto Stoffel <arstoffel@gmail.com> writes:
>>
>>>> W/o further reading, this cannot be right. Elements of
>>>> `process-environment' have always the syntax "VAR=VALUE". You add only a
>>>> VALUE (`exec-path') in `bufferenv'. I understand your intention, using
>>>> `bufferenv' as transportation vehicle, but I believe this is
>>>> mis-designed, and good for trouble. If not now, then later.
>>>
>>> This is a temporary, lexical variable that is inaccessible to the user
>>> directly.  It's "destructured" again in the next hunk of the patch.
>>
>> I've seen this. But we shall avoid this dirty kind of programming, even
>> if it is internal only.
>
> I don't see anything dirty about it.

I agree, but since we are here we can discuss where we want to be
slightly wrong:

(1) when `process-environment' is local but `exec-path' isn't

or

(2) when `process-environment' or `exec-path' is nil

The first patch suffers from (1), the second from (2).

>
> I think you're just getting confused by the fact that the code uses
> `cons` to build a pair of a string and a list and you read it as adding
> add a string to the list.
>
> Maybe the confusion would be avoided by switching the two (i.e. use
> `(cons process-environment exec-path)`).
>
>
>         Stefan

Re: [PATCH] Buffer-local process environments

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: Augusto Stoffel <arstoffel@gmail.com>,  michael.albinus@gmx.de,
>>   monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sat, 28 Aug 2021 16:06:19 +0200
>> 
>> > And I'm not sure I understand the rationale, and you didn't describe
>> > it.  Environment variables and PATH in particular are generally global
>> > on the entire system, so what does this feature correspond to?
>> > separate environment variables in each shell window?  Why would anyone
>> > want to do that?  And why should we make it easier by providing
>> > buffer-local values of those, instead of letting each Lisp program
>> > that needs it let-bind the variables instead?
>> 
>> Excuse me if I ask here, it is probably more help than contribution to this
>> discussion. Is it possible to let-bind environment variables so that a
>> subprocess inherit those? 
>
> Of course.  You let-bind process-environment, and then change the
> local value.  We have a number of places where we do that.

Ah, ok, I thought it process-environment was for entire Emacs, globally. I just
was looking at docs, I haven't seen examples in the code. Thanks.

>> (defun emacs-vision--get-date ()
>>   (let ((lang (concat "LANG=" emacs-vision-locale))
>>         (lctime (concat "LC_TIME=" emacs-vision-locale)))
>>     (with-temp-buffer
>>       (insert
>>        (shell-command-to-string
>>         (concat lang lctime " date +'%A, %B %d'")))
>>       (goto-char (- (point-max) 1))
>>       (buffer-substring-no-properties
>>        (line-beginning-position) (line-end-position)))))
>> 
>> Do I understand correctly, this patch will let me have separate process
>> environment, so I could use separate environment variables in subprocess?
>
> Yes, but that's not what you were asking about, I believe.

Now you make me confused again :). I want my process to use different env for
LANG and LC_TIME, but Emacs should continue to work with old values. Concretely
I want english date string, but my Emacs should continue to use swedish for it's
normal use.

So if I let-bind, it will be reflected only localy in that let scope, and the
subprocess will inherit that local environment too? That is what I want, and why
I went via shell command. Forgive me if I am confused. I hope what you said
above is what I want :-).

Re: [PATCH] Buffer-local process environments

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: arstoffel@gmail.com,  michael.albinus@gmx.de,  monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> Date: Sat, 28 Aug 2021 17:27:18 +0200
> 
> >> (defun emacs-vision--get-date ()
> >>   (let ((lang (concat "LANG=" emacs-vision-locale))
> >>         (lctime (concat "LC_TIME=" emacs-vision-locale)))
> >>     (with-temp-buffer
> >>       (insert
> >>        (shell-command-to-string
> >>         (concat lang lctime " date +'%A, %B %d'")))
> >>       (goto-char (- (point-max) 1))
> >>       (buffer-substring-no-properties
> >>        (line-beginning-position) (line-end-position)))))
> >> 
> >> Do I understand correctly, this patch will let me have separate process
> >> environment, so I could use separate environment variables in subprocess?
> >
> > Yes, but that's not what you were asking about, I believe.
> 
> Now you make me confused again :). I want my process to use different env for
> LANG and LC_TIME, but Emacs should continue to work with old values. Concretely
> I want english date string, but my Emacs should continue to use swedish for it's
> normal use.
> 
> So if I let-bind, it will be reflected only localy in that let scope, and the
> subprocess will inherit that local environment too? That is what I want, and why
> I went via shell command. Forgive me if I am confused. I hope what you said
> above is what I want :-).

The code above changes the environment only of that single shell
command.  Moreover, it's non-portable: it only works with Posix
shells, and only for commands invoked via those shells.  Local
modifications of process-environment are free from these limitations.

Re: [PATCH] Buffer-local process environments

[Augusto Stoffel]

> Now you make me confused again :). I want my process to use different env for
> LANG and LC_TIME, but Emacs should continue to work with old values. Concretely
> I want english date string, but my Emacs should continue to use swedish for it's
> normal use.
>
> So if I let-bind, it will be reflected only localy in that let scope, and the
> subprocess will inherit that local environment too? That is what I want, and why
> I went via shell command. Forgive me if I am confused. I hope what you said
> above is what I want :-).


If it's a one-off thing, you can always do

    (let ((process-environment (cons "LANG=en" process-environemnt)))
      (process-launching-stuff))

Now, suppose you want to automate this within a certain project.  Then
you want to arrange for

    (setq-local process-environment (cons "LANG=en" process-environemnt))

to be called in every buffer of the project (there are packages out
there to help doing this).  This mostly works, but some commands get
confused when `process-environment' has a buffer-local value.  The patch
is a fix for `compile'.

Re: [PATCH] Buffer-local process environments

[Arthur Miller]

Augusto Stoffel <arstoffel@gmail.com> writes:

>> Now you make me confused again :). I want my process to use different env for
>> LANG and LC_TIME, but Emacs should continue to work with old values. Concretely
>> I want english date string, but my Emacs should continue to use swedish for it's
>> normal use.
>>
>> So if I let-bind, it will be reflected only localy in that let scope, and the
>> subprocess will inherit that local environment too? That is what I want, and why
>> I went via shell command. Forgive me if I am confused. I hope what you said
>> above is what I want :-).
>
>
> If it's a one-off thing, you can always do
>
>     (let ((process-environment (cons "LANG=en" process-environemnt)))
>       (process-launching-stuff))
>
> Now, suppose you want to automate this within a certain project.  Then
> you want to arrange for
>
>     (setq-local process-environment (cons "LANG=en" process-environemnt))
>
> to be called in every buffer of the project (there are packages out
> there to help doing this).  This mostly works, but some commands get
> confused when `process-environment' has a buffer-local value.  The patch
> is a fix for `compile'.

Yes, yes. Thanks both. I guess I can switch to call-process now with let-bind :).

Re: [PATCH] Buffer-local process environments

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: arstoffel@gmail.com,  michael.albinus@gmx.de,  monnier@iro.umontreal.ca,
>>   emacs-devel@gnu.org
>> Date: Sat, 28 Aug 2021 17:27:18 +0200
>> 
>> >> (defun emacs-vision--get-date ()
>> >>   (let ((lang (concat "LANG=" emacs-vision-locale))
>> >>         (lctime (concat "LC_TIME=" emacs-vision-locale)))
>> >>     (with-temp-buffer
>> >>       (insert
>> >>        (shell-command-to-string
>> >>         (concat lang lctime " date +'%A, %B %d'")))
>> >>       (goto-char (- (point-max) 1))
>> >>       (buffer-substring-no-properties
>> >>        (line-beginning-position) (line-end-position)))))
>> >> 
>> >> Do I understand correctly, this patch will let me have separate process
>> >> environment, so I could use separate environment variables in subprocess?
>> >
>> > Yes, but that's not what you were asking about, I believe.
>> 
>> Now you make me confused again :). I want my process to use different env for
>> LANG and LC_TIME, but Emacs should continue to work with old values. Concretely
>> I want english date string, but my Emacs should continue to use swedish for it's
>> normal use.
>> 
>> So if I let-bind, it will be reflected only localy in that let scope, and the
>> subprocess will inherit that local environment too? That is what I want, and why
>> I went via shell command. Forgive me if I am confused. I hope what you said
>> above is what I want :-).
>
> The code above changes the environment only of that single shell
> command.  Moreover, it's non-portable: it only works with Posix
Yes, that was what I wanted in this particular case.

>                                                          Local
> modifications of process-environment are free from these limitations.

Yes, I'll switch to call-process with let-bind. Thank you.

I used shell as intermediate because I didn't know how modify environment in
process to be create. I really wasn't happy with using shell there. I got it now
thanks :).

Re: [PATCH] Buffer-local process environments

[Stephen Leake]

Augusto Stoffel <arstoffel@gmail.com> writes:

> Many languages and build tools rely on environment variables, so those
> variables are not global, but rather local to a project.  In Python, for
> instance, the standard way to choose a particular version of the
> interpreter and of various packages is by setting the PATH and perhaps
> PYTHONPATH variables.  It's a dumb and simple system, which has its
> advantages.

Ada has a similar situation; GPR_PROJECT_PATH specifies the path to
search for dependent libraries, and is per-project.

> If you are working on two projects with different requirements, you need
> to either launch an Emacs instance for each project, which is the
> pedestrian option, or have buffer-local environment variables.

The GNU ELPA package wisi takes an intermediate approach; it maintains a
list of project-specific environment variables in the project object,
and let-binds them in process-environment whenever it runs a process for
the project.

-- 
-- Stephe

Re: [PATCH] Buffer-local process environments

[Augusto Stoffel]

On Wed,  1 Sep 2021 at 03:42, Stephen Leake <stephen_leake@stephe-leake.org> wrote:

> Ada has a similar situation; GPR_PROJECT_PATH specifies the path to
> search for dependent libraries, and is per-project.

Great, so we have some more evidence that support for project-specific
env vars is a relevant feature.

> The GNU ELPA package wisi takes an intermediate approach; it maintains a
> list of project-specific environment variables in the project object,
> and let-binds them in process-environment whenever it runs a process for
> the project.

If I understand correctly, this works only when launching the WisiToken
program, am I correct?

python.el also supports Python "virtual environments", but this only
applies when running the Python shell.  In my opinion this approach is
too limited, because one also needs the correct environment to start
other types of processes: compile, flymake, language servers, random
private user commands, etc.

Re: [PATCH] Buffer-local process environments

[Stephen Leake]

Augusto Stoffel <arstoffel@gmail.com> writes:

> On Wed,  1 Sep 2021 at 03:42, Stephen Leake <stephen_leake@stephe-leake.org> wrote:
>
>> Ada has a similar situation; GPR_PROJECT_PATH specifies the path to
>> search for dependent libraries, and is per-project.
>
> Great, so we have some more evidence that support for project-specific
> env vars is a relevant feature.
>
>> The GNU ELPA package wisi takes an intermediate approach; it maintains a
>> list of project-specific environment variables in the project object,
>> and let-binds them in process-environment whenever it runs a process for
>> the project.
>
> If I understand correctly, this works only when launching the WisiToken
> program, am I correct?

No, any process. In ada-mode, that includes the compiler and the cross
reference tool.

Perhaps you meant using wisi requires using WisiToken to generate a
parser from a grammar; that's mostly true. There are some things you can
do with wisi that don't require a grammar, but the main focus is on
exploiting the parser.

-- 
-- Stephe

Re: [PATCH] Buffer-local process environments

[Augusto Stoffel]

On Wed,  1 Sep 2021 at 15:38, Stephen Leake <stephen_leake@stephe-leake.org> wrote:

>>> The GNU ELPA package wisi takes an intermediate approach; it maintains a
>>> list of project-specific environment variables in the project object,
>>> and let-binds them in process-environment whenever it runs a process for
>>> the project.
>>
>> If I understand correctly, this works only when launching the WisiToken
>> program, am I correct?
>
> No, any process. In ada-mode, that includes the compiler and the cross
> reference tool.

Okay, I've found wisi-prj.el, but I can't find any documentation for the
process environment aspect of it.  Therefore it's still not clear to me
how it relates with this discussion.

Wisi claims to be a code-parsing tool.  Are there any new ideas in it
that pertain to project-wise process environments specifically and could
be adopted more broadly, say by project.el?

At the risk of repeating myself too much, here is my idea of how
project-specific environments would work:

- Every command that creates a process should use the buffer-local value
  of `process-environment' if applicable (this is what the attached
  patch addresses for `compile').
- Every buffer that "belongs" to a project gets the right buffer-local
  value of `process-enviroment' (as well as `exec-path') by one of
  several possible techniques:
  - Via an exec directive in the dir-locals file
  - Using a direnv-like package such as `envrc' or `buffer-env' (both
    currently on MELPA).

Re: [PATCH] Buffer-local process environments

[Stephen Leake]

Augusto Stoffel <arstoffel@gmail.com> writes:

> On Wed,  1 Sep 2021 at 15:38, Stephen Leake <stephen_leake@stephe-leake.org> wrote:
>
>>>> The GNU ELPA package wisi takes an intermediate approach; it maintains a
>>>> list of project-specific environment variables in the project object,
>>>> and let-binds them in process-environment whenever it runs a process for
>>>> the project.
>>>
>>> If I understand correctly, this works only when launching the WisiToken
>>> program, am I correct?
>>
>> No, any process. In ada-mode, that includes the compiler and the cross
>> reference tool.
>
> Okay, I've found wisi-prj.el, but I can't find any documentation for the
> process environment aspect of it.  Therefore it's still not clear to me
> how it relates with this discussion.

The cl-defstruct wisi-prj has these slots (among others):

  compile-env
  ;; List of strings NAME=VALUE for `compilation-environment'; used
  ;; when running the compiler or makefile. Also prepended to
  ;; `process-environment' when the project file is parsed, or when
  ;; the project file is used by a tool in an external process.

  file-env
  ;; Environment (list of strings NAME=VALUE) set in project file;
  ;; prepended to `process-environment' running tools in an external
  ;; process.


> Wisi claims to be a code-parsing tool.  

Actually it's an elisp interface to an external generated parser,
allowing Emacs to take advantage of the parser.

> Are there any new ideas in it that pertain to project-wise process
> environments specifically and could be adopted more broadly, say by
> project.el?

file-env (probably renamed to project-env) would make sense in any
project that uses environment variables.

compile-env would make sense in any project that runs a compiler, or
uses compilation-mode to run a tool.

I guess both would be cl-defgeneric functions, with default
implementation returning nil.

-- 
-- Stephe