Environment variables for remote processes (was: bug#18940: 24.4; vc-hg does not disable pager, leading to hangs (at least with tramp))

Environment variables for remote processes (was: bug#18940: 24.4; vc-hg does not disable pager, leading to hangs (at least with tramp))

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Maybe I'm missing something, but what's wrong with let-binding
>> process-environment?
>
> Nothing wrong with it.  The problem is that Tramp ignores those
> let-bindings because it fails to propagate this environment to its
> remote sub-processed.

The problem is, that not all settings of process-environment might be
desired on remote hosts. process-environment keeps *local* variables.

Furthermore, some remote settings might be requested which are not in
process-environment by default. A user shall not be responsible for
those settings; that's why there is tramp-remote-process-environment.
But this is also not satisfactory, because there might be even different
settings required for different hosts.

> `tramp-sh-handle-process-file' really needs to compare
> process-environment with (default-toplevel-value 'process-environment)
> to infer the env-vars that have been added via let-binding and then
> propagate those to the remote sub-process.

tramp-sh-handle-process-file does not start a new process, it reuses the
existing one. The propagation to the remote sub-process must ensure,
that those settings are not permanent. Via a subshell, or alike.

It's not that easy as it looks at first glance. That's why I would like
a kind of design, before starting to implement.

>         Stefan

Best regards, Michael.

Re: Environment variables for remote processes

[Stefan Monnier]

>> Nothing wrong with it.  The problem is that Tramp ignores those
>> let-bindings because it fails to propagate this environment to its
>> remote sub-processed.
> The problem is, that not all settings of process-environment might be
> desired on remote hosts. process-environment keeps *local* variables.

I'm not so sure.  We're talking here about the settings which are in 
process-environment but not in (default-toplevel-value
'process-environment), so these are all settings added via let-binding
process-environment, and in all the cases I can think of, these seem to
be either useful or harmless to propagate.

But maybe we need to tweak the heuristic by excluding some env-vars
(like PATH).

> Furthermore, some remote settings might be requested which are not in
> process-environment by default.

Not sure what you're referring to here, but it seems like a different
issue than the one at hand (which is to propagate let-bound
process-environment values).

> tramp-sh-handle-process-file does not start a new process, it reuses the
> existing one. The propagation to the remote sub-process must ensure,
> that those settings are not permanent. Via a subshell, or alike.

Of course.


        Stefan

Re: Environment variables for remote processes

[Michael Albinus]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> The problem is, that not all settings of process-environment might be
>> desired on remote hosts. process-environment keeps *local* variables.
>
> I'm not so sure.  We're talking here about the settings which are in 
> process-environment but not in (default-toplevel-value
> 'process-environment), so these are all settings added via let-binding
> process-environment, and in all the cases I can think of, these seem to
> be either useful or harmless to propagate.

That sounds terrible: two classes of citizens in
process-environment. Some of them being there before Tramp connection
happened, and some of them added later, via let-bind or permanently.

How do you want to explain the difference to a user? It would make a
difference, whether an entry has been added to process-environment
before a Tramp connection, or afterwards.

And how would you deal with deleted entries of process-environment?
Something like

(let ((process-environment process-environment))
  (setenv "DISPLAY")
  (process-file ...))
  
>> Furthermore, some remote settings might be requested which are not in
>> process-environment by default.
>
> Not sure what you're referring to here, but it seems like a different
> issue than the one at hand (which is to propagate let-bound
> process-environment values).

I'm speaking about tramp-remote-process-environment, which uses another
mechanism. But if we have an accepted mechanism for environment
variables on remote hosts, there shall be only The One Way to set them.

>         Stefan

Best regards, Michael.

Re: Environment variables for remote processes

[Stefan Monnier]

> That sounds terrible: two classes of citizens in
> process-environment.  Some of them being there before Tramp connection
> happened, and some of them added later, via let-bind or permanently.

Not sure what you mean by "before Tramp connection".  The issue is what
environment to give to the command run via `process-file'.
Whether there's a pre-existing Tramp connection or not is an
implementation detail of Tramp.  But the semantics of `process-file' is
clearly that it should receive in its environment the things that are in
`process-environment', and currently Tramp fails to obey this part of
the semantics of `process-file'.

Of course, it wouldn't be correct to inherit the whole of
`process-environment' either.  And of course, I understand that
implementing this environment handling might take more than a quick
half-hour hack.

> How do you want to explain the difference to a user?  It would make a
> difference, whether an entry has been added to process-environment
> before a Tramp connection, or afterwards.

No, the particular time at which the Tramp connection was made shouldn't
make any appreciable difference.

> (let ((process-environment process-environment))
>   (setenv "DISPLAY")
>   (process-file ...))

In my naive mental model, Tramp's implementation of `process-file' will
run "env <ENVVARS> <COMMAND>" on the remote host, so we could use "-u"
to remove elements from the environment.

But let's start by handling additions, and then we'll see if/when we need to
handle removals.

>>> Furthermore, some remote settings might be requested which are not in
>>> process-environment by default.
>> Not sure what you're referring to here, but it seems like a different
>> issue than the one at hand (which is to propagate let-bound
>> process-environment values).
> I'm speaking about tramp-remote-process-environment, which uses another
> mechanism. But if we have an accepted mechanism for environment
> variables on remote hosts, there shall be only The One Way to set them.

So it does seem like a completely different issue.  What I'm concerned
here is about the environment that is "per subprocess" rather than "per
connection".


        Stefan

Re: Environment variables for remote processes

[Michael Albinus]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> Not sure what you mean by "before Tramp connection".  The issue is what
> environment to give to the command run via `process-file'.
> Whether there's a pre-existing Tramp connection or not is an
> implementation detail of Tramp.  But the semantics of `process-file' is
> clearly that it should receive in its environment the things that are in
> `process-environment', and currently Tramp fails to obey this part of
> the semantics of `process-file'.
>
> Of course, it wouldn't be correct to inherit the whole of
> `process-environment' either.  And of course, I understand that
> implementing this environment handling might take more than a quick
> half-hour hack.

It's not so simple to decide what's appropriate settings. You do NOT
want to propagate all your local environment variables to the remote
process. Your local settings of DISPLAY or DBUS_SESSION_BUS_ADDRESS, to
give prominent examples, would confuse your remote process heavily.

And there might be application specific environment variables we've
never heard about, which could damage the remote process as well.

How to decide what's appropriate for the remote process? I don't know.

> In my naive mental model, Tramp's implementation of `process-file' will
> run "env <ENVVARS> <COMMAND>" on the remote host, so we could use "-u"
> to remove elements from the environment.

No. With a sufficient amount of environment variables, you would exceed
the maximum length of shell command lines. Tramp did it several times
already, and I had to work-around that. Then "env ..." construct is
applicable only, when you know in advance that there aren't too many settings.

> But let's start by handling additions, and then we'll see if/when we need to
> handle removals.
>
>>>> Furthermore, some remote settings might be requested which are not in
>>>> process-environment by default.
>>> Not sure what you're referring to here, but it seems like a different
>>> issue than the one at hand (which is to propagate let-bound
>>> process-environment values).
>> I'm speaking about tramp-remote-process-environment, which uses another
>> mechanism. But if we have an accepted mechanism for environment
>> variables on remote hosts, there shall be only The One Way to set them.
>
> So it does seem like a completely different issue.  What I'm concerned
> here is about the environment that is "per subprocess" rather than "per
> connection".

Yes, but here we must speak about implementation. Tramp opens a
connection (a shell) on the remote host, and sends all its internal
commands via this shell. It sends also the command intended for
process-file to that shell. It does *not* open a new (sub)process, which
could inherit environment variables from Emacs, and alike. The
environment is the same as when that shell has been started. That's why
it is, at least as of today, "per connection".

>         Stefan

Best regards, Michael.

Re: Environment variables for remote processes

[Stefan Monnier]

> It's not so simple to decide what's appropriate settings. You do NOT
> want to propagate all your local environment variables to the remote
> process. Your local settings of DISPLAY or DBUS_SESSION_BUS_ADDRESS, to
> give prominent examples, would confuse your remote process heavily.

Check again: these will not be present in "the difference between
process-environment and (default-toplevel-value 'process-environment)".

>> In my naive mental model, Tramp's implementation of `process-file' will
>> run "env <ENVVARS> <COMMAND>" on the remote host, so we could use "-u"
>> to remove elements from the environment.
> No. With a sufficient amount of environment variables, you would exceed
> the maximum length of shell command lines. Tramp did it several times
> already, and I had to work-around that. Then "env ..." construct is
> applicable only, when you know in advance that there aren't too
> many settings.

But whichever other scheme we end up using, removing env-vars shouldn't
be any harder than adding some.

> Yes, but here we must speak about implementation. Tramp opens a
> connection (a shell) on the remote host, and sends all its internal
> commands via this shell. It sends also the command intended for
> process-file to that shell. It does *not* open a new (sub)process, which
> could inherit environment variables from Emacs, and alike. The
> environment is the same as when that shell has been started. That's why
> it is, at least as of today, "per connection".

I know that.  That's why I suggested to send "env <ENV> <CMD>".
Another option might be:

    (export VAR1=VAL1
     unset VAR2
     export VAR3=VAL3
     ...
     <CMD>)


-- Stefan

Re: Environment variables for remote processes

[andres.ramirez]

Hi guys this is  my case. I think it qualifies as as subprocess. I hope it helps to clarify some ideas:

* how I start emacs
It is started from a crontab and using daemon:
@reboot emacs --quick --daemon
** Notice: known condition emacs daemon issue with gtk (this gtk bug have years )
http://bugzilla.gnome.org/show_bug.cgi?id=85715
* What's the problem with emacs started before X?
 no 'DBUS_SESSION_BUS_ADDRESS' and other variables.
Test the condition with an external process notify-send (test the line below on eshell):
notify-send 'wellcome' 'Visit <a href="http://www.kipuamutay.com">kipuamutay</a>'
** enviroment variables from emacs started on cron job (very short)
COLUMNS=80
HOME=/home/aramirez
LANG=en_US.UTF-8
LINES=34
LOGNAME=aramirez
OLDPWD=/home/aramirez/downloads
PATH=/usr/bin:/bin
PWD=/home/aramirez
SHELL=/bin/sh
SHLVL=1
TERM=dumb
USER=aramirez
XAUTHORITY=/home/aramirez/.Xauthority
_=/usr/bin/emacs
** enviroment variables from emacs started after X
ANT_HOME=/usr/share/apache-ant
CLASSPATH=/home/aramirez/GNUstep/Library/Libraries/Java:/usr/lib/GNUstep/Libraries/Java
COLUMNS=80
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-uxYK2KP5AO,guid=bde80a1c96d99a424bfa48eb546cacf2
GUILE_LOAD_PATH=/home/aramirez/GNUstep/Library/Libraries/Guile:/usr/lib/GNUstep/Libraries/Guile
HOME=/home/aramirez
INFOPATH=/usr/share/info::/home/aramirez/GNUstep/Library/Documentation/info:
LANG=en_US.UTF-8
LC_COLLATE=C
LD_LIBRARY_PATH=/home/aramirez/GNUstep/Library/Libraries:/usr/lib
LIBRARY_COMBO=gnu-gnu-gnu
LINES=34
LOGNAME=aramirez
MAIL=/var/spool/mail/aramirez
MOZ_PLUGIN_PATH=/usr/lib/mozilla/plugins
OLDPWD=/home/aramirez/downloads
PATH=/home/aramirez/GNUstep/Tools:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/local/bin:/home/aramirez/bin:/usr/local/bin:/home/aramirez/bin
PWD=/home/aramirez
SHELL=/bin/bash
SHLVL=5
TERM=dumb
USER=aramirez
WINDOWID=14680098
WINDOWPATH=1
XAUTHORITY=/home/aramirez/.Xauthority
XDG_RUNTIME_DIR=/run/user/1000
XDG_SEAT=seat0
XDG_SESSION_ID=c1
XDG_VTNR=1
XTERM_LOCALE=en_US.UTF-8
XTERM_SHELL=/bin/bash
XTERM_VERSION=XTerm(312)
_=/usr/bin/emacs
* CONCLUSION: In this particular case. It should be possible to add the necessary environment variables for notify-send to work.
(dbus-init-bus my-dbus-address) is not enough

** This could be unrelated to this thread, but related to this case;when emacs is started from a cron job before X: gnupg validation is not possible
*** issue with this when restarting emacs remotely (GPG developers and the gui-popup 4 requesting the passphrase), when no X then no popup :(
http://permalink.gmane.org/gmane.emacs.devel/169412

Best Regards

Re: Environment variables for remote processes

[Michael Albinus]

andres.ramirez <andres.ramirez@kipuamutay.com> writes:

> Hi guys this is  my case. I think it qualifies as as subprocess. I hope it helps to clarify some ideas:

Hi Andres,

> * how I start emacs
> It is started from a crontab and using daemon:
> @reboot emacs --quick --daemon
> ** Notice: known condition emacs daemon issue with gtk (this gtk bug have years )
> http://bugzilla.gnome.org/show_bug.cgi?id=85715

I guess you connect that daemonized Emacs via emacsclient.

> * What's the problem with emacs started before X?
>  no 'DBUS_SESSION_BUS_ADDRESS' and other variables.

For all missing environment variables, you must add them from your
client session to the daemonized Emacs via setenv. But I'm pretty sure
you know that already :-)

> ** enviroment variables from emacs started on cron job (very short)
> ** enviroment variables from emacs started after X
> * CONCLUSION: In this particular case. It should be possible to add the necessary environment variables for notify-send to work.
> (dbus-init-bus my-dbus-address) is not enough

As said above, setenv will be your friend. This extends process-environment.

However, this is not sufficient for remote processes started via
Tramp. Here you have now (after the recent change) two options:

- Add the environment variable to tramp-remote-process-environment
  before opening a remote connection:

  (setq tramp-remote-process-environment
        (cons
          "DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-uxYK2KP5AO,guid=bde80a1c96d99a424bfa48eb546cacf2"
          tramp-remote-process-environment))

  This environment variable will exist permanently on the remote side
  while running Tramp.

- Let-bind the environment variable to process-environment before
  calling process-file or start-file-process:

  (let ((process-environment
          (cons "DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-uxYK2KP5AO,guid=bde80a1c96d99a424bfa48eb546cacf2"
                process-environment)))
     (process-file ...))

  This environment variable will exist just for the process started on
  the remote side.
  
> Best Regards

Best regards, Michael.

Re: Environment variables for remote processes

[Michael Albinus]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> It's not so simple to decide what's appropriate settings. You do NOT
>> want to propagate all your local environment variables to the remote
>> process. Your local settings of DISPLAY or DBUS_SESSION_BUS_ADDRESS, to
>> give prominent examples, would confuse your remote process heavily.
>
> Check again: these will not be present in "the difference between
> process-environment and (default-toplevel-value
> 'process-environment)".

But then you change the definition of process-environment. The docstring
says "let-binding `process-environment' is an easy way to temporarily
change the value of an environment variable". It does not say it should
be the only way to do that, especially for process-file.

And you still have the distinction between processes on the local host,
and processes on the remote host. The former ones can expect all
settings from process-environment, the latter ones can expect only the
environment vatriables which are different to the toplevel value. Up to
now, nobody had to care about processes running locally or remote;
process-file did arrange it magically.

I do not say that I completely oppose (I start to understand your
proposal), but this might break existing code.

Fortunately, we do not need to care XEmacs compatibility here: there is
no process-file :-)

> That's why I suggested to send "env <ENV> <CMD>".  Another option
> might be:
>
>     (export VAR1=VAL1
>      unset VAR2
>      export VAR3=VAL3
>      ...
>      <CMD>)

Another possibility might be heredoc commands. But that's an
implementation detail.

> -- Stefan

Best regards, Michael.

Re: Environment variables for remote processes

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

> I do not say that I completely oppose (I start to understand your
> proposal), but this might break existing code.

Nitpicking: there is no clean way to remove an environment variable,
which might have been set remotely and which you do not want to have
set while executing process-file.

You can check, whether an environment variable exists on toplevel
process-environment, and does not exist on the let-bound
process-environment. Then you can call "unset ...".

But how do you do it, when this variable does not exists in your
toplevel process-environment?

Best regards, Michael.

Re: Environment variables for remote processes

[Stefan Monnier]

> But then you change the definition of process-environment.  The docstring
> says "let-binding `process-environment' is an easy way to temporarily
> change the value of an environment variable".  It does not say it should
> be the only way to do that, especially for process-file.

What other way do you have in mind?

If you just use straight `setq' this will affect all subsequent
executions of subprocesses, so it only makes sense for env settings
which are not specific for one particular subprocess.

Of course, you could use

     (setq process-environment <foo1>)
     ...
     (setq process-environment <revert>)

but that's quite less convenient and efficient than using `let', so I'm
not worried if we don't handle that case.

> And you still have the distinction between processes on the local host,
> and processes on the remote host.  The former ones can expect all
> settings from process-environment, the latter ones can expect only the
> environment vatriables which are different to the toplevel value.

That's right.

> Up to now, nobody had to care about processes running locally or
> remote; process-file did arrange it magically.

That's not true: vc-hg.el now even has to test file-remote-p to decide
which command to run, specifically because process-file does not arrange
it magically.
Currently, process-file simply ignores process-environment, which works
OK in some cases but is clearly not quite right.

> Nitpicking: there is no clean way to remove an environment variable,
> which might have been set remotely and which you do not want to have
> set while executing process-file.

Actually, C-h v process-environment says:

   irrespective of where it comes from.  To use `process-environment' to
   remove an environment variable, include only its name in the list,
   without "=VALUE".


-- Stefan

Re: Environment variables for remote processes

[Michael Albinus]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> But then you change the definition of process-environment.  The docstring
>> says "let-binding `process-environment' is an easy way to temporarily
>> change the value of an environment variable".  It does not say it should
>> be the only way to do that, especially for process-file.
>
> What other way do you have in mind?
>
> If you just use straight `setq' this will affect all subsequent
> executions of subprocesses, so it only makes sense for env settings
> which are not specific for one particular subprocess.
>
> but that's quite less convenient and efficient than using `let', so I'm
> not worried if we don't handle that case.

There's setenv. And it is the recommended way to manipulate process-environment.

>> Nitpicking: there is no clean way to remove an environment variable,
>> which might have been set remotely and which you do not want to have
>> set while executing process-file.
>
> Actually, C-h v process-environment says:
>
>    irrespective of where it comes from.  To use `process-environment' to
>    remove an environment variable, include only its name in the list,
>    without "=VALUE".

You are right, I forgot this.

> -- Stefan

Best regards, Michael.

Re: Environment variables for remote processes

[Stefan Monnier]

>> What other way do you have in mind?
>> If you just use straight `setq' this will affect all subsequent
>> executions of subprocesses, so it only makes sense for env settings
>> which are not specific for one particular subprocess.
>> but that's quite less convenient and efficient than using `let', so I'm
>> not worried if we don't handle that case.
> There's setenv.  And it is the recommended way to manipulate
> process-environment.

I think it's basically the same situation as with setq: either it's done
within a `let' (and we're back to the let-binding case), or it affects
all subsequent subprocesses.


        Stefan

Re: Environment variables for remote processes

[Michael Albinus]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> I think it's basically the same situation as with setq: either it's done
> within a `let' (and we're back to the let-binding case), or it affects
> all subsequent subprocesses.

Well, a pushed a respective patch to the trunk. Seems to work at least
fine with `eshell' and `shell', which are not trivial examples.

Let's see, whether other people start to rant.

>         Stefan

Best regards, Michael.

Re: Environment variables for remote processes

[Stefan Monnier]

>> I think it's basically the same situation as with setq: either it's done
>> within a `let' (and we're back to the let-binding case), or it affects
>> all subsequent subprocesses.
> Well, a pushed a respective patch to the trunk.  Seems to work at least
> fine with `eshell' and `shell', which are not trivial examples.

Does it mean that the file-remote-p hack in vc-hg.el is redundant now?


        Stefan

Re: Environment variables for remote processes

[Michael Albinus]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> Does it mean that the file-remote-p hack in vc-hg.el is redundant now?

Yes. You might have seen, that I have removed it.

>         Stefan

Best regards, Michael.