exec-path and PATH

exec-path and PATH

[Christoph Scholtes]

I was troubleshooting a problem with Emacs not being able to find
certain external programs on Windows, e.g. grep or find.

I noticed that there seem to be two ways that external programs are
invoked:

1. directly, via start-process
2. indirectly, through the shell

It was a little confusing that it was not enough to add the path to
grep.exe to exec-path for Emacs to find it. The same would work fine for
svn.exe when invoked by the vc backend. 

I add the paths to certain programs in my init.el, e.g. GnuWin32 for
grep. For them to work in every situation I have to remember to add them
to exec-path and the Windows PATH environment variable. This seems
somewhat redundant, no? I would prefer only manipulating the exec-path
to tell Emacs where to find the programs.

Maybe this is a silly question, but why are some of the programs invoked
different than others? 

Christoph

PS: Here is a small docfix patch for process.c for review:

=== modified file 'src/ChangeLog'
--- src/ChangeLog	2011-03-17 19:55:40 +0000
+++ src/ChangeLog	2011-03-19 02:30:14 +0000
@@ -1,3 +1,7 @@
+2011-03-19  Christoph Scholtes  <cschol2112@googlemail.com>
+
+	* process.c (Fstart_process): Doc fix.
+
 2011-03-17  Eli Zaretskii  <eliz@gnu.org>
 
 	* makefile.w32-in ($(BLD)/unexw32.$(O)): Depend on $(SRC)/unexec.h.

=== modified file 'src/process.c'
--- src/process.c	2011-03-17 05:18:33 +0000
+++ src/process.c	2011-03-19 02:26:39 +0000
@@ -1510,7 +1510,7 @@
 function to handle the output.  BUFFER may also be nil, meaning that
 this process is not associated with any buffer.
 
-PROGRAM is the program file name.  It is searched for in PATH.  If
+PROGRAM is the program file name.  It is searched for in EXEC-PATH.  If
 nil, just associate a pty with the buffer.  Remaining arguments are
 strings to give program as arguments.

Re: exec-path and PATH

[Eli Zaretskii]

> From: Christoph Scholtes <cschol2112@googlemail.com>
> Date: Fri, 18 Mar 2011 21:19:55 -0600
> 
> I noticed that there seem to be two ways that external programs are
> invoked:
> 
> 1. directly, via start-process
> 2. indirectly, through the shell
> 
> It was a little confusing that it was not enough to add the path to
> grep.exe to exec-path for Emacs to find it. The same would work fine for
> svn.exe when invoked by the vc backend. 
> 
> I add the paths to certain programs in my init.el, e.g. GnuWin32 for
> grep. For them to work in every situation I have to remember to add them
> to exec-path and the Windows PATH environment variable. This seems
> somewhat redundant, no? I would prefer only manipulating the exec-path
> to tell Emacs where to find the programs.

When a program is invoked via the shell, the shell will look for it
via PATH.  More generally, any program invoked by Emacs that needs to
invoke other programs (e.g., compiler, Make, etc.) will look for those
subordinate programs via PATH.  Only Emacs knows about exec-path.  So
for your life quality in Emacs to be much better, I suggest to always
have exec-path and PATH in sync.

Maybe we should consider the possibility to automatically set PATH
from exec-path each time Emacs invokes a subprocess, because this
issue comes up time and again in gnu.emacs.help.

> Maybe this is a silly question, but why are some of the programs invoked
> different than others? 

Because the shell has features ready for our use that would be a
wasted effort to re-implement in Lisp or C.  E.g., complex
redirection, pipes, etc.

> PS: Here is a small docfix patch for process.c for review:

This is obviously correct, so please commit it on the emacs-23
branch.

> +PROGRAM is the program file name.  It is searched for in EXEC-PATH.  If

"PATH" is a literal name of an environment variable, that's why it is
in caps.  But `exec-path' is spelled in lower case, so please don't
up-case it here.  Also, please enclose in quotes, so it produces a
link in Emacs.  I would even say


  It is searched for in `exec-path' (which see).

Thanks.

Re: exec-path and PATH

[Christoph Scholtes]

Eli Zaretskii <eliz@gnu.org> writes:

> Maybe we should consider the possibility to automatically set PATH
> from exec-path each time Emacs invokes a subprocess, because this
> issue comes up time and again in gnu.emacs.help.

This would be helpful, I think.

Can we set PATH to exec-path for the duration of the shell
execution? IIRC, if you change the PATH in a cmd session it only applies to
that session. Is it possible to something similar here?

>> Maybe this is a silly question, but why are some of the programs invoked
>> different than others? 
>
> Because the shell has features ready for our use that would be a
> wasted effort to re-implement in Lisp or C.  E.g., complex
> redirection, pipes, etc.

That makes sense. Thanks.

>> PS: Here is a small docfix patch for process.c for review:
>
> This is obviously correct, so please commit it on the emacs-23
> branch.

OK.

> "PATH" is a literal name of an environment variable, that's why it is
> in caps.  But `exec-path' is spelled in lower case, so please don't
> up-case it here.  Also, please enclose in quotes, so it produces a
> link in Emacs.  I would even say
>
>
>   It is searched for in `exec-path' (which see).
>
> Thanks.

OK, I will change it.

Thanks,
Christoph

Re: exec-path and PATH

[Christoph Scholtes]

Eli Zaretskii <eliz@gnu.org> writes:

> I would even say
>
>
>   It is searched for in `exec-path' (which see).

Literally? What does the `(which see)' mean?

Christoph

Re: exec-path and PATH

[PJ Weisberg]

On 3/19/11, Christoph Scholtes <cschol2112@googlemail.com> wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I would even say
>>
>>
>>   It is searched for in `exec-path' (which see).
>
> Literally? What does the `(which see)' mean?

Same thing as "(see exec-path)".  A reference to another part of the
documentation.

-PJ

Re: exec-path and PATH

[Christoph Scholtes]

PJ Weisberg <pj@irregularexpressions.net> writes:

> On 3/19/11, Christoph Scholtes <cschol2112@googlemail.com> wrote:
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> I would even say
>>>
>>>
>>>   It is searched for in `exec-path' (which see).
>>
>> Literally? What does the `(which see)' mean?
>
> Same thing as "(see exec-path)".  A reference to another part of the
> documentation.

Thanks.

Christoph

Re: exec-path and PATH

[Eli Zaretskii]

> From: Christoph Scholtes <cschol2112@googlemail.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 19 Mar 2011 10:36:44 -0600
> 
> Can we set PATH to exec-path for the duration of the shell
> execution?

What do you mean by "shell execution"?

What I had in mind was that we should synchronize PATH with exec-path
each time we invoke a subsidiary process, either through call-process
or start-process.

There could be an issue with async subprocesses that are already
running, though: they still use the old PATH, while Emacs will see the
new value.  This could lead to subtle bugs.

> IIRC, if you change the PATH in a cmd session it only applies to
> that session.

But we don't change PATH _in_ the cmd session, we change it _before_
the session begins.

Re: exec-path and PATH

[Eli Zaretskii]

> From: Christoph Scholtes <cschol2112@googlemail.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 19 Mar 2011 10:49:08 -0600
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I would even say
> >
> >
> >   It is searched for in `exec-path' (which see).
> 
> Literally?

Yes.

> What does the `(which see)' mean?

A translation of "q.v.", see http://en.wiktionary.org/wiki/q.v.

Re: exec-path and PATH

[Harald Hanche-Olsen]

[PJ Weisberg <pj@irregularexpressions.net> (2011-03-19 17:18:33 UTC)]

> On 3/19/11, Christoph Scholtes <cschol2112@googlemail.com> wrote:
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> I would even say
> >>
> >>
> >>   It is searched for in `exec-path' (which see).
> >
> > Literally? What does the `(which see)' mean?
> 
> Same thing as "(see exec-path)".  A reference to another part of the
> documentation.

And so it is entirely redundant, since the "exec-path" will already be
marked as a link.

- Harald

Re: exec-path and PATH

[Christoph Scholtes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Can we set PATH to exec-path for the duration of the shell
>> execution?
>
> What do you mean by "shell execution"?

While the shell is executing the external program.

> What I had in mind was that we should synchronize PATH with exec-path
> each time we invoke a subsidiary process, either through call-process
> or start-process.

That should work, but I wonder if we want to do this work before every
process. It would be nicer to sync the two whenever exec-path is
modified. I don't know if this is possible though when modifying
exec-path diretcly via setq, for example.

Maybe that's premature optimization tough.

> There could be an issue with async subprocesses that are already
> running, though: they still use the old PATH, while Emacs will see the
> new value.  This could lead to subtle bugs.

This would only be a problem if the exec-path was modified between two
processes. Also, this problem exists today, if I modify the PATH in
between two processes via setenv, right?

>> IIRC, if you change the PATH in a cmd session it only applies to
>> that session.
>
> But we don't change PATH _in_ the cmd session, we change it _before_
> the session begins.

I can change the PATH after running an instance of cmd.exe, no? It will
change the PATH for that session only, iirc.

Christoph

Re: exec-path and PATH

[Eli Zaretskii]

> From: Christoph Scholtes <cschol2112@googlemail.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 19 Mar 2011 13:50:03 -0600
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Can we set PATH to exec-path for the duration of the shell
> >> execution?
> >
> > What do you mean by "shell execution"?
> 
> While the shell is executing the external program.

I don't see a point in this.  PATH in process-environment is not used
within Emacs except to pass it to sub-processes.  Why would we want to
restore its value after the sub-process exits?

> > What I had in mind was that we should synchronize PATH with exec-path
> > each time we invoke a subsidiary process, either through call-process
> > or start-process.
> 
> That should work, but I wonder if we want to do this work before every
> process.

Concatenating a few dozen of strings cannot take too much.

> >> IIRC, if you change the PATH in a cmd session it only applies to
> >> that session.
> >
> > But we don't change PATH _in_ the cmd session, we change it _before_
> > the session begins.
> 
> I can change the PATH after running an instance of cmd.exe, no? It will
> change the PATH for that session only, iirc.

Yes, but how is this relevant to the issue at hand?

Re: exec-path and PATH

[Stefan Monnier]

> Maybe we should consider the possibility to automatically set PATH
> from exec-path each time Emacs invokes a subprocess, because this
> issue comes up time and again in gnu.emacs.help.

That sounds like a good idea, although there are some directories in
exec-path which I think should not be transferred to PATH
(e.g. /usr/lib/emacs/23.2/i486-linux-gnu).  Not sure how best to do that.


        Stefan

Re: exec-path and PATH

[Stefan Monnier]

>> >> I would even say
>> >>   It is searched for in `exec-path' (which see).
>> > Literally? What does the `(which see)' mean?
>> Same thing as "(see exec-path)".  A reference to another part of the
>> documentation.
> And so it is entirely redundant, since the "exec-path" will already be
> marked as a link.

It's not entirely redundant, since the `exec-path' just names the
variable and provides a link in case you want to look it up, whereas the
"which see" bit actually encourages the user to follow the link.
Whether the nuance matters, I don't know.


        Stefan

Re: exec-path and PATH

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Christoph Scholtes <cschol2112@googlemail.com>,  emacs-devel@gnu.org
> Date: Sat, 19 Mar 2011 22:50:25 -0400
> 
> > Maybe we should consider the possibility to automatically set PATH
> > from exec-path each time Emacs invokes a subprocess, because this
> > issue comes up time and again in gnu.emacs.help.
> 
> That sounds like a good idea, although there are some directories in
> exec-path which I think should not be transferred to PATH
> (e.g. /usr/lib/emacs/23.2/i486-linux-gnu).  Not sure how best to do that.

What harm could be done if those directories are added to PATH?

Re: exec-path and PATH

[Christoph Scholtes]

Eli Zaretskii <eliz@gnu.org> writes:

> PATH in process-environment is not used within Emacs except to pass it
> to sub-processes.  Why would we want to restore its value after the
> sub-process exits?

Sorry Eli. This was my mistake. I was under the wrong impression that
Emacs worked with the OS's PATH directly and not with its own copy of
it. After looking at process-environment and the setenv function
documentation I see how it works.

However, now I wonder why we have a facility like exec-path at
all. Wouldn't the process-environment's PATH be enough?
(start|call)-process could look at PATH directly instead of
exec-path. This would also solve the problem of having to keep two
separate path collections (PATH and exec-path) up to date.

I saw that quite a few functions use exec-path, so I am not necessarily
suggesting to change it. I am just curious.

Christoph

Re: exec-path and PATH

[Eli Zaretskii]

> From: Christoph Scholtes <cschol2112@googlemail.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 20 Mar 2011 13:45:50 -0600
> 
> However, now I wonder why we have a facility like exec-path at
> all. Wouldn't the process-environment's PATH be enough?

Because working with a list is much easier than with a single string?

Re: exec-path and PATH

[Eric Hanchrow]

On Sun, Mar 20, 2011 at 1:36 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> However, now I wonder why we have a facility like exec-path at
>> all. Wouldn't the process-environment's PATH be enough?
> Because working with a list is much easier than with a single string?

That would argue for making "exec-path" a sort of magical variable --
one whose value changes all by itself, whenever the PATH environment
variable changes.  Would that be feasible?

Re: exec-path and PATH

[Wojciech Meyer]

Hi Eli,

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Christoph Scholtes <cschol2112@googlemail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 20 Mar 2011 13:45:50 -0600
>> 
>> However, now I wonder why we have a facility like exec-path at
>> all. Wouldn't the process-environment's PATH be enough?
>
> Because working with a list is much easier than with a single string?

From the user perspective - yes.

We could have a module to handle operations on PATH which would include
a function to append/prepend a list of paths and then start some
function or code with a path set-up. I know this would be a major change
though. Other solution is exec-path to be just additional set of the
directories that we pre-pend to the PATH.  In both cases however we'd
lose ability to amend the environment which might be not desirable I
agree.

Thanks,
Wojciech

Re: exec-path and PATH

[Eric Schulte]

Eric Hanchrow <eric.hanchrow@gmail.com> writes:

> On Sun, Mar 20, 2011 at 1:36 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> However, now I wonder why we have a facility like exec-path at
>>> all. Wouldn't the process-environment's PATH be enough?
>> Because working with a list is much easier than with a single string?
>
> That would argue for making "exec-path" a sort of magical variable --
> one whose value changes all by itself, whenever the PATH environment
> variable changes.  Would that be feasible?

Or rather replace the exec-path variable with an exec-path function,
along the lines of 

  (defun exec-path ()
    (split-string (getenv "PATH") ":"))

I have seen the existence of both exec-path and PATH (two separate
variables for one single concept) serve as a stumbling block for a
number of new Emacs users.