Managing environments (Python venv, guix environment, etc.)

Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Hi emacs-devel,

There are a number of tools which create a reproducible working
environment for projects by changing environment variables such as PATH
and LIBRARY_PATH to point at project-specific directories containing
project-specific versions of libraries and executables. Two examples
among many: Python's "virtualenv" and Guix's "guix environment".

A typical shell-based workflow with these tools would involve starting
your shell and activating the tool so it mutates your environment and
puts you into the working environment specific to that project. With
multiple terminals, one can have multiple shells open at the same time
to work on multiple different projects each with a different
environment.

One could run several Emacs processes with different environments to
work on different projects, but that's very painful. So I'm trying to
teach Emacs how to handle multiple environments within the same
process. Currently, Emacs can only operate in one environment at a time,
since exec-path and process-environment are single global variables that
are referenced directly by many different Elisp functions.

Here's the ideal: The recently added support for projects in project.el
is augmented to also know about the environment of those projects. When
compile, async-shell-command, etc., are invoked in a buffer belonging to
a project, the executed process is run with the environment defined for
that project. Any other Elisp functions that inspect the environment
should reference the environment defined for the current project.

But I would count it as a failure if multiple-environment support
required every Elisp package that runs processes to be explicitly
modified to support it. Even just modifying everything in Emacs itself
would be a major undertaking. So backwards-compatibility through some
clever hacks is important.

But I am only a humble novice, and am unsure about the best way to
achieve this. Here are a few ideas I had:

- Make exec-path and process-environment buffer-local

  I am told that this will cause a lot of bizarre behavior. In
  particular, I understand that this won't work with M-x compile, which
  reuses another buffer to run the compilation.

- Make exec-path and process-environment a different kind of
  buffer-local dynamic variable

  The behavior of this variable would be that when it is defined locally
  in the current buffer, it behaves like a normal dynamic variable. And
  when it is not defined locally in the current buffer, looking up the
  variable will look up the variable dynamically instead of looking for
  a global value set with setq-default.
  This would mean that when M-x compile is invoked from some buffer, if
  exec-path and process-environment are not buffer-local in the compile
  buffer, those variables will be looked up dynamically, and found in
  the buffer-locals of the buffer that was current at the time of the
  M-x compile invocation.
  I don't know if this is possible given the implementation of dynamic
  variables in Emacs Lisp.

- Use the same functionality as TRAMP, somehow

  TRAMP supports transparently running processes in multiple arbitrary
  environments (usually ones accessible over the network, but local
  environments should work too).
  But I'm not really familiar with how TRAMP works, so I'm not sure if
  it's possible to make use of its process-running capabilities without
  using magic filenames, and without copying files back and forth (an
  ugly overhead in this case since the files would all be local).

- Use dir-locals

  I'm not sure how this could work, given the problems with making
  exec-path and process-environment buffer-local, but including it to be
  complete.

If any of the great masters have suggestions about how to achieve this,
I would greatly appreciate it. Thanks.

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

> - Make exec-path and process-environment buffer-local
>   I am told that this will cause a lot of bizarre behavior.  In
>   particular, I understand that this won't work with M-x compile, which
>   reuses another buffer to run the compilation.

Indeed, it probably won't work right.

> - Make exec-path and process-environment a different kind of
>   buffer-local dynamic variable
[...]
>   the buffer-locals of the buffer that was current at the time of the
>   M-x compile invocation.

Too hand-wavy and the details are likely to be pretty ugly (lots of
guessing involved).

> - Use dir-locals

>   I'm not sure how this could work, given the problems with making
>   exec-path and process-environment buffer-local, but including it to be
>   complete.

If M-x compile obeys dir-locals that could work, yes.

Another option is to use process-file's file-name-handler-alist.

And of course, yet another is to advice-add on call-process and start-process.


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

> If M-x compile obeys dir-locals that could work, yes.

I checked (by creating a .dir-locals.el containing a setting for the
exec-path variable in compliation-mode) and it doesn't seem to read
exec-path from dir-locals.

> Another option is to use process-file's file-name-handler-alist.

So we'd maintain a mapping from visited filenames to the environment
that that filename is in, and make sure that visited filenames in a
nonstandard environment are present in file-name-handler-alist.

Tying environments to filenames in this way makes sense when an
environment corresponds to a project which exists in the filesystem. In
that case we can just put the project root directory into
file-name-handler-alist, and instantly everything in the project works
in the new environment.

And for running one-off commands (to send my mail, or something else not
associated with any project) I've always been able to just wrap the
command with a let-binding of exec-path and process-environment. (The
environment package could provide a new macro to do that)

So then there would be two ways to enter a different environment for Emacs:
One for projects and based on file-name-handler-alist, and one for single
commands and based on let-binding. I think that would be enough to start.

Does this approach (putting project root directories into
file-name-handler-alist) sound like something that could be accepted
into core Emacs? If so I'll get started.

Also, in the future, it would be nice to have the ability to just run
(something like) "M-x environment-switch" to switch the current buffer
to a new environment, and have an invocation of e.g. "M-x compile" just
work. Does anyone have any thoughts about how to eventually support
that? Is there a clever way to do it, or will we just have to end up
rewriting the world?

> And of course, yet another is to advice-add on call-process and
> start-process.

Yeah, but I'd like to be able to get this into Emacs (hopefully by 26).


Thanks for the advice!

Re: Managing environments (Python venv, guix environment, etc.)

[Andreas Röhler]

On 18.07.2016 00:41, sbaugh@catern.com wrote:
>> If M-x compile obeys dir-locals that could work, yes.
> I checked (by creating a .dir-locals.el containing a setting for the
> exec-path variable in compliation-mode) and it doesn't seem to read
> exec-path from dir-locals.
>
>> Another option is to use process-file's file-name-handler-alist.
> So we'd maintain a mapping from visited filenames to the environment
> that that filename is in, and make sure that visited filenames in a
> nonstandard environment are present in file-name-handler-alist.
>
> Tying environments to filenames in this way makes sense when an
> environment corresponds to a project which exists in the filesystem. In
> that case we can just put the project root directory into
> file-name-handler-alist, and instantly everything in the project works
> in the new environment.
>
> And for running one-off commands (to send my mail, or something else not
> associated with any project) I've always been able to just wrap the
> command with a let-binding of exec-path and process-environment. (The
> environment package could provide a new macro to do that)
>
> So then there would be two ways to enter a different environment for Emacs:
> One for projects and based on file-name-handler-alist, and one for single
> commands and based on let-binding. I think that would be enough to start.
>
> Does this approach (putting project root directories into
> file-name-handler-alist) sound like something that could be accepted
> into core Emacs? If so I'll get started.

Hi,

not being an Emacs core-developer, you may safely ignore my answer.

The problem here IMHO is, that "environment" and resp. also "project" 
might mean quite different things according to language and concrete0 
task. Certainly there are groups of apps which demand the similar, 
others will be not comparable.

So far extending the related packages --projectile or whatever-- seems 
the way to go. DRY by separating basic stuff should always be an option.

Thanks for your care,

Andreas

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

> Tying environments to filenames in this way makes sense when an
> environment corresponds to a project which exists in the filesystem.

That was my assumption: your "environments" correspond to
specific subdirectories in the file-system.

> Does this approach (putting project root directories into
> file-name-handler-alist) sound like something that could be accepted
> into core Emacs? If so I'll get started.

Hard to tell.  Even the question is unclear to me.  The approach
I suggest assumes you don't need/want to make changes to what I consider
"core" and instead use existing hooks (in this case
file-name-handler-alist).  So it could be distributed via GNU ELPA,
for example.

> Also, in the future, it would be nice to have the ability to just run
> (something like) "M-x environment-switch" to switch the current buffer
> to a new environment, and have an invocation of e.g. "M-x compile" just
> work.

That seems to assume that an environment could be bound to a buffer
rather than a subdirectory, IOW it wouldn't be chained to
a subdirectory.  Such environments seem to introduce further questions
about how to determine which env to use for which buffer (IOW what UI
to expose to the user).

> Does anyone have any thoughts about how to eventually support that?

You could set file-name-handler-alist buffer-locally, maybe?

Still, the main issue is what to do about things that happen in other
buffers but which the user would consider as related.

>> And of course, yet another is to advice-add on call-process and
>> start-process.
> Yeah, but I'd like to be able to get this into Emacs (hopefully by 26).

You mean you want it distributed with Emacs?  Most new packages nowadays
end up in GNU ELPA rather than in Emacs, unless they offer functionality
needed by other packages that are distributed with Emacs.


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> That was my assumption: your "environments" correspond to
> specific subdirectories in the file-system.

Nope, not necessarily. Python virtualenvs do have an associated
directory that contains the installed libraries for that
environment. But guix environment only updates environment variables to
point to the locations of already existing libraries, and so guix
environments don't exist in the filesystem at all. They're only a
runtime thing. (Libraries are installed on demand at fixed paths when
constructing an environment, and garbage-collected later)

However, people do mostly use environments only when working on a
specific project which exists in the filesystem, so I don't think it's
too big of a compromise to start with.

>> Does this approach (putting project root directories into
>> file-name-handler-alist) sound like something that could be accepted
>> into core Emacs? If so I'll get started.
>
> Hard to tell.  Even the question is unclear to me.  The approach
> I suggest assumes you don't need/want to make changes to what I consider
> "core" and instead use existing hooks (in this case
> file-name-handler-alist).  So it could be distributed via GNU ELPA,
> for example.

Oh, no, I do want to get this into Emacs. Because the use of these kind
of environments is pretty common at this point, and it would be nice if
Emacs natively understood them in a generic way. Then, yes, ELPA
packages could build on that functionality to support specific
kinds of environments. Same with the new project and xref functionality.

> That seems to assume that an environment could be bound to a buffer
> rather than a subdirectory, IOW it wouldn't be chained to
> a subdirectory. Such environments seem to introduce further questions
> about how to determine which env to use for which buffer (IOW what UI
> to expose to the user).

If an environment is manually set for the current buffer, then use that
one for all commands run.
Otherwise, if a buffer is visiting a file in a project, check for a
manually set or automatically detected environment for that project,
then use that environment for all commands run.
Otherwise, use the global environment for all commands run.

That seems like a decent UI. But in the manually set case, it is tricky
to implement, because of the related buffers problem...

> You could set file-name-handler-alist buffer-locally, maybe?
>
> Still, the main issue is what to do about things that happen in other
> buffers but which the user would consider as related.

Yes, otherwise just buffer-local exec-path and process-environment would
work okay.

> You mean you want it distributed with Emacs?  Most new packages nowadays
> end up in GNU ELPA rather than in Emacs, unless they offer functionality
> needed by other packages that are distributed with Emacs.

Yes, I think this is important functionality that other packages
distributed with Emacs should be able to build on. Using virtualenvs is
very common for Python, for example, and Emacs should support that.

Also, since this is a headline feature of GNU Guix, it would be nice to
support it in GNU Emacs.

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

>> Hard to tell.  Even the question is unclear to me.  The approach
>> I suggest assumes you don't need/want to make changes to what I consider
>> "core" and instead use existing hooks (in this case
>> file-name-handler-alist).  So it could be distributed via GNU ELPA,
>> for example.
> Oh, no, I do want to get this into Emacs.  Because the use of these kind
> of environments is pretty common at this point, and it would be nice if
> Emacs natively understood them in a generic way.  Then, yes, ELPA
> packages could build on that functionality to support specific
> kinds of environments. Same with the new project and xref functionality.

I don't know enough about what you'd like to put in core to comment, then.

> That seems like a decent UI. But in the manually set case, it is tricky
> to implement, because of the related buffers problem...

My impression is that this will have to be fixed case-by-case.  There is
a precedent for that, which is the handling of buffer-local settings.
E.g. vc-diff is expected to obey the vc-diff-switches settings of the
"original buffer" rather than those of the buffer where the process
runs.  So we sometimes go through the trouble to read some variables
before switching to the destination buffer.  Environments would "simply"
fall into this case.  But yes, those issues will have to fixed
one-by-one.

> Also, since this is a headline feature of GNU Guix, it would be nice to
> support it in GNU Emacs.

I'm not maintainer any more so I don't need to make those decisions.
But if I were maintainer, I'd need to see the code before I could make
a decision.


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

>> Does this approach (putting project root directories into
>> file-name-handler-alist) sound like something that could be accepted
>> into core Emacs? If so I'll get started.

OK, I implemented a very rough prototype of this. It is
attached. environment.el is the API that would go into Emacs,
guix-environment.el provides a command to use an guix-provided
environment, and test-environment.el is just a sequence of expressions
to evaluate to demonstrate them. (Not tests)


[-- Attachment #2: environment.el --]
[-- Type: application/emacs-lisp, Size: 4302 bytes --]

[-- Attachment #3: guix-environment.el --]
[-- Type: application/emacs-lisp, Size: 2365 bytes --]

[-- Attachment #4: test-environment.el --]
[-- Type: application/emacs-lisp, Size: 720 bytes --]

[-- Attachment #5: Type: text/plain, Size: 3141 bytes --]


I have run into a problem though with the file-name-handler-alist
approach. I don't think it will work in the long run. To recap, I am
using file-name-handler-alist to make sure that all operations on files
that are descendants in the filesystem tree of some project directories,
are specially handled by my environment code.

The problem is that paths passed to find-file-name-handler, such as
default-directory, are not necessarily canonical paths. But
file-name-handler-alist is an list of (REGEX . HANDLER), so I am only
able to textually match against the input path. So if a path passed to
find-file-name-handler contains ~ or .. or ., it might be that it is in
fact a descendant of a project directory, but it won't be matched by the
regex, and so it won't get specially handled.

I don't think path resolution can be done with regular expressions,
so this approach will never really work.

>My impression is that this will have to be fixed case-by-case.  There is
>a precedent for that, which is the handling of buffer-local settings.
>E.g. vc-diff is expected to obey the vc-diff-switches settings of the
>"original buffer" rather than those of the buffer where the process
>runs.  So we sometimes go through the trouble to read some variables
>before switching to the destination buffer.  Environments would "simply"
>fall into this case.  But yes, those issues will have to fixed
>one-by-one.

That treatment of vc-diff-switches does look very similar to the correct
treatment of environments. Since there are other variables that need to
be treated in this way, beyond just environments, maybe a generic
solution is possible? Perhaps the "new kind of dynamic buffer-local
variable" I mentioned earlier:

>- Make exec-path and process-environment a different kind of
>  buffer-local dynamic variable
>
>  The behavior of this variable would be that when it is defined locally
>  in the current buffer, it behaves like a normal dynamic variable. And
>  when it is not defined locally in the current buffer, looking up the
>  variable will look up the variable dynamically instead of looking for
>  a global value set with setq-default.
>  This would mean that when M-x compile is invoked from some buffer, if
>  exec-path and process-environment are not buffer-local in the compile
>  buffer, those variables will be looked up dynamically, and found in
>  the buffer-locals of the buffer that was current at the time of the
>  M-x compile invocation.
>  I don't know if this is possible given the implementation of dynamic
>  variables in Emacs Lisp.

This could be used for vc-diff-switches too, and other variables like
it. Having now read more about how dynamic variables are implemented in
Emacs, I guess this would require a fair bit of work though.

If a generic solution like this isn't possible, I can start working on
fixing things case-by-case as soon as a good design is worked out.

>I'm not maintainer any more so I don't need to make those decisions.
>But if I were maintainer, I'd need to see the code before I could make
>a decision.

OK, I'll wait until John expresses his thoughts. (Possibly after the
release?)

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

> (require 'cl-lib)
> (require 'cl)

Please don't require both.

> (defvar environment-path-alist nil
>   "Alist of elements (PATH . ENV) for paths and special environments.")

In the GNU project (and in Emacs in particular), we use PATH to mean "a
list of directories" as in load-path, $PATH, $MANPATH.  So this should
probably be renamed to environment-directory-alist.

> (defun environment-lookup-path (path)
                                  ^^^^
Same here.

> The problem is that paths passed to find-file-name-handler, such as
> default-directory, are not necessarily canonical paths.

Yes, that's indeed a problem.  In practice it tends to work OK, tho,
because file and directory names to be constructed from each other, so
even if there are many different ways to name a particular directory,
there's typically only one that's used in a particular Emacs session.

And in any case it'd be good to fix the cases where we end up using
different names for the same dir (because they end up bumping into
similar problems).

> I don't think path resolution can be done with regular expressions,
> so this approach will never really work.

Feel free to use `advice-add' if you feel it's better.  If/when the code
gets integrated into Emacs, these kinds of problems can be resolved.

> OK, I'll wait until John expresses his thoughts. (Possibly after the
> release?)

You might also contact the authors of python's virtualenv support (and
anything else related) to get their opinions as supporting evidence for
the benefits of your suggestion.


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 22 Jul 2016 16:19:38 -0400
> 
> > The problem is that paths passed to find-file-name-handler, such as
> > default-directory, are not necessarily canonical paths.
> 
> Yes, that's indeed a problem.  In practice it tends to work OK, tho,
> because file and directory names to be constructed from each other, so
> even if there are many different ways to name a particular directory,
> there's typically only one that's used in a particular Emacs session.
> 
> And in any case it'd be good to fix the cases where we end up using
> different names for the same dir (because they end up bumping into
> similar problems).

Actually, I don't like to see any feature, let alone a core one, use
file-name-handler-alist for local files.  Lots of code in Emacs,
including primitives, really handle such files as remote, so the
results are likely to be incorrect and subtly broken.

> > OK, I'll wait until John expresses his thoughts. (Possibly after the
> > release?)
> 
> You might also contact the authors of python's virtualenv support (and
> anything else related) to get their opinions as supporting evidence for
> the benefits of your suggestion.

I always thought that features like this one should use and extent
the infrastructure in project.el.  And yet Dmitry is silent in this
discussion.  What am I missing?

Re: Managing environments (Python venv, guix environment, etc.)

[Dmitry Gutov]

Hi Eli,

I was going to comment, but this thread has run away quicker than I 
managed to compose a reply.

On 07/23/2016 10:10 AM, Eli Zaretskii wrote:
> Actually, I don't like to see any feature, let alone a core one, use
> file-name-handler-alist for local files.  Lots of code in Emacs,
> including primitives, really handle such files as remote, so the
> results are likely to be incorrect and subtly broken.

We do have file-remote-p, don't we? Maybe the code in question should be 
fixed.

> I always thought that features like this one should use and extent
> the infrastructure in project.el.  And yet Dmitry is silent in this
> discussion.  What am I missing?

I don't know exactly if project.el is the best place for it. Maybe we 
should see how well environment.el works as a separate feature, and if 
that makes sense, incorporate it after.

My main concern with this proposal is avoiding to have to purposefully 
modify "every Elisp package that runs processes". If every process call 
will pass through the environment dispatcher, that's bound to add some 
overhead. How will that affect the code that makes multiple process 
calls at once, such as VC? I'd like to see some numbers.

And if we base this feature on project.el right away, it will add a 
stricter requirement on the performance of `project-current', which is 
currently called at most once per user command. We could add some cache 
there, of course.

Alternatively, we can keep environment.el separate, and any advanced 
minor mode, such as EDE, would both hook into project.el and set up any 
necessary environment.el variables.

Re: Managing environments (Python venv, guix environment, etc.)

[Dmitry Gutov]

On 07/18/2016 10:13 PM, sbaugh@catern.com wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> That was my assumption: your "environments" correspond to
>> specific subdirectories in the file-system.
>
> Nope, not necessarily. Python virtualenvs do have an associated
> directory that contains the installed libraries for that
> environment. But guix environment only updates environment variables to
> point to the locations of already existing libraries, and so guix
> environments don't exist in the filesystem at all. They're only a
> runtime thing. (Libraries are installed on demand at fixed paths when
> constructing an environment, and garbage-collected later)

How will the guix environment to use be determined, then?

If it's strictly "prompt the user", that might be incompatible with the 
way project.el determines the current project.

> Oh, no, I do want to get this into Emacs. Because the use of these kind
> of environments is pretty common at this point, and it would be nice if
> Emacs natively understood them in a generic way. Then, yes, ELPA
> packages could build on that functionality to support specific
> kinds of environments. Same with the new project and xref functionality.

We can't move xref to ELPA because Emacs's default key bindings use it. 
And project.el, because it's used by xref.

> Yes, I think this is important functionality that other packages
> distributed with Emacs should be able to build on. Using virtualenvs is
> very common for Python, for example, and Emacs should support that.

I don't see why virtualenvs support can't be distributed via GNU ELPA, 
too. Anyway, as soon as python.el decides to depend on it, we could move 
environment.el to the core.

> Also, since this is a headline feature of GNU Guix, it would be nice to
> support it in GNU Emacs.

I'm sure people working on that distribution can pre-install a few ELPA 
packages.

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Dmitry Gutov <dgutov@yandex.ru> writes:

> My main concern with this proposal is avoiding to have to purposefully
> modify "every Elisp package that runs processes". If every process
> call will pass through the environment dispatcher,

Just a note: Ideally, Elisp code would also (somehow) go through the
environment dispatcher, and see the exec-path and process-environment of
the appropriate environment. There is surely Elisp code makes direct use
of exec-path/process-environment and it should behave the same as
external processes.

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Dmitry Gutov <dgutov@yandex.ru> writes:
> How will the guix environment to use be determined, then?
>
> If it's strictly "prompt the user", that might be incompatible with
> the way project.el determines the current project.

That's something we can determine. I was thinking a dir-local would be
ideal.

>> Yes, I think this is important functionality that other packages
>> distributed with Emacs should be able to build on. Using virtualenvs is
>> very common for Python, for example, and Emacs should support that.
>
> I don't see why virtualenvs support can't be distributed via GNU ELPA,
> too. Anyway, as soon as python.el decides to depend on it, we could
> move environment.el to the core.

If proper virtualenvs support turns out to require changes to core
Emacs, it can't be distributed (solely) via GNU ELPA.

It's possible it might even require hacking on the C implementation of
the process primitives to do properly. (which I'm perfectly happy to do)

>> Also, since this is a headline feature of GNU Guix, it would be nice to
>> support it in GNU Emacs.
>
> I'm sure people working on that distribution can pre-install a few
> ELPA packages.

Sure, fair enough.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 24 Jul 2016 06:26:41 +0300
> 
> On 07/23/2016 10:10 AM, Eli Zaretskii wrote:
> > Actually, I don't like to see any feature, let alone a core one, use
> > file-name-handler-alist for local files.  Lots of code in Emacs,
> > including primitives, really handle such files as remote, so the
> > results are likely to be incorrect and subtly broken.
> 
> We do have file-remote-p, don't we? Maybe the code in question should be 
> fixed.

No, I meant that we assume these are remote files without testing,
e.g. that primitives which support local files, like expand-file-name,
don't DTRT for them.  The code is written with that assumption in
mind.

> > I always thought that features like this one should use and extent
> > the infrastructure in project.el.  And yet Dmitry is silent in this
> > discussion.  What am I missing?
> 
> I don't know exactly if project.el is the best place for it. Maybe we 
> should see how well environment.el works as a separate feature, and if 
> that makes sense, incorporate it after.

It would sound strange to me that project.el fails to support
essentially the first serious client of its intended audience.

> My main concern with this proposal is avoiding to have to purposefully 
> modify "every Elisp package that runs processes".

Right.  Which is why I thought we should come up with some reasonable
way of supporting that without infecting "every package".

> If every process call will pass through the environment dispatcher,
> that's bound to add some overhead. How will that affect the code
> that makes multiple process calls at once, such as VC? I'd like to
> see some numbers.
> 
> And if we base this feature on project.el right away, it will add a 
> stricter requirement on the performance of `project-current', which is 
> currently called at most once per user command. We could add some cache 
> there, of course.
> 
> Alternatively, we can keep environment.el separate, and any advanced 
> minor mode, such as EDE, would both hook into project.el and set up any 
> necessary environment.el variables.

I actually am not sure what exactly is required by environment.el.
This discussion started by talking about environment variables, but
somehow we are now talking about such core functionality as load-path
and exec-path.  The latter is a far cry from the former, IMO.  I'm not
sure how we made that leap in the discussion.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Sun, 24 Jul 2016 04:25:03 -0400
> 
> Ideally, Elisp code would also (somehow) go through the environment
> dispatcher, and see the exec-path and process-environment of the
> appropriate environment.

I don't see why this would be needed.  Emacs can load Lisp packages
and run programs even if they are not on load-path/exec-path.  Why do
we need to tweak these lists which are so central to Emacs core?

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: sbaugh@catern.com
>> Date: Sun, 24 Jul 2016 04:25:03 -0400
>> 
>> Ideally, Elisp code would also (somehow) go through the environment
>> dispatcher, and see the exec-path and process-environment of the
>> appropriate environment.
>
> I don't see why this would be needed.  Emacs can load Lisp packages
> and run programs even if they are not on load-path/exec-path.  Why do
> we need to tweak these lists which are so central to Emacs core?

Maybe I was unclear. I don't think load-path should be tweaked by
environment.el (even though, since it can be initialized by
EMACSLOADPATH, and that can change in different environments, you could
conceivably argue that it should be) since it is so central to
Emacs.

But there are many Elisp functions that use exec-path. For example,
browse-url uses executable-find (which uses exec-path) to determine
which browsers are available. If exec-path (or maybe just all its users
in core) is not tweaked by environment.el, but the process functions
are, these functions like browse-url will behave in a broken way.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Sun, 24 Jul 2016 13:45:21 -0400
> 
> But there are many Elisp functions that use exec-path. For example,
> browse-url uses executable-find (which uses exec-path) to determine
> which browsers are available. If exec-path (or maybe just all its users
> in core) is not tweaked by environment.el, but the process functions
> are, these functions like browse-url will behave in a broken way.

If the problem is only with executable-find, we could provide a
solution for that one alone.  Starting with the simplest one: if the
specific environment knows in which directory to look for the
browsers, it could simply prepend the directory, making the browser
file name an absolute one.  Or use locate-file, which accepts path as
an argument.  Or something.

Tweaking exec-path will have a much more dramatic and wide effect, so
we should try to avoid that, IMO.

Re: Managing environments (Python venv, guix environment, etc.)

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
> Tweaking exec-path will have a much more dramatic and wide effect, so
> we should try to avoid that, IMO.

Why would the effect be so significant? Is exec-path used internally
somehow in some confusing and tricky way? If it's just Elisp using
exec-path, it doesn't seem like messing with exec-path would be that subtle or difficult.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@catern.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 24 Jul 2016 15:05:52 -0400
> 
> Why would the effect be so significant?

For the same reason as load-path.

> Is exec-path used internally somehow in some confusing and tricky
> way?

It's used in the same messy way as load-path, just for looking for
different kind of files.  And it is as pervasive as load-path.

> If it's just Elisp using exec-path, it doesn't seem like messing
> with exec-path would be that subtle or difficult.

Not just Elisp, C primitives as well.  And even Elisp can be messy if
it's in preloaded packages like files.el, subr.el and simple.el.

IOW, I simply fail to see how we will be able to avoid disrupting the
most basic features if we modify exec-path.  Even visiting a file can
fire up a subprocess -- how do we make sure the right program for that
will still be found, if we let some project's environment mess with
exec-path behind the user's back?  And what about spellers (e.g., for
ispell-check-comments-and-strings)?  Etc. etc.

Once again, why not use locate-file?  CEDET and EDE already do, and I
assume for good reasons.

More generally, why would a project want to modify exec-path? to find
which programs?  Is it only the compiler/interpreter of the
programming language used by the project, or something else as well,
and why?  E.g., I don't really understand your example with browse-url
-- why would I want to change a browser as function of the project I'm
working on?

Re: Managing environments (Python venv, guix environment, etc.)

[Dmitry Gutov]

On 07/24/2016 11:25 AM, sbaugh@catern.com wrote:

> Just a note: Ideally, Elisp code would also (somehow) go through the
> environment dispatcher, and see the exec-path and process-environment of
> the appropriate environment. There is surely Elisp code makes direct use
> of exec-path/process-environment and it should behave the same as
> external processes.

Like Eli said, this seems unwise. But it hints at a logical problem: if 
start-process and friends are simply documented to use exec-path and 
process-environment, but use their modified values in some 
circumstances, someone trying to debug this could have a big problem.

Especially if the feature is implemented not using nadvice, but with 
file handlers.

So I think we should either fall back to the idea of using buffer-local 
values and make it work somehow, or have environment.el provide its own 
versions of call-process, etc, that use the modified environments.

The latter option will require modifying all appropriate callers, but 
seems like the cleanest approach to me.

Alternatively, the docstrings of all process-interacting functions 
should be updated to mention environments.el, and at that point it would 
make sense to just integrate it into the core. Maybe the result would 
look similar to what you said, with all process primitives using the new 
functions (called e.g. environment-exec-path and 
environment-process-environment).

Re: Managing environments (Python venv, guix environment, etc.)

[Dmitry Gutov]

On 07/24/2016 11:25 AM, sbaugh@catern.com wrote:

>> If it's strictly "prompt the user", that might be incompatible with
>> the way project.el determines the current project.
>
> That's something we can determine. I was thinking a dir-local would be
> ideal.

If it's going to be strictly directory-dependent, project.el integration 
could work. But then, I don't know if you want to tie environment 
detection to particular project implementations, such as vc-project 
(which is only used when there's a VCS repository).

> If proper virtualenvs support turns out to require changes to core
> Emacs, it can't be distributed (solely) via GNU ELPA.
>
> It's possible it might even require hacking on the C implementation of
> the process primitives to do properly. (which I'm perfectly happy to do)

Guess we'll have to see.

Re: Managing environments (Python venv, guix environment, etc.)

[Dmitry Gutov]

On 07/24/2016 05:24 PM, Eli Zaretskii wrote:

>> We do have file-remote-p, don't we? Maybe the code in question should be
>> fixed.
>
> No, I meant that we assume these are remote files without testing,
> e.g. that primitives which support local files, like expand-file-name,
> don't DTRT for them.  The code is written with that assumption in
> mind.

Couldn't that be fixed? Anyway, I think using file-name-handler-alist is 
not a good idea because it'll make for worse debugging experience.

>>> I always thought that features like this one should use and extent
>>> the infrastructure in project.el.  And yet Dmitry is silent in this
>>> discussion.  What am I missing?
>>
>> I don't know exactly if project.el is the best place for it. Maybe we
>> should see how well environment.el works as a separate feature, and if
>> that makes sense, incorporate it after.
>
> It would sound strange to me that project.el fails to support
> essentially the first serious client of its intended audience.

Maybe project.el will end up a misnomer (but I don't have a better name 
that is not too long). Essentially, if we tie the new feature to 
projects, that will likely limit its usability. Or at least if we do 
that in the obvious way I can think of.

Consider: if we just add a new defgeneric (or two) to project.el, any 
specialized environment implementation would either have to provide its 
own full project implementation (i.e. it'll be unable to use 
vc-project), or it'll have to define specialized implementations of 
these methods just for certain types of projects it knows about.

That's not to say that we couldn't add new defdenerics to project.el for 
the purpose of overriding the current environment setup... but I'd like 
to see some real use case for that first.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 25 Jul 2016 02:04:42 +0300
> 
> On 07/24/2016 11:25 AM, sbaugh@catern.com wrote:
> 
> > Just a note: Ideally, Elisp code would also (somehow) go through the
> > environment dispatcher, and see the exec-path and process-environment of
> > the appropriate environment. There is surely Elisp code makes direct use
> > of exec-path/process-environment and it should behave the same as
> > external processes.
> 
> Like Eli said, this seems unwise. But it hints at a logical problem: if 
> start-process and friends are simply documented to use exec-path and 
> process-environment, but use their modified values in some 
> circumstances, someone trying to debug this could have a big problem.

One possible idea is to identify the programs that an environment
would like to depend on it (like the compiler), define a variable for
each one of them (compiler-program etc.), and then modify the
corresponding commands to use that variable, if bound to non-nil, in
preference to exec-path search.

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Eli Zaretskii <eliz@gnu.org> writes:
> IOW, I simply fail to see how we will be able to avoid disrupting the
> most basic features if we modify exec-path.  Even visiting a file can
> fire up a subprocess -- how do we make sure the right program for that
> will still be found, if we let some project's environment mess with
> exec-path behind the user's back?  And what about spellers (e.g., for
> ispell-check-comments-and-strings)?  Etc. etc.

I don't think that would be an issue. At the moment, the primary place
to use these kind of special environments is a Unix shell. If basic
tools like coreutils weren't in the environment, the environment would
already be useless for the user. And the only way to get some kind of
useless environment like that would be to explicitly configure it; I'm
inclined to call that user error. Also: These special environments
typically just prepend new directories to PATH and other vars,
"inheriting" the binaries that were in the user's pre-existing
environment, and just overriding some, so in the typical case things
will certainly work fine.

Also note: If a binary from outside some project-specific environment is
run within that environment, it might break. (Maybe you preload some
crazy library within the project-specific environment, and have a set of
project-specific binaries that know how to cope with that.)

So if we don't modify exec-path to be project-specific, then anything
that wants to run a process inside the project-specific environment
would need to be explicitly modified to search the project-specific
path, to avoid running outside binaries inside the special
environment.

> Once again, why not use locate-file?  CEDET and EDE already do, and I
> assume for good reasons.

Yes, a new program that wants to search per-environment search paths
could use locate-file. But I still hope for not needing to modify things
to be environment-aware.

> More generally, why would a project want to modify exec-path? to find
> which programs?  Is it only the compiler/interpreter of the
> programming language used by the project, or something else as well,
> and why?  E.g., I don't really understand your example with browse-url
> -- why would I want to change a browser as function of the project I'm
> working on?

Well, also the build system and the debugger. And maybe if you're
developing a browser, you'd want browse-url to pick up the version
compiled in the project-specific environment when you're working on it.

In general I think it's entirely reasonable to want to be able to run
project-specific version of developer tools. The compiler/interpreter is
the most obvious project-specific program, but there are benefits to
supporting running every developer tool inside the project-specific
environment.

Here are three benefits that I see for running everything within the
project-specific environment by default:
- This works identically (and therefore compatibly) to the Unix shell
inside a special environment.
- Any other project-specific developer tools that we don't know about or
don't yet exist, will be automatically supported.
- And most importantly: all the myriad programming modes for Emacs with
their many custom methods for running their compiler/interpreter, will
automatically support running their compilers/interpreters in an
environment-aware way.

However I admit that if the user wants to run a development program from
outside the project-specific environment (or even a development program
from another environment) it could become a little more tricky...

And the conceptually easiest way is still just updating Elisp commands
(such as compile or shell-command) one by one to be environment-aware.

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> Like Eli said, this seems unwise. But it hints at a logical problem: if 
>> start-process and friends are simply documented to use exec-path and 
>> process-environment, but use their modified values in some 
>> circumstances, someone trying to debug this could have a big problem.
>
> One possible idea is to identify the programs that an environment
> would like to depend on it (like the compiler), define a variable for
> each one of them (compiler-program etc.), and then modify the
> corresponding commands to use that variable, if bound to non-nil, in
> preference to exec-path search.

The environment also matters, so you'd need to have a
compiler-process-environment too.

Note: The compile command already makes use of a compilation-environment
variable.

That would work, but again, it means updating the entire world to
support this. Also, if we're going to update the world: Perhaps just add
buffer-local variables "local-exec-path" and
"local-process-environment", and flags like
"compile-use-local-environment" which if non-nil cause the relevant
command, when invoked, to use the local-exec-path and
local-process-environment variables from (current-buffer). That seems
more natural to me.

Re: Managing environments (Python venv, guix environment, etc.)

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

> So I think we should either fall back to the idea of using
> buffer-local values and make it work somehow, or have environment.el
> provide its own versions of call-process, etc, that use the modified
> environments.
>
> The latter option will require modifying all appropriate callers, but
> seems like the cleanest approach to me.

Conceptually, you kick off all packages then which support remote
processes. They use proess-file instead of call-process, etc.

I don't know whether environment.el plans to support this case. But all
callers, which would change such a way, don't support remote processes
anymore.

Best regards, Michael.

Re: Managing environments (Python venv, guix environment, etc.)

[Dmitry Gutov]

On 07/25/2016 10:07 AM, Michael Albinus wrote:

> Conceptually, you kick off all packages then which support remote
> processes. They use proess-file instead of call-process, etc.

Why? There would be a wrapper for process-file, too.

Re: Managing environments (Python venv, guix environment, etc.)

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/25/2016 10:07 AM, Michael Albinus wrote:
>
>> Conceptually, you kick off all packages then which support remote
>> processes. They use proess-file instead of call-process, etc.
>
> Why? There would be a wrapper for process-file, too.

And for start-file-process. And for shell-command. And for ...

Doesn't sound applicable to me. Do you believe all callers will adapt in
finite time? It took years to let them move to process-file and friends,
and there are still bug reports about packages which didn't care.

And now we would ask to change again, with all the pain of backward
compatibility.

Best regards, Michael.

Re: Managing environments (Python venv, guix environment, etc.)

[Dmitry Gutov]

On 07/25/2016 04:03 PM, Michael Albinus wrote:
> Do you believe all callers will adapt in
> finite time? It took years to let them move to process-file and friends,
> and there are still bug reports about packages which didn't care.

I believe it is possible, yes. But you're welcome to discuss alternative 
proposals.

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

> And now we would ask to change again, with all the pain of backward
> compatibility.

Agreed.  We should make the change apply to existing uses of
process-file and start-file-process.  Maybe it's OK if it doesn't apply
to call-process and start-process, OTOH.


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 25 Jul 2016 03:05:37 +0300
> 
> > No, I meant that we assume these are remote files without testing,
> > e.g. that primitives which support local files, like expand-file-name,
> > don't DTRT for them.  The code is written with that assumption in
> > mind.
> 
> Couldn't that be fixed?

We try, but it isn't always easy.  Here's one example, from
file-truename:

      (let ((handler (find-file-name-handler filename 'file-truename)))
	;; For file name that has a special handler, call handler.
	;; This is so that ange-ftp can save time by doing a no-op.
	(if handler
	    (setq filename (funcall handler 'file-truename filename)
		  done t)
	  (let ((dir (or (file-name-directory filename) default-directory))
		target dirfile)
	    ;; Get the truename of the directory.
	    (setq dirfile (directory-file-name dir))
	    ;; If these are equal, we have the (or a) root directory.
	    (or (string= dir dirfile)
		(and (memq system-type '(windows-nt ms-dos cygwin nacl))
		     (eq (compare-strings dir 0 nil dirfile 0 nil t) t))

As you see, we only apply the MS-Windows specific treatment to files
which don't have a handler, on the assumption that those which do are
not subject to rules that govern local files on MS-Windows.

We have several similar examples elsewhere.

> Consider: if we just add a new defgeneric (or two) to project.el, any 
> specialized environment implementation would either have to provide its 
> own full project implementation (i.e. it'll be unable to use 
> vc-project), or it'll have to define specialized implementations of 
> these methods just for certain types of projects it knows about.

Why can't the default implementation behave as if "environments"
didn't exist?  Then the behavior will be exactly as it is now, no?  Or
am I missing something?

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Mon, 25 Jul 2016 00:50:34 -0400
> 
> > IOW, I simply fail to see how we will be able to avoid disrupting the
> > most basic features if we modify exec-path.  Even visiting a file can
> > fire up a subprocess -- how do we make sure the right program for that
> > will still be found, if we let some project's environment mess with
> > exec-path behind the user's back?  And what about spellers (e.g., for
> > ispell-check-comments-and-strings)?  Etc. etc.
> 
> I don't think that would be an issue. At the moment, the primary place
> to use these kind of special environments is a Unix shell. If basic
> tools like coreutils weren't in the environment, the environment would
> already be useless for the user. And the only way to get some kind of
> useless environment like that would be to explicitly configure it; I'm
> inclined to call that user error. Also: These special environments
> typically just prepend new directories to PATH and other vars,
> "inheriting" the binaries that were in the user's pre-existing
> environment, and just overriding some, so in the typical case things
> will certainly work fine.
> 
> Also note: If a binary from outside some project-specific environment is
> run within that environment, it might break. (Maybe you preload some
> crazy library within the project-specific environment, and have a set of
> project-specific binaries that know how to cope with that.)

See, the above 2 paragraphs contradict each other.  The second one is
precisely the kind of breakage I had in mind and feared of, and I see
now that we are in violent agreement regarding such a possibility.

Prepending directories to PATH doesn't solve these problems.  E.g., if
the prepended directory holds a version of gzip that is "broken" in
the above-mentioned meaning in the environment of a project, then
Emacs will be unable to visit gzip-compressed files, such as Info
manuals and its own Lisp sources.

I think we need to avoid this danger.

> So if we don't modify exec-path to be project-specific, then anything
> that wants to run a process inside the project-specific environment
> would need to be explicitly modified to search the project-specific
> path, to avoid running outside binaries inside the special
> environment.

I think the suggestion to define a series of variables, which, if
non-nil, will override the exec-path search by the relevant
primitives, avoids this issue, because these variables will be handled
on the infrastructure level.

> > Once again, why not use locate-file?  CEDET and EDE already do, and I
> > assume for good reasons.
> 
> Yes, a new program that wants to search per-environment search paths
> could use locate-file. But I still hope for not needing to modify things
> to be environment-aware.

We cannot avoid some modifications anyway.  You are talking about
changing Emacs on a very low and delicate level.

> > More generally, why would a project want to modify exec-path? to find
> > which programs?  Is it only the compiler/interpreter of the
> > programming language used by the project, or something else as well,
> > and why?  E.g., I don't really understand your example with browse-url
> > -- why would I want to change a browser as function of the project I'm
> > working on?
> [...]
> maybe if you're developing a browser, you'd want browse-url to pick
> up the version compiled in the project-specific environment when
> you're working on it.

browse-url already has browse-url-*-program and
browse-url-default-browser variables that should be enough to cater to
this use case.  A project can simply customize their values, and make
them buffer-local if needed.

> In general I think it's entirely reasonable to want to be able to run
> project-specific version of developer tools. The compiler/interpreter is
> the most obvious project-specific program, but there are benefits to
> supporting running every developer tool inside the project-specific
> environment.

But exec-path is not just for development tools.  It's for _any_
program Emacs might need to run.  Thus changing it has an effect that
is much more wide and prominent.

> Here are three benefits that I see for running everything within the
> project-specific environment by default:
> - This works identically (and therefore compatibly) to the Unix shell
> inside a special environment.

Actually, many development environments are configured by pointing at
specific absolute file names of the tools, such that any changes in
PATH don't affect them.

> - Any other project-specific developer tools that we don't know about or
> don't yet exist, will be automatically supported.
> - And most importantly: all the myriad programming modes for Emacs with
> their many custom methods for running their compiler/interpreter, will
> automatically support running their compilers/interpreters in an
> environment-aware way.

I think the downsides of this are too serious to ignore.  Like this
one, for example:

> However I admit that if the user wants to run a development program from
> outside the project-specific environment (or even a development program
> from another environment) it could become a little more tricky...

More generally, running _any_ program that doesn't belong to the
environment might subtly break.

> And the conceptually easiest way is still just updating Elisp commands
> (such as compile or shell-command) one by one to be environment-aware.

The challenge is to make changes to the infrastructure that cover the
80% of use cases in environment-aware development, without making the
cure worse than the disease.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Mon, 25 Jul 2016 01:01:49 -0400
> 
> > One possible idea is to identify the programs that an environment
> > would like to depend on it (like the compiler), define a variable for
> > each one of them (compiler-program etc.), and then modify the
> > corresponding commands to use that variable, if bound to non-nil, in
> > preference to exec-path search.
> 
> The environment also matters, so you'd need to have a
> compiler-process-environment too.

Sometimes, yes.  But this is definitely for the project to set up.

> Note: The compile command already makes use of a compilation-environment
> variable.

Then we already do what you want, right?

> That would work, but again, it means updating the entire world to
> support this.

No, it means we need to make specific changes in specific primitives
and infrastructure functions.  Those which currently only look at
exec-path, and are involved in invoking development tools for which we
will want to provide this support.  E.g., "M-x compile" will be able
to invoke the compiler specified by the environment, "M-x gdb" will be
able to invoke the debugger specified by the environment, etc.

We could also have wrappers like call-compiler, call-debugger, etc.

> Also, if we're going to update the world: Perhaps just add
> buffer-local variables "local-exec-path" and
> "local-process-environment", and flags like
> "compile-use-local-environment" which if non-nil cause the relevant
> command, when invoked, to use the local-exec-path and
> local-process-environment variables from (current-buffer). That seems
> more natural to me.

But that's exactly the danger I think we should try to avoid: a Lisp
application doesn't have good control of when buffer-local bindings
will be in effect, because Emacs internals feel at liberty to
temporarily switch to any buffer when they feel like it.  These
buffer-local bindings will bite you when you don't expect.

That's why I think the environment should be only allowed to override
file names of a specific limited set of tools, not the entire
exec-path.

Re: Managing environments (Python venv, guix environment, etc.)

[Dmitry Gutov]

On 07/25/2016 08:00 PM, Eli Zaretskii wrote:

>> Couldn't that be fixed?
>
> We try, but it isn't always easy.  Here's one example, from
> file-truename:
>
>       (let ((handler (find-file-name-handler filename 'file-truename)))
> 	;; For file name that has a special handler, call handler.
> 	;; This is so that ange-ftp can save time by doing a no-op.
> 	(if handler
> 	    (setq filename (funcall handler 'file-truename filename)
> 		  done t)
> 	  (let ((dir (or (file-name-directory filename) default-directory))
> 		target dirfile)
> 	    ;; Get the truename of the directory.
> 	    (setq dirfile (directory-file-name dir))
> 	    ;; If these are equal, we have the (or a) root directory.
> 	    (or (string= dir dirfile)
> 		(and (memq system-type '(windows-nt ms-dos cygwin nacl))
> 		     (eq (compare-strings dir 0 nil dirfile 0 nil t) t))
>
> As you see, we only apply the MS-Windows specific treatment to files
> which don't have a handler, on the assumption that those which do are
> not subject to rules that govern local files on MS-Windows.

Maybe default handler implementation could bind the handlers alist to 
nil and call file-truename again.

>> Consider: if we just add a new defgeneric (or two) to project.el, any
>> specialized environment implementation would either have to provide its
>> own full project implementation (i.e. it'll be unable to use
>> vc-project), or it'll have to define specialized implementations of
>> these methods just for certain types of projects it knows about.
>
> Why can't the default implementation behave as if "environments"
> didn't exist?

Then those projects will miss out on Guix, and similar, integrations.

> Then the behavior will be exactly as it is now, no?

It would. That doesn't sound optimal to me.

My point is, whether a project uses Guix, or a similar tool, should be 
in vast majority of cases orthogonal to the type of the project 
(language used, frameworks, configuration file format, etc).

So far I'm inclined to believe that there should be about as many 
project implementations as there are project file formats (e.g. Maven, 
Ant, Autotools, Gradle configs, specialized formats used by other 
editors, etc). There's no reason to multiply that by the number of 
environment types we want to support.

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

> Like Eli said, this seems unwise. But it hints at a logical problem: if
> start-process and friends are simply documented to use exec-path and
> process-environment, but use their modified values in some circumstances,
> someone trying to debug this could have a big problem.

process-file and start-file-process are documented to obey
file-name-handlers-alist, so their callers should expect
directory-dependent behavior, including potentially not blindly obeying
exec-path.


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

>> As you see, we only apply the MS-Windows specific treatment to files
>> which don't have a handler, on the assumption that those which do are
>> not subject to rules that govern local files on MS-Windows.
> Maybe default handler implementation could bind the handlers alist to nil
> and call file-truename again.

And actually that's exactly how file-name-handlers are expected to
behave in general.


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Maybe default handler implementation could bind the handlers alist to nil
>> and call file-truename again.
>
> And actually that's exactly how file-name-handlers are expected to
> behave in general.

Not really. They bind inhibit-file-name-handlers and
inhibit-file-name-operation.

>         Stefan

Best regards, Michael.

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

>>> Maybe default handler implementation could bind the handlers alist to nil
>>> and call file-truename again.
>> And actually that's exactly how file-name-handlers are expected to
>> behave in general.
> Not really. They bind inhibit-file-name-handlers and
> inhibit-file-name-operation.

Yes, they do it in a more parsimonious way, but the result is the same:
if they don't have a special implementation of file-truename, they
should call back file-truename in such a way that the file-name-handler
won't be re-triggered.


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 26 Jul 2016 05:08:31 +0300
> 
> >> Consider: if we just add a new defgeneric (or two) to project.el, any
> >> specialized environment implementation would either have to provide its
> >> own full project implementation (i.e. it'll be unable to use
> >> vc-project), or it'll have to define specialized implementations of
> >> these methods just for certain types of projects it knows about.
> >
> > Why can't the default implementation behave as if "environments"
> > didn't exist?
> 
> Then those projects will miss out on Guix, and similar, integrations.

I don't understand why.  Can you elaborate?

> My point is, whether a project uses Guix, or a similar tool, should be 
> in vast majority of cases orthogonal to the type of the project 
> (language used, frameworks, configuration file format, etc).
> 
> So far I'm inclined to believe that there should be about as many 
> project implementations as there are project file formats (e.g. Maven, 
> Ant, Autotools, Gradle configs, specialized formats used by other 
> editors, etc). There's no reason to multiply that by the number of 
> environment types we want to support.

Maybe I misunderstand, but I thought requirements for supporting
Maven, Ant, etc., are indeed orthogonal to those imposed by Guix etc.,
so no multiplication should be necessary.  If not, then how do people
use these combinations outside of Emacs -- do they also create all the
combinations?

Maybe we should discuss what is and what isn't part of a "project".

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Michael Albinus <michael.albinus@gmx.de> writes:
> And for start-file-process. And for shell-command. And for ...
>
> Doesn't sound applicable to me. Do you believe all callers will adapt in
> finite time? It took years to let them move to process-file and friends,
> and there are still bug reports about packages which didn't care.
>
> And now we would ask to change again, with all the pain of backward
> compatibility.
>
> Best regards, Michael.

I do think it's best to use the same mechanism (process-file) that is
used to support remote access.

Really, remote access is very similar to entering a different local
environment. They're different contexts in which to execute
commands.

And consider: Much of the work in determining what is sensible to
execute in the default context, and what is sensible to execute in the
remote/custom context, has already been done for remote access. ispell
runs the spell-checker in the default context; M-x compile runs the
compile in the custom context; and so on. I think this solves the issues
Eli brought up in another mail.

Furthermore, I think it would be beneficial for the experience of using
a different local environment to be very similar to the experience of
using a remote server, given their similarity.

Perhaps instead of inserting regular filenames into
file-name-handler-alist, environment.el should use filenames with a
custom prefix (like "/?" or something) and add a new handler for
filenames with that prefix. The prefix can be followed by the name of
the environment, then some separator character, then a real local
filename. (Much like TRAMP. Unlike TRAMP, file access will still be done
directly on the local filesystem.)

Pros:
- Resolves the issue with canonical file names
- Allows a file to be accessed from multiple environments freely at the
  same time
- Powerful and full-featured way to implement this - just as powerful as
  TRAMP, which is certainly powerful enough.
- Maybe the easiest way to get this to actually work, since it reuses
  all the work done on getting things to use process-file

Cons:
- The user interface might be tricky and cumbersome and unnatural.
  On the other hand, TRAMP is hugely useful, even though it uses exactly
  this interface.

I'm thinking that this is the best way. Thoughts?

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Eli Zaretskii <eliz@gnu.org> writes:
> See, the above 2 paragraphs contradict each other.  The second one is
> precisely the kind of breakage I had in mind and feared of, and I see
> now that we are in violent agreement regarding such a possibility.
>
> Prepending directories to PATH doesn't solve these problems.  E.g., if
> the prepended directory holds a version of gzip that is "broken" in
> the above-mentioned meaning in the environment of a project, then
> Emacs will be unable to visit gzip-compressed files, such as Info
> manuals and its own Lisp sources.
>
> I think we need to avoid this danger.

That is all true. I was of the opinion that it was not quite so
dangerous because this method worked fine most of the time for
shells. But I agree now - we should be careful, and abide by higher
standards.

And I think the suggestion I made in the email I sent just before this,
to use a new kind of magic path, and thereby reuse all the process-file
infrastructure, is the best way to be careful in this matter.

Only the programs that can and should be run remotely, use process-file;
and I contend that those programs are exactly and exclusively the things
we want to run inside a custom environment.

Re: Managing environments (Python venv, guix environment, etc.)

[Michael Albinus]

sbaugh@catern.com writes:

> Perhaps instead of inserting regular filenames into
> file-name-handler-alist, environment.el should use filenames with a
> custom prefix (like "/?" or something) and add a new handler for
> filenames with that prefix. The prefix can be followed by the name of
> the environment, then some separator character, then a real local
> filename. (Much like TRAMP. Unlike TRAMP, file access will still be done
> directly on the local filesystem.)
>
> Pros:
> - Resolves the issue with canonical file names
> - Allows a file to be accessed from multiple environments freely at the
>   same time
> - Powerful and full-featured way to implement this - just as powerful as
>   TRAMP, which is certainly powerful enough.
> - Maybe the easiest way to get this to actually work, since it reuses
>   all the work done on getting things to use process-file

I doubt that you could reuse much of Tramp's `process-file'
implementations. They use pretty much Tramp internal functions.

Btw, this approach could still be applied to remote files. File name
handler calls can be cascaded.

"/?my-environment?~/.emacs" would work on a local file, and
"/?my-environment?~/sudo::.emacs" would work on a remote file.

Saying this does not mean that I agree with your proposal. Being
undecided.

Best regards, Michael.

Re: Managing environments (Python venv, guix environment, etc.)

[Spencer Baugh]

[-- Attachment #1: Type: text/html, Size: 965 bytes --]

Re: Managing environments (Python venv, guix environment, etc.)

[Michael Albinus]

Spencer Baugh <sbaugh@catern.com> writes:

>> I doubt that you could reuse much of Tramp's `process-file'
>> implementations. They use pretty much Tramp internal functions.
>
> Not sure what you mean. I didn't hope to reuse TRAMP implementation,
> just to use process-file which so many other Elisp programs have
> changed their implementation to use.

Ahh, I misunderstood you.

Best regards, Michael.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Wed, 27 Jul 2016 20:02:15 -0400
> 
> I do think it's best to use the same mechanism (process-file) that is
> used to support remote access.
> 
> Really, remote access is very similar to entering a different local
> environment. They're different contexts in which to execute
> commands.
> 
> And consider: Much of the work in determining what is sensible to
> execute in the default context, and what is sensible to execute in the
> remote/custom context, has already been done for remote access. ispell
> runs the spell-checker in the default context; M-x compile runs the
> compile in the custom context; and so on. I think this solves the issues
> Eli brought up in another mail.

It may seem to you that remote access is similar to a different
environment, but in fact they are very different.  Environment is not
a property of a file name, it is a property of a project.  Moreover,
past attempts I know about to use file handlers for processing local
files turned out to have subtle misfeatures.  Even more importantly,
the _conceptual_ difference is huge, and it's bound to bite us in the
long run, because people who work on the file-name handlers has the
remote-access concept on their mind, and will write code that caters
to those use cases, so we will sooner or later end up with code that
subtly breaks in the environment use cases.

So I'm very much opposed to using file-name handlers for supporting
environments.

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Eli Zaretskii <eliz@gnu.org> writes:
> It may seem to you that remote access is similar to a different
> environment, but in fact they are very different.  Environment is not
> a property of a file name, it is a property of a project.

Environment is not a property of a project either, it's something else
entirely. A project doesn't necessarily have an environment associated
with it, and an environment isn't necessarily asociated with a
project. Treating it as a property of a file name or a property of a
project are just two possible interfaces/implementations; both will
limit the user.

Remote access was turned into a property of a file name by embedding the
remote host identifier into the file name. If we embed the environment
into the file name, then the environment also becomes a property of a
file name.

> Moreover, past attempts I know about to use file handlers for
> processing local files turned out to have subtle misfeatures.

I would be interested to see previous failed attempts to use file
handlers on local files, can you point me to an example?

AFAIK jka-compr.el works fine.

> Even more importantly, the _conceptual_ difference is huge, and it's
> bound to bite us in the long run, because people who work on the
> file-name handlers has the remote-access concept on their mind, and
> will write code that caters to those use cases, so we will sooner or
> later end up with code that subtly breaks in the environment use
> cases.

That is a very fair point, but I think the current state of file-name
handlers is functional for the environment use-case. And so if it works
out, not breaking environment.el can be considered in future
modifications to file-name handlers.

Anyway, isn't the file-name handler interface a generic way to abstract
over access to the underlying operating system, not just a method for
implementing remote-access? Wouldn't a change that caused file-name
handlers to be only usable for remote acces, be a bug?

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

> I do think it's best to use the same mechanism (process-file) that is
> used to support remote access.

Fully agreed.  Which is why I thought the file-name-handler-alist was
probably a good way to do it.  It's true that there are places where
file-name-handlers are misused, so I understand Eli's concerns, but I'm
not as worried about it as he is:
- as long as you don't use "environments" nothing should be affected, so
  if there are bad interactions, they will only show up when you use the
  new feature.
- we already have the tools to fix those interactions (such as
  `file-remote-p').
- we already have other non-remote file-name-handlers (jka-compr, and
  /: come to mind, and you could argue that some uses of /su: and /sudo:
  fall in the same category), so it's not a new problem.
- if we bump into problems, these are probably problems that also
  affect other uses, so we should fix them anyway.

This said, maybe it would work better to add environment support
directly to process-file and start-file-process (and anywhere else that
needs it) rather than use file-name-handler-alist.

I'm not worried about the performance impact on process-file and
friends, since it shouldn't affect that performance any more than
file-name-handler-alist already affects it and it should only affect it
when environments are actually used (and even in that case, the impact
should be dwarfed by the time it takes to fork, set things up, exec,
...).

> Perhaps instead of inserting regular filenames into
> file-name-handler-alist, environment.el should use filenames with a
> custom prefix (like "/?" or something) and add a new handler for
> filenames with that prefix.

That would solve the problem of failing to catch the environment name
because an alias was used, indeed.  OTOH if those environments already
exist outside Emacs and are already associated with existing directories
outside Emacs, it would be a lot more convenient for the user to be able
to use those directories directly instead of having to use a special
/?... syntax (with the added burden of having to tell Emacs which env
corresponds to which dir, ...).


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Fri, 29 Jul 2016 13:59:33 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > It may seem to you that remote access is similar to a different
> > environment, but in fact they are very different.  Environment is not
> > a property of a file name, it is a property of a project.
> 
> Environment is not a property of a project either, it's something else
> entirely. A project doesn't necessarily have an environment associated
> with it, and an environment isn't necessarily asociated with a
> project.

Then perhaps it's time to clarify what an environment really is?  I
don't think that has been done in this discussion; if I missed
something, please point that out.

> If we embed the environment into the file name, then the environment
> also becomes a property of a file name.

But since an environment is not a property of a file name, doing that
would be a mistake, it seems.

> > Moreover, past attempts I know about to use file handlers for
> > processing local files turned out to have subtle misfeatures.
> 
> I would be interested to see previous failed attempts to use file
> handlers on local files, can you point me to an example?

cygwin-mount, for one.

> AFAIK jka-compr.el works fine.

Because it takes great care not to touch anything that changes the
semantics of file names: the file name still behaves the same with and
without jka-compr.  By contrast, AFAIU environments cannot do that by
their very nature.

> Anyway, isn't the file-name handler interface a generic way to abstract
> over access to the underlying operating system, not just a method for
> implementing remote-access?

No.

> Wouldn't a change that caused file-name handlers to be only usable
> for remote acces, be a bug?

Yes, but the best way of fixing a bug is to avoid it in the first
place.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 29 Jul 2016 14:42:26 -0400
> 
> This said, maybe it would work better to add environment support
> directly to process-file and start-file-process (and anywhere else that
> needs it) rather than use file-name-handler-alist.

That's what I suggested up-thread, but that suggestion remained
unnoticed for some reason.

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

> cygwin-mount, for one.

Hmm... I know it has limitations, but I haven't heard of problems with
it linked to issues such as cygwin file names being confused as remote
file names.

>> AFAIK jka-compr.el works fine.
> Because it takes great care not to touch anything that changes the
> semantics of file names: the file name still behaves the same with and
> without jka-compr.  By contrast, AFAIU environments cannot do that by
> their very nature.

I think environments implemented via file-name-handlers-alist would only
affect process-file, start-file-process, and maybe a couple more
functions, and wouldn't otherwise "change the semantics of file names"
any more than jka-compr.


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Eli Zaretskii <eliz@gnu.org> writes:
> Then perhaps it's time to clarify what an environment really is?  I
> don't think that has been done in this discussion; if I missed
> something, please point that out.

An environment is a collection of environment variables paired with
values for those variables. A program is said to run in the environment
when it runs with those environment variables set to those values. Other
environment variables might be set to arbitrary values.

A program is running in a (making up some jargon for this thread):
- "pure environment" if the only environment variables set are those
  specified in the environment.
- "combined environment" if environment variables are set to a
  combination of values from two different environments; generally, a
  combination of a custom environment and the pre-existing
  environment. The custom environment values are generally prepended to
  the pre-existing environment values, which means the custom
  environment search paths will be searched first.

A program running in a Unix distribution uses some environment variables
("search paths") to locate distribution resources and some environment
variables ("configuration variables") to determine its
configuration. For example, PATH is a search path for executables,
LIBRARY_PATH is a search path for libraries, and LANG is a configuration
variable containing the name of the current locale.

The typical Unix distribution has some default environment in which
programs run to access the default resources and configuration of the
distribution.

The "pre-existing environment" is the one which a program started in;
generally this is inherited from the program's parent process without
modification.

An interesting "custom environment" is an environment containing
settings which are not default for the Unix distribution on which the
environment is being used. Instead, a custom environment contains
settings for search paths which point to directories that contain some
custom binaries and libaries, and settings for configuration variables
to further manipulate the behavior of programs running within the custom
environment.

Custom environments are useful for development because they allow
developing and compiling programs in a portable, reproducible,
site-local-configuration-independent way. They are useful also for
distribution by allowing finnicky software to run in an environment that
is exactly what it expects. (These use cases are similar to how chroots
and containers are used, though environments are clearly lighter-weight)

Supporting the use of custom environments in Emacs is the point of this
mailing list thread. Custom environments are (generally) what I have
been referring to when I've said "environment" in this thread.

environment.el would allow using a given "custom environment" to run a
program in Emacs in either a "pure environment", or a "combined
environment" created from the custom environment and the "pre-existing
environment" of Emacs. Ideally, environment.el would provide a nice user
interface for doing that.

>> If we embed the environment into the file name, then the environment
>> also becomes a property of a file name.
>
> But since an environment is not a property of a file name, doing that
> would be a mistake, it seems.

Sorry, I don't think I was clear.

TRAMP has the function of visiting files located on remote hosts, and it
needs to identify the files located on remote hosts. To do this, it
combines an identifier for the remote host, with the filename of the
file on the remote host, and then using that combination as the filename
within Emacs. For environment.el, there is no analogous need to identify
files located inside environments, since the files "located" within an
environment are exactly the same files with the same paths located
outside the environment. (With one key exception - the meaning of ~
depends on the value of HOME in the environment. But that's minor.)

However, TRAMP also needs to be able to support running processes on
remote hosts. And there its mechanism is rather unnatural - why
should we use a filename to identify the remote host on which we want to
run a process? But we chose to do it, and I think it was a pretty good
choice which results in pretty intuitive behavior in the rest of
Emacs. Using a filename to identify the environment to run a process in
is exactly as unnatural as this use of TRAMP, but I think it will result
in equally intuitive behavior.

By the way: In future work I would like to support editing files located
inside chroots, containers, different mount namespaces, and so on.  In
all those cases it makes sense to have a magic prefix to identify where
to look, and then the file name; and in all those cases the files could
(eventually, with work in C) be directly visited on the local machine
rather than transferred over the network. So in those cases it's very
clear to me that file-name-handler-alist should be used, and since TRAMP
basically assumes transferring files over the network/through a
subprocess, a new file-name handler should be made. But all of those
things are quite similar to environments, and it would be natural to
reuse a magic prefix created for environments, to support those
things. So that's another reason why using a magic prefix would be a
sane choice for environments.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 29 Jul 2016 15:20:11 -0400
> 
> >> AFAIK jka-compr.el works fine.
> > Because it takes great care not to touch anything that changes the
> > semantics of file names: the file name still behaves the same with and
> > without jka-compr.  By contrast, AFAIU environments cannot do that by
> > their very nature.
> 
> I think environments implemented via file-name-handlers-alist would only
> affect process-file, start-file-process, and maybe a couple more
> functions, and wouldn't otherwise "change the semantics of file names"
> any more than jka-compr.

They must change the semantics, because there's no other way of making
an environment be a property of a file name.  Look at the syntax
suggested for that, and you will see it.  A file 'foo' will no longer
be called 'foo', but something else.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Fri, 29 Jul 2016 16:57:46 -0400
> 
> An environment is a collection of environment variables paired with
> values for those variables. A program is said to run in the environment
> when it runs with those environment variables set to those values. Other
> environment variables might be set to arbitrary values.

Thanks.  This needs to be augmented by a list of the environment
variables that are handled specially in Emacs and therefore needs more
than just asking the "environment" in its Emacs incarnation to modify
process-environment as it sees fit.

And having read all that, I still don't see why we would like to use
file-name handlers for supporting these features.  My suggestion to
embed such support directly in the affected primitives, like
process-file, call-process, etc., still stands.  Would you please
consider this possibility, and if you think it's inappropriate,
explain why you think so?

> LANG is a configuration variable containing the name of the current
> locale.

If you want an environment to be able to change its locale, then we
must also pay attention to the following subtlety: the value of
process-environment is stored by Emacs in its original undecoded form,
i.e. in the encoding used by the "pre-existing environment"s codeset.
Thus, supporting locale changes seem to require a separate variable
that is used in preference to process-environment, and whose value is
encoded according to the customized locale.

> environment.el would allow using a given "custom environment" to run a
> program in Emacs in either a "pure environment", or a "combined
> environment" created from the custom environment and the "pre-existing
> environment" of Emacs. Ideally, environment.el would provide a nice user
> interface for doing that.

Would having a single user variable for controlling the above be a
sufficiently nice UI?  If not, why not?

> However, TRAMP also needs to be able to support running processes on
> remote hosts. And there its mechanism is rather unnatural - why
> should we use a filename to identify the remote host on which we want to
> run a process?

In the context of running processes on remote hosts, this is actually
very natural.  It allows Lisp applications to be stateless in this
regard.

By contrast, environments support cannot be stateless, because by the
very definition, when a command is executed in a certain environment,
then _all_ of the files and environment variables should be as the
environment defines.

> By the way: In future work I would like to support editing files located
> inside chroots, containers, different mount namespaces, and so on.  In
> all those cases it makes sense to have a magic prefix to identify where
> to look, and then the file name; and in all those cases the files could
> (eventually, with work in C) be directly visited on the local machine
> rather than transferred over the network. So in those cases it's very
> clear to me that file-name-handler-alist should be used, and since TRAMP
> basically assumes transferring files over the network/through a
> subprocess, a new file-name handler should be made.

We will have to discuss those separately.  In any case, these features
are much more about files and filesystems than environments, as you
have described them.

Re: Managing environments (Python venv, guix environment, etc.)

[Michael Albinus]

sbaugh@catern.com writes:

> However, TRAMP also needs to be able to support running processes on
> remote hosts. And there its mechanism is rather unnatural - why
> should we use a filename to identify the remote host on which we want to
> run a process?

It's rather the question which default directory shall be used for
running a process. And indeed, Tramp comes into play when
`default-directory' is a remote one. A file name handler, rather natural.

Best regards, Michael.

Re: Managing environments (Python venv, guix environment, etc.)

[sbaugh]

Eli Zaretskii <eliz@gnu.org> writes:
> And having read all that, I still don't see why we would like to use
> file-name handlers for supporting these features.  My suggestion to
> embed such support directly in the affected primitives, like
> process-file, call-process, etc., still stands.  Would you please
> consider this possibility, and if you think it's inappropriate,
> explain why you think so?

I think I missed the suggestion when you made it earlier (or don't
recognize it as the same suggestion now) and I'm not sure I understand
it in this condensed form. Could you elaborate?

In particular I'm missing how directly adding environment support in
process-file etc. will handle the issue of, e.g., a user invoking M-x
compile, where we need to communicate the environment to use between
buffers.

> If you want an environment to be able to change its locale, then we
> must also pay attention to the following subtlety: the value of
> process-environment is stored by Emacs in its original undecoded form,
> i.e. in the encoding used by the "pre-existing environment"s codeset.
> Thus, supporting locale changes seem to require a separate variable
> that is used in preference to process-environment, and whose value is
> encoded according to the customized locale.

Hmm, it's indeed tricky that as locale changes, the encoding of all
other environment variables also needs to change (if I understand you
correctly). It was probably a bad example though because I doubt that
any environment-creating tools will set a different locale. If there are
some that do, it might be useful to look at what they do to handle
this...

> Would having a single user variable for controlling the above be a
> sufficiently nice UI?  If not, why not?

A single user variable controlling what environment we are in? Meaning
that Emacs is only ever in exactly one environment? That is not
adequate, it makes it difficult to work on different projects in the
same Emacs. And anyway a single environment approach can already be done
easily enough by writing a command to just swap between different values
for process-environment, exec-path, and so on.

>> However, TRAMP also needs to be able to support running processes on
>> remote hosts. And there its mechanism is rather unnatural - why
>> should we use a filename to identify the remote host on which we want to
>> run a process?
>
> In the context of running processes on remote hosts, this is actually
> very natural.  It allows Lisp applications to be stateless in this
> regard.
>
> By contrast, environments support cannot be stateless, because by the
> very definition, when a command is executed in a certain environment,
> then _all_ of the files and environment variables should be as the
> environment defines.

In what way does this allow statelessness, that it doesn't also allow
for environments?

This mechanism allows a Lisp application to ignore whether there is
currently a connection to a remote host, when trying to run a process on
a remote host, since the connection will be created by TRAMP
automatically - is that what you mean by stateless?

Environment usage can be just as stateless. The custom environment
identifier embedded into the file name can and should be enough
information to locate the configuration for the environment, load it
into Emacs, and start using it. Thus a Lisp application trying to run a
process in an environment can ignore whether the environment is
currently in use by Emacs.

Re: Managing environments (Python venv, guix environment, etc.)

[Stefan Monnier]

> They must change the semantics, because there's no other way of making
> an environment be a property of a file name.  Look at the syntax
> suggested for that, and you will see it.  A file 'foo' will no longer
> be called 'foo', but something else.

No, the /?... suggestion is a way to try and solve some problem expected
in the original suggestion which was to keep the names unchanged.

More specifically, the original idea was to associate environments with
specific existing directories, and then setup file-name-handlers which
would catch uses of things like process-file in those
existing directories.  In that case file names would not be changed.
The downside is that a file-name-handler catching /home/monnier/myenv1
may fail to be used if I happen to use refer to ~/myenv1 instead (and
other similar problems with symlinks and such).


        Stefan

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Sat, 30 Jul 2016 10:42:46 -0400
> 
> The downside is that a file-name-handler catching /home/monnier/myenv1
> may fail to be used if I happen to use refer to ~/myenv1 instead (and
> other similar problems with symlinks and such).

Right.

Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Sat, 30 Jul 2016 09:30:59 -0400
> 
> I think I missed the suggestion when you made it earlier (or don't
> recognize it as the same suggestion now) and I'm not sure I understand
> it in this condensed form. Could you elaborate?
> 
> In particular I'm missing how directly adding environment support in
> process-file etc. will handle the issue of, e.g., a user invoking M-x
> compile, where we need to communicate the environment to use between
> buffers.

The communication will be through variables that process-file and
suchlikes will use in preference to exec-path.

If that's still not detailed enough, please ask specific questions,
because I don't understand what could be unclear about that.

> > If you want an environment to be able to change its locale, then we
> > must also pay attention to the following subtlety: the value of
> > process-environment is stored by Emacs in its original undecoded form,
> > i.e. in the encoding used by the "pre-existing environment"s codeset.
> > Thus, supporting locale changes seem to require a separate variable
> > that is used in preference to process-environment, and whose value is
> > encoded according to the customized locale.
> 
> Hmm, it's indeed tricky that as locale changes, the encoding of all
> other environment variables also needs to change (if I understand you
> correctly). It was probably a bad example though because I doubt that
> any environment-creating tools will set a different locale. If there are
> some that do, it might be useful to look at what they do to handle
> this...

So can we agree that customizing LANG b y an environment is currently
out of scope?

> > Would having a single user variable for controlling the above be a
> > sufficiently nice UI?  If not, why not?
> 
> A single user variable controlling what environment we are in? Meaning
> that Emacs is only ever in exactly one environment? That is not
> adequate, it makes it difficult to work on different projects in the
> same Emacs.

A variable can be buffer specific.  When I said "single", I meant a
single symbol, not a single instance (a.k.a. "global variable").

So the question still stands.

> >> However, TRAMP also needs to be able to support running processes on
> >> remote hosts. And there its mechanism is rather unnatural - why
> >> should we use a filename to identify the remote host on which we want to
> >> run a process?
> >
> > In the context of running processes on remote hosts, this is actually
> > very natural.  It allows Lisp applications to be stateless in this
> > regard.
> >
> > By contrast, environments support cannot be stateless, because by the
> > very definition, when a command is executed in a certain environment,
> > then _all_ of the files and environment variables should be as the
> > environment defines.
> 
> In what way does this allow statelessness, that it doesn't also allow
> for environments?

I thought I just explained that (above).  I'm not sure what to say
more.  Lisp applications don't need to know whether the process will
run on the local or remote host.  Perhaps you should look at the
internals of process-file.

> This mechanism allows a Lisp application to ignore whether there is
> currently a connection to a remote host, when trying to run a process on
> a remote host, since the connection will be created by TRAMP
> automatically - is that what you mean by stateless?

Yes, and more: the application simply doesn't know where the process
will run.

> Environment usage can be just as stateless. The custom environment
> identifier embedded into the file name can and should be enough
> information to locate the configuration for the environment, load it
> into Emacs, and start using it.

I don't think we want the environment embedded in file names.

> Thus a Lisp application trying to run a process in an environment
> can ignore whether the environment is currently in use by Emacs.

I don't see how the application could ignore that, since it must
create those special file names to begin with, right?