Re: Python and Tramp

[Michael Albinus <michael.albinus@gmx.de>]

Augusto Stoffel <arstoffel@gmail.com> writes:

[Cc Fabián, Stefan and the emacs-devel crowd]

> Hi Michael,

Hi Augusto,

Finally, I found the time to start working on this. And on Wedenesday
there is the Berlin Emacs meetup, likely we'll see each other. Sorry
again for the delay.

> Unless the Python use-cases are very uncommon, which I doubt, python.el
> is reinventing the wheel big time.  Or maybe inventing the wheel in the
> wrong place.

Yes.

> I don't have much clarity about the requisites of a “generalized
> virtualenv support” in Emacs.  But I think at least the following is
> clear:
>
> 1. A mechanism is needed to configure the execution of external
>    processes in a per-project fashion.  This includes REPLs, linters and
>    checkers, language servers, compilers, etc.
>
> 2. This should work whether the project is local or remote, as
>    transparently as possible.

In general, I agree. However, in the remote case I would like to
concentrate (first?) on one project per remote host. This doesn't seem
to be a real constraint, as this is the situation today for the majority
of users. You have described such a setting below.

> 3. We should also support the case where the project files are local,
>    but some external process are to be executed remotely.
>
> As to requirement (3), recall my specific situation: I often work with a
> heavy-duty VM available on the Google cloud.  It's not practical for me
> to process the data in my local laptop, since I don't have enough RAM or
> GPUs (but I'm also not allowed to download the data anyway, for data
> protection reasons).  On the other hand, I don't want to edit the code
> in the VM; for one thing, there's an annoying latency if I want grep the
> code or ask for completions.
>
> Someone at the meetup said that (3) seemed rather particular to my
> specific needs, but I think it's only going to become more common.  I
> can also very well imagine the case where someone is using a
> Docker-based workflow, where they want to edit the code in a normal
> directory but lint/compile/execute it inside an image.

This is a different topic, which shall be investigated independently.

> My way of doing (1) at this moment is by making the
> `process-environment' and `exec-path` buffer local.  This fulfills all
> my personal requirements but probably doesn't help for requirement (2);
> I don't care about it because I have essentially one VM per project.
>
> Currently, I achieve (3) basically by calling things like
>
>    (let ((default-directory "/ssh:remote-machine:/some/path"))
>      (run-python))
>
> This works, but is a bit clunky.

As I said a while ago, we should use connection-local variables. This is
a mechanism, which sets buffer-local variables (as you propose)
depending on which remote connection is related to the buffer.

Let's say you need to set process-environment and exec-path
differently for different remote hosts. Then you can declare

--8<---------------cut here---------------start------------->8---
(let ((default-directory "/ssh:remote-machine:/some/path"))
  (connection-local-set-profile-variables
    'profile
    `((process-environment . ,(python-shell-calculate-process-environment))
      (exec-path . ,(python-shell-calculate-exec-path))))

  (connection-local-set-profiles
    (connection-local-criteria-for-default-directory) 'profile)

  (run-python))
--8<---------------cut here---------------end--------------->8---

This is a very shortened code snippet, of course it needs some more
grain distinguishing different remote hosts. I have added the
let-binding just for illustration, in python.el this wouldn't be needed.

python.el uses remote processes, either process-file or make-process
(all other process starting functions end up in one of them). Tramp
acknowledges the connection-local variables, and uses them when starting
the respective process.

What I don't understand is the need to send explicit commands to the
remote process or to manipulate Tramp's cache, as done in
python-shell-tramp-refresh-remote-path and
python-shell-tramp-refresh-process-environment. Why is this neded? It
shouldn't, otherwise it might be a bug in Tramp.

> Anyway, I don't know if all of this mumbling helps in any way.  I hope
> it does, somehow.

It does, indeed. If we start to change python.el, it will depend on
Emacs 27.1, due to connection-local variables. Would that be OK?

> Best,
> Augusto

Best regards, Michael.