Re: Python and Tramp

Re: Python and Tramp

[Michael Albinus]

Augusto Stoffel <arstoffel@gmail.com> writes:

[Cc Fabián, Stefan and the emacs-devel crowd]

> Hi Michael,

Hi Augusto,

Finally, I found the time to start working on this. And on Wedenesday
there is the Berlin Emacs meetup, likely we'll see each other. Sorry
again for the delay.

> Unless the Python use-cases are very uncommon, which I doubt, python.el
> is reinventing the wheel big time.  Or maybe inventing the wheel in the
> wrong place.

Yes.

> I don't have much clarity about the requisites of a “generalized
> virtualenv support” in Emacs.  But I think at least the following is
> clear:
>
> 1. A mechanism is needed to configure the execution of external
>    processes in a per-project fashion.  This includes REPLs, linters and
>    checkers, language servers, compilers, etc.
>
> 2. This should work whether the project is local or remote, as
>    transparently as possible.

In general, I agree. However, in the remote case I would like to
concentrate (first?) on one project per remote host. This doesn't seem
to be a real constraint, as this is the situation today for the majority
of users. You have described such a setting below.

> 3. We should also support the case where the project files are local,
>    but some external process are to be executed remotely.
>
> As to requirement (3), recall my specific situation: I often work with a
> heavy-duty VM available on the Google cloud.  It's not practical for me
> to process the data in my local laptop, since I don't have enough RAM or
> GPUs (but I'm also not allowed to download the data anyway, for data
> protection reasons).  On the other hand, I don't want to edit the code
> in the VM; for one thing, there's an annoying latency if I want grep the
> code or ask for completions.
>
> Someone at the meetup said that (3) seemed rather particular to my
> specific needs, but I think it's only going to become more common.  I
> can also very well imagine the case where someone is using a
> Docker-based workflow, where they want to edit the code in a normal
> directory but lint/compile/execute it inside an image.

This is a different topic, which shall be investigated independently.

> My way of doing (1) at this moment is by making the
> `process-environment' and `exec-path` buffer local.  This fulfills all
> my personal requirements but probably doesn't help for requirement (2);
> I don't care about it because I have essentially one VM per project.
>
> Currently, I achieve (3) basically by calling things like
>
>    (let ((default-directory "/ssh:remote-machine:/some/path"))
>      (run-python))
>
> This works, but is a bit clunky.

As I said a while ago, we should use connection-local variables. This is
a mechanism, which sets buffer-local variables (as you propose)
depending on which remote connection is related to the buffer.

Let's say you need to set process-environment and exec-path
differently for different remote hosts. Then you can declare

--8<---------------cut here---------------start------------->8---
(let ((default-directory "/ssh:remote-machine:/some/path"))
  (connection-local-set-profile-variables
    'profile
    `((process-environment . ,(python-shell-calculate-process-environment))
      (exec-path . ,(python-shell-calculate-exec-path))))

  (connection-local-set-profiles
    (connection-local-criteria-for-default-directory) 'profile)

  (run-python))
--8<---------------cut here---------------end--------------->8---

This is a very shortened code snippet, of course it needs some more
grain distinguishing different remote hosts. I have added the
let-binding just for illustration, in python.el this wouldn't be needed.

python.el uses remote processes, either process-file or make-process
(all other process starting functions end up in one of them). Tramp
acknowledges the connection-local variables, and uses them when starting
the respective process.

What I don't understand is the need to send explicit commands to the
remote process or to manipulate Tramp's cache, as done in
python-shell-tramp-refresh-remote-path and
python-shell-tramp-refresh-process-environment. Why is this neded? It
shouldn't, otherwise it might be a bug in Tramp.

> Anyway, I don't know if all of this mumbling helps in any way.  I hope
> it does, somehow.

It does, indeed. If we start to change python.el, it will depend on
Emacs 27.1, due to connection-local variables. Would that be OK?

> Best,
> Augusto

Best regards, Michael.

Re: Python and Tramp

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

> I think, when reading the python.el code, I'm bothered by 2 things:
> - the presence of Tramp code, of course.
> - the fact that I can't clearly see what is the intention of that code
>   because it is intermingled with Tramp code.

See my other message. Both shouldn't be needed.

> I suggest we start by changing
> `python-shell-calculate-process-environment` so it doesn't return the
> full environment, but only the entries that should be added to the
> environment.  This should make it possible to write it without any
> Tramp code.

Yes.

> Also we should split `python-shell-with-environment` into the Tramp and
> the non-Tramp case (there's very little overlap).  E.g. the code below
> has most of the Tramp-specific code confined to functions that have
> `tramp` in their name and which mostly (but not quite ;-( ) don't know
> about Python.

Could we start with your patch? That means, could you push it to master?
I would continue then with my proposals from the other message.

@Augusto: I don't write python code, so I haven't an idea what is
important. Could you pls send my a small remote test project, including
the settings you apply in your emacs, which I can use for my own tests?

>         Stefan

Best regards, Michael.

Re: Python and Tramp

[Stefan Monnier]

Michael Albinus [2021-10-25 14:36:49] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> I think, when reading the python.el code, I'm bothered by 2 things:
>> - the presence of Tramp code, of course.
>> - the fact that I can't clearly see what is the intention of that code
>>   because it is intermingled with Tramp code.
> See my other message. Both shouldn't be needed.
>> I suggest we start by changing
>> `python-shell-calculate-process-environment` so it doesn't return the
>> full environment, but only the entries that should be added to the
>> environment.  This should make it possible to write it without any
>> Tramp code.
> Yes.
>> Also we should split `python-shell-with-environment` into the Tramp and
>> the non-Tramp case (there's very little overlap).  E.g. the code below
>> has most of the Tramp-specific code confined to functions that have
>> `tramp` in their name and which mostly (but not quite ;-( ) don't know
>> about Python.
> Could we start with your patch? That means, could you push it to master?

Of course.  Done.

> I would continue then with my proposals from the other message.

I count on you to do it as promptly as I just did ;-)


        Stefan