Issue with remote async processes.

Issue with remote async processes.

[Ergus]

Hi:

I have this simple code:

(defun my/async-sentinel (process _msg)
   "Sentinel function for an asynchronous counsel PROCESS."
   (when (eq (process-status process) 'exit)
     (with-current-buffer " *string-output*"
       (while (accept-process-output process))
       (message "%s" (buffer-substring-no-properties (point-min) (point-max))))))

(let ((my/output (generate-new-buffer " *string-output*" t))
       process)
   (with-connection-local-variables
    (setq process (start-file-process "myls" my/output "ls"))
    (set-process-sentinel process #'my/async-sentinel)))

This works in local processes without issues, but on remote I don't get
any output.  Probably it is an error in the code, but I don't understand
what's missing. And something similar to this used to work before.

Any help please?

Re: Issue with remote async processes.

[Michael Albinus]

Ergus <spacibba@aol.com> writes:

> Hi:

Hi,

> I have this simple code:
>
> (defun my/async-sentinel (process _msg)
>   "Sentinel function for an asynchronous counsel PROCESS."
>   (when (eq (process-status process) 'exit)
>     (with-current-buffer " *string-output*"
>       (while (accept-process-output process))
>       (message "%s" (buffer-substring-no-properties (point-min) (point-max))))))
>
> (let ((my/output (generate-new-buffer " *string-output*" t))
>       process)
>   (with-connection-local-variables
>    (setq process (start-file-process "myls" my/output "ls"))
>    (set-process-sentinel process #'my/async-sentinel)))
>
> This works in local processes without issues, but on remote I don't get
> any output.  Probably it is an error in the code, but I don't understand
> what's missing. And something similar to this used to work before.
>
> Any help please?

You call (generate-new-buffer " *string-output*" t) which means several
instances of this buffer might exist in parallel, with different
names. So you cannot expect, that

--8<---------------cut here---------------start------------->8---
(with-current-buffer " *string-output*"
--8<---------------cut here---------------end--------------->8---

works. Instead, use

--8<---------------cut here---------------start------------->8---
(with-current-buffer (process-buffer process)
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

Re: Issue with remote async processes.

[Ergus]

On Fri, Mar 25, 2022 at 09:55:55AM +0100, Michael Albinus wrote:
>Ergus <spacibba@aol.com> writes:
>
>> Hi:
>
>Hi,
>
>
>You call (generate-new-buffer " *string-output*" t) which means several
>instances of this buffer might exist in parallel, with different
>names. So you cannot expect, that
>
>--8<---------------cut here---------------start------------->8---
>(with-current-buffer " *string-output*"
>--8<---------------cut here---------------end--------------->8---
>
>works. Instead, use
>
>--8<---------------cut here---------------start------------->8---
>(with-current-buffer (process-buffer process)
>--8<---------------cut here---------------end--------------->8---
>
>Best regards, Michael.
>
Very thanks!

Sorry to bother... I have another question. When we use process-file and
start-file-process with tramp, how can we get the error output in a
buffer?

Is that even possible?

Re: Issue with remote async processes.

[Michael Albinus]

Ergus <spacibba@aol.com> writes:

Hi,

> Sorry to bother... I have another question. When we use process-file and
> start-file-process with tramp, how can we get the error output in a
> buffer?

The argument BUFFER can be a list. The car is the output (string or
buffer), the cdr is the stderr (just a string, a filename).

--8<---------------cut here---------------start------------->8---
(let ((default-directory "/ssh::")
      (output (generate-new-buffer "*Output*"))
      (stderr "/ssh::/tmp/xxx"))
  (process-file "echo" nil (list output stderr) nil "Hallo")
  (process-file "echa" nil (list output stderr) nil "Holla")
  (find-file stderr))
--8<---------------cut here---------------end--------------->8---

In start-file-process, it is similar. The difference is, that stderr can
be either a string (filename) or a buffer.

--8<---------------cut here---------------start------------->8---
(let ((default-directory "/ssh::")
      (output (generate-new-buffer "*Output*"))
      (stderr (generate-new-buffer "*Stderr*")))
  (start-file-process "test1" (list output stderr) "echo" "Hallo")
  (start-file-process "test2" (list output stderr) "echa" "Holla"))
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.

Re: Issue with remote async processes.

[Ergus]

On Sun, Mar 27, 2022 at 09:22:53AM +0200, Michael Albinus wrote:
>Ergus <spacibba@aol.com> writes:
>
>Hi,
>
>> Sorry to bother... I have another question. When we use process-file and
>> start-file-process with tramp, how can we get the error output in a
>> buffer?
>
>The argument BUFFER can be a list. The car is the output (string or
>buffer), the cdr is the stderr (just a string, a filename).
>
>--8<---------------cut here---------------start------------->8---
>(let ((default-directory "/ssh::")
>      (output (generate-new-buffer "*Output*"))
>      (stderr "/ssh::/tmp/xxx"))
>  (process-file "echo" nil (list output stderr) nil "Hallo")
>  (process-file "echa" nil (list output stderr) nil "Holla")
>  (find-file stderr))
>--8<---------------cut here---------------end--------------->8---
>
>In start-file-process, it is similar. The difference is, that stderr can
>be either a string (filename) or a buffer.
>
>--8<---------------cut here---------------start------------->8---
>(let ((default-directory "/ssh::")
>      (output (generate-new-buffer "*Output*"))
>      (stderr (generate-new-buffer "*Stderr*")))
>  (start-file-process "test1" (list output stderr) "echo" "Hallo")
>  (start-file-process "test2" (list output stderr) "echa" "Holla"))
>--8<---------------cut here---------------end--------------->8---
>
>Best regards, Michael.
>
Very thanks Michael... This is useful.

What's reason process-file doesn't support to use a buffer for stderr?

Performance? Technical?

Re: Issue with remote async processes.

[Michael Albinus]

Ergus <spacibba@aol.com> writes:

Hi,

> What's reason process-file doesn't support to use a buffer for stderr?
>
> Performance? Technical?

History. process-file and start-process are derived from call-process
and start-file-process.

call-process allows only (REAL-BUFFER STDERR-FILE), and so does
process-file.

start-process allows only BUFFER in its docstring (no separation between
stdout and stderr). So start-file-process did originally. Later, when
I've reimplemented it in Tramp using make-process, I wanted to support
the separate stderr buffer of make-process, and I have extended it,
therefore. However, this is not reflected in the docstring, and you
might be better served to follow the official API, and use make-process
instead.

Best regards, Michael.

Re: Issue with remote async processes.

[Ergus]

On Mon, Mar 28, 2022 at 11:14:20AM +0200, Michael Albinus wrote:
>Ergus <spacibba@aol.com> writes:
>
>Hi,
>
>> What's reason process-file doesn't support to use a buffer for stderr?
>>
>> Performance? Technical?
>
>History. process-file and start-process are derived from call-process
>and start-file-process.
>
>call-process allows only (REAL-BUFFER STDERR-FILE), and so does
>process-file.
>
>start-process allows only BUFFER in its docstring (no separation between
>stdout and stderr). So start-file-process did originally. Later, when
>I've reimplemented it in Tramp using make-process, I wanted to support
>the separate stderr buffer of make-process, and I have extended it,
>therefore. However, this is not reflected in the docstring, and you
>might be better served to follow the official API, and use make-process
>instead.
>
>Best regards, Michael.
>
Again very thanks...

So far I have complains with the emacs process API, there are many
function sometimes redundant and with confusing similar names, some of
them are only available for one specific case (like process-lines*,
which is available as a wrapper of call-process but not for the
process-file case considering if there is a find-file-name-handler)...

I would actually expect a simpler function interface with two extra
parameters (remote async).

The async may unify make-process and call-process families
The remote may unify process-file with call-process families

That may reduce the 4 functions to a single one...

I use Tramp extensively and I have observed that several packages are
broken with tramp just because of that... the developers use one of them
without been aware of the others...

Any way...

An extra comment:

(process-command process) always returns `/bin/sh -i` when the process
was executed remotely. I understand the reasons, but I am not sure if
this is intended... I mean, I am interested in the real executed command
when it failed for example. (either the complete one or the one passed
to start-file-process)

Is there a way to retrieve the full command like when the process
executed locally??

Sorry to bother so sooooo much, but documentation is sometimes not very
clear about these sort of details...

Again, thanks in advance,
Ergus

Re: Issue with remote async processes.

[Michael Albinus]

Ergus <spacibba@aol.com> writes:

Hi,

> So far I have complains with the emacs process API, there are many
> function sometimes redundant and with confusing similar names, some of
> them are only available for one specific case (like process-lines*,
> which is available as a wrapper of call-process but not for the
> process-file case considering if there is a find-file-name-handler)...

It is just history ...

> I would actually expect a simpler function interface with two extra
> parameters (remote async).
>
> The async may unify make-process and call-process families

call-process is synchronous. Likely, you mean unification of
make-process and start{-file}-process? Technically, it would be possible
to obsolete start{-file}-process, make-process would be sufficient. But
there's ton of packages in the wild using that, so it might create more
trouble than it is worth for.

These days, writing new code, I would simply use make-process, that's it.

> The remote may unify process-file with call-process families

Not so simple. There is a reason for call-process: a guarantee that it is
executed always locally, whatever default-directory.

> That may reduce the 4 functions to a single one...

I don't believe it will work. But perhaps we could at least bring the
arguments of the local process functions and the remote process
functions in line, at least for the stdout and stderr handling.

> An extra comment:
>
> (process-command process) always returns `/bin/sh -i` when the process
> was executed remotely. I understand the reasons, but I am not sure if
> this is intended... I mean, I am interested in the real executed command
> when it failed for example. (either the complete one or the one passed
> to start-file-process)
>
> Is there a way to retrieve the full command like when the process
> executed locally??

Not yet (you must look into the Tramp traces). Maybe you write a
wishlist bug report? Tramp adds already the remote-pid and remote-tty
properties to process objects. It would be easy to add another property
remote-command which would keep a list of the program and its arguments.

> Again, thanks in advance, Ergus

Best regards, Michael.

Re: Issue with remote async processes.

[Ergus]

Hi:

I wonder about this because I find myself modifiying packages constantly
to use them with tramp.

On Tue, Mar 29, 2022 at 10:55:38AM +0200, Michael Albinus wrote:
>Ergus <spacibba@aol.com> writes:
>
>Hi,
>
>> So far I have complains with the emacs process API, there are many
>> function sometimes redundant and with confusing similar names, some of
>> them are only available for one specific case (like process-lines*,
>> which is available as a wrapper of call-process but not for the
>> process-file case considering if there is a find-file-name-handler)...
>
>It is just history ...
>
But from the practical point of view it is very confusing to keep this
interface just because of this...

>> I would actually expect a simpler function interface with two extra
>> parameters (remote async).
>>
>> The async may unify make-process and call-process families
>
>call-process is synchronous. Likely, you mean unification of
>make-process and start{-file}-process? Technically, it would be possible
>to obsolete start{-file}-process, make-process would be sufficient. But
>there's ton of packages in the wild using that, so it might create more
>trouble than it is worth for.
>
With the obsoletion policies in emacs you will have the function with
the obsolete warning for the next 10 years probably... The packages that
don't update during that time, probably may need to be removed too
because that means they are abandoned.

>These days, writing new code, I would simply use make-process, that's it.
>
How do you use find-file-name-handler then? Just with :file-handler t?

>> The remote may unify process-file with call-process families
>
>Not so simple. There is a reason for call-process: a guarantee that it is
>executed always locally, whatever default-directory.
>
Couldn't that be solved with the variable? I called it remote, but may
call it `local` instead... so when local not-nil guarantees... But the
idea is the same...

You could even reuse the file-handler parameter for example??

>> That may reduce the 4 functions to a single one...
>
>I don't believe it will work. But perhaps we could at least bring the
>arguments of the local process functions and the remote process
>functions in line, at least for the stdout and stderr handling.
>
Please...

>> An extra comment:
>>
>> (process-command process) always returns `/bin/sh -i` when the process
>> was executed remotely. I understand the reasons, but I am not sure if
>> this is intended... I mean, I am interested in the real executed command
>> when it failed for example. (either the complete one or the one passed
>> to start-file-process)
>>
>> Is there a way to retrieve the full command like when the process
>> executed locally??
>
>Not yet (you must look into the Tramp traces). Maybe you write a
>wishlist bug report? Tramp adds already the remote-pid and remote-tty
>properties to process objects. It would be easy to add another property
>remote-command which would keep a list of the program and its arguments.
>
Please... ;)

>> Again, thanks in advance, Ergus
>
>Best regards, Michael.
>

Re: Issue with remote async processes.

[Michael Albinus]

Ergus <spacibba@aol.com> writes:

> Hi:

Hi,

>>call-process is synchronous. Likely, you mean unification of
>>make-process and start{-file}-process? Technically, it would be possible
>>to obsolete start{-file}-process, make-process would be sufficient. But
>>there's ton of packages in the wild using that, so it might create more
>>trouble than it is worth for.
>>
> With the obsoletion policies in emacs you will have the function with
> the obsolete warning for the next 10 years probably... The packages that
> don't update during that time, probably may need to be removed too
> because that means they are abandoned.

10 years would be too long for me. No idea, whether I'll still use Emacs
then :-)

>>These days, writing new code, I would simply use make-process, that's it.
>>
> How do you use find-file-name-handler then? Just with :file-handler t?

Yes.

>>> The remote may unify process-file with call-process families
>>
>>Not so simple. There is a reason for call-process: a guarantee that it is
>>executed always locally, whatever default-directory.
>>
> Couldn't that be solved with the variable? I called it remote, but may
> call it `local` instead... so when local not-nil guarantees... But the
> idea is the same...
>
> You could even reuse the file-handler parameter for example??

We've discussed this decades ago, when process-file was invented. The
agreement was not to touch packages which use call-process for a reason,
even an additional customization of a variable would be too much. Here
we are ...

>>> That may reduce the 4 functions to a single one...
>>
>>I don't believe it will work. But perhaps we could at least bring the
>>arguments of the local process functions and the remote process
>>functions in line, at least for the stdout and stderr handling.
>>
> Please...

Hmm. Let's see. No promise!

>>It would be easy to add another property
>>remote-command which would keep a list of the program and its arguments.
>>
> Please... ;)

I've implemented this, pushed to master. The Tramp manual got a new
subsection:

--8<---------------cut here---------------start------------->8---
5.6.8 Process properties of asynchronous remote processes
---------------------------------------------------------

When available, TRAMP adds process properties to process objects of
asynchronous properties.  However, it is not guaranteed that all these
properties are set.

   • ‘remote-tty’

     This is the name of the terminal a PROCESS uses on the remote
     host, i.e., it reads and writes on.

   • ‘remote-pid’

     The process id of the command executed on the remote host.  This
     is used when sending signals remotely.

   • ‘remote-command’

     The remote command which has been invoked via ‘make-process’ or
     ‘start-file-process’, a list of strings (program and its
     arguments).  This does not show the additional shell sugar TRAMP
     makes around the commands, in order to see this you must inspect
     TRAMP *note traces: Traces and Profiles.
--8<---------------cut here---------------end--------------->8---

Best regards, Michael.