unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Forwarding stdout/stderr of a child process in batch mode
@ 2020-12-06 13:30 Paul Pogonyshev
  2020-12-06 13:36 ` Zhu Zihao
  0 siblings, 1 reply; 6+ messages in thread
From: Paul Pogonyshev @ 2020-12-06 13:30 UTC (permalink / raw)
  To: Emacs developers

Hi,

I'm using Emacs in batch mode. I need to invoke a child process that
is a longish operation (a few minutes). During this time, it writes to
its stdout, so user will see that it is working and what exactly is
being done. However, if I invoke it from Emacs (e.g. using
`call-process') I see no way of forwarding this output to the "real"
stdout. So, for a user this looks like the process (or batch Emacs on
top of it) is hung.

Am I missing a way to forward output?

Paul



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Forwarding stdout/stderr of a child process in batch mode
  2020-12-06 13:30 Forwarding stdout/stderr of a child process in batch mode Paul Pogonyshev
@ 2020-12-06 13:36 ` Zhu Zihao
  2020-12-06 21:40   ` Paul Pogonyshev
  0 siblings, 1 reply; 6+ messages in thread
From: Zhu Zihao @ 2020-12-06 13:36 UTC (permalink / raw)
  To: Paul Pogonyshev; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]


IMO, in batch mode, `message` writes to stderr, `princ` writes to
stdout. You can install a filter for childprocess, and run functions I
mentioned above to forward these outputs.


Paul Pogonyshev writes:

> Hi,
>
> I'm using Emacs in batch mode. I need to invoke a child process that
> is a longish operation (a few minutes). During this time, it writes to
> its stdout, so user will see that it is working and what exactly is
> being done. However, if I invoke it from Emacs (e.g. using
> `call-process') I see no way of forwarding this output to the "real"
> stdout. So, for a user this looks like the process (or batch Emacs on
> top of it) is hung.
>
> Am I missing a way to forward output?
>
> Paul


-- 
Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 515 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Forwarding stdout/stderr of a child process in batch mode
  2020-12-06 13:36 ` Zhu Zihao
@ 2020-12-06 21:40   ` Paul Pogonyshev
  2020-12-08 15:55     ` Paul Pogonyshev
  0 siblings, 1 reply; 6+ messages in thread
From: Paul Pogonyshev @ 2020-12-06 21:40 UTC (permalink / raw)
  To: Zhu Zihao; +Cc: Emacs developers

Thank you, this seems to work (haven't tried separating stderr from
stdout yet). Is there a normal way to wait for asynchronous process to
terminate? Currently I have managed to make it work by throwing from
sentinel, but this feels a bit hackish, maybe there is something
better?

Currently I have something like this:

    (let ((process (make-process ... :sentinel (lambda (_process
_event) (throw 'done nil)))))
      (catch 'done
        (while (process-live-p process)
          (accept-process-output nil 60)))
      ...

Paul

On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote:
>
>
> IMO, in batch mode, `message` writes to stderr, `princ` writes to
> stdout. You can install a filter for childprocess, and run functions I
> mentioned above to forward these outputs.
>
>
> Paul Pogonyshev writes:
>
> > Hi,
> >
> > I'm using Emacs in batch mode. I need to invoke a child process that
> > is a longish operation (a few minutes). During this time, it writes to
> > its stdout, so user will see that it is working and what exactly is
> > being done. However, if I invoke it from Emacs (e.g. using
> > `call-process') I see no way of forwarding this output to the "real"
> > stdout. So, for a user this looks like the process (or batch Emacs on
> > top of it) is hung.
> >
> > Am I missing a way to forward output?
> >
> > Paul
>
>
> --
> Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp
>
> Zihao



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Forwarding stdout/stderr of a child process in batch mode
  2020-12-06 21:40   ` Paul Pogonyshev
@ 2020-12-08 15:55     ` Paul Pogonyshev
  2020-12-08 21:43       ` Bruno Barbier
  0 siblings, 1 reply; 6+ messages in thread
From: Paul Pogonyshev @ 2020-12-08 15:55 UTC (permalink / raw)
  To: Zhu Zihao; +Cc: Emacs developers

[-- Attachment #1: Type: text/plain, Size: 3320 bytes --]

So, I got to separating stdout and stderr, but I get them in a wrong
order now...  Here is an example Python program that I use to generate
output (store as `10.py'):

    #! /usr/bin/env python

    import sys
    import time

    for k in range (10):
        sys.stdout.write ('line %d to stdout\n' % k)
        sys.stderr.write ('line %d to stderr\n' % k)
        time.sleep (0.2)

And the Emacs script (store as `fwd.el') to be executed as

    $ emacs --batch --script fwd.el

that gives output in a wrong (compared to simply executing `./10.py'
from console, including M-x shell or M-x eshell) order:

    (let ((process (make-process :name     "10"
                                 :command  `(,(expand-file-name "10.py"))
                                 :filter   (lambda (_process string) (princ
string))
                                 :stderr   (make-pipe-process :name
"10/stderr"
                                                              :filter
(lambda (_process string) (message "%s" string)))
                                 :sentinel (lambda (&rest _ignored) (throw
'interrupted nil)))))
      (while (process-live-p process)
        (catch 'interrupted
          (accept-process-output nil 60))))

When I run the Emacs script, I receive all stderr output line-by-line
(which is fine), but all stdout output comes in one chunk at the very
end, which is not...

I cannot really use `pty' connection type, because I want (in real
usecase, not in the script) to additionally keep stdout and stderr in
separate buffers, for possible later processing.  Also, with pty I
wouldn't be able to use `message' or `princ' depending on the stream
used by the child process.

Is there any way to fix this?

Paul


On Sun, 6 Dec 2020 at 22:40, Paul Pogonyshev <pogonyshev@gmail.com> wrote:

> Thank you, this seems to work (haven't tried separating stderr from
> stdout yet). Is there a normal way to wait for asynchronous process to
> terminate? Currently I have managed to make it work by throwing from
> sentinel, but this feels a bit hackish, maybe there is something
> better?
>
> Currently I have something like this:
>
>     (let ((process (make-process ... :sentinel (lambda (_process
> _event) (throw 'done nil)))))
>       (catch 'done
>         (while (process-live-p process)
>           (accept-process-output nil 60)))
>       ...
>
> Paul
>
> On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote:
> >
> >
> > IMO, in batch mode, `message` writes to stderr, `princ` writes to
> > stdout. You can install a filter for childprocess, and run functions I
> > mentioned above to forward these outputs.
> >
> >
> > Paul Pogonyshev writes:
> >
> > > Hi,
> > >
> > > I'm using Emacs in batch mode. I need to invoke a child process that
> > > is a longish operation (a few minutes). During this time, it writes to
> > > its stdout, so user will see that it is working and what exactly is
> > > being done. However, if I invoke it from Emacs (e.g. using
> > > `call-process') I see no way of forwarding this output to the "real"
> > > stdout. So, for a user this looks like the process (or batch Emacs on
> > > top of it) is hung.
> > >
> > > Am I missing a way to forward output?
> > >
> > > Paul
> >
> >
> > --
> > Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp
> >
> > Zihao
>

[-- Attachment #2: Type: text/html, Size: 4429 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Forwarding stdout/stderr of a child process in batch mode
  2020-12-08 15:55     ` Paul Pogonyshev
@ 2020-12-08 21:43       ` Bruno Barbier
  2020-12-09  1:14         ` Paul Pogonyshev
  0 siblings, 1 reply; 6+ messages in thread
From: Bruno Barbier @ 2020-12-08 21:43 UTC (permalink / raw)
  To: Paul Pogonyshev, Zhu Zihao; +Cc: Emacs developers


Hi Paul,

Paul Pogonyshev <pogonyshev@gmail.com> writes:

> So, I got to separating stdout and stderr, but I get them in a wrong
> order now...  Here is an example Python program that I use to generate
> output (store as `10.py'):
...

> When I run the Emacs script, I receive all stderr output line-by-line
> (which is fine), but all stdout output comes in one chunk at the very
> end, which is not...
>

You may want to pass the "-u" option to Python, so that it uses
unbuffered outputs.

You're example is working for me, when using the "-u" option; i.e. I'm
getting interleaved chunks of stderr/stdout.

Bruno

> I cannot really use `pty' connection type, because I want (in real
> usecase, not in the script) to additionally keep stdout and stderr in
> separate buffers, for possible later processing.  Also, with pty I
> wouldn't be able to use `message' or `princ' depending on the stream
> used by the child process.
>
> Is there any way to fix this?
>
> Paul
>
>
> On Sun, 6 Dec 2020 at 22:40, Paul Pogonyshev <pogonyshev@gmail.com> wrote:
>
>> Thank you, this seems to work (haven't tried separating stderr from
>> stdout yet). Is there a normal way to wait for asynchronous process to
>> terminate? Currently I have managed to make it work by throwing from
>> sentinel, but this feels a bit hackish, maybe there is something
>> better?
>>
>> Currently I have something like this:
>>
>>     (let ((process (make-process ... :sentinel (lambda (_process
>> _event) (throw 'done nil)))))
>>       (catch 'done
>>         (while (process-live-p process)
>>           (accept-process-output nil 60)))
>>       ...
>>
>> Paul
>>
>> On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote:
>> >
>> >
>> > IMO, in batch mode, `message` writes to stderr, `princ` writes to
>> > stdout. You can install a filter for childprocess, and run functions I
>> > mentioned above to forward these outputs.
>> >
>> >
>> > Paul Pogonyshev writes:
>> >
>> > > Hi,
>> > >
>> > > I'm using Emacs in batch mode. I need to invoke a child process that
>> > > is a longish operation (a few minutes). During this time, it writes to
>> > > its stdout, so user will see that it is working and what exactly is
>> > > being done. However, if I invoke it from Emacs (e.g. using
>> > > `call-process') I see no way of forwarding this output to the "real"
>> > > stdout. So, for a user this looks like the process (or batch Emacs on
>> > > top of it) is hung.
>> > >
>> > > Am I missing a way to forward output?
>> > >
>> > > Paul
>> >
>> >
>> > --
>> > Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp
>> >
>> > Zihao
>>



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Forwarding stdout/stderr of a child process in batch mode
  2020-12-08 21:43       ` Bruno Barbier
@ 2020-12-09  1:14         ` Paul Pogonyshev
  0 siblings, 0 replies; 6+ messages in thread
From: Paul Pogonyshev @ 2020-12-09  1:14 UTC (permalink / raw)
  To: Bruno Barbier; +Cc: Zhu Zihao, Emacs developers

[-- Attachment #1: Type: text/plain, Size: 3848 bytes --]

> You may want to pass the "-u" option to Python, so that it uses
> unbuffered outputs.
>
> You're example is working for me, when using the "-u" option; i.e. I'm
> getting interleaved chunks of stderr/stdout.

It is only an example. In real usecase I will not call a Python script, in
fact, it can be any executable whatsoever, I don't know in advance. Also,
shell itself doesn't need `-u'. For example:

    sh -c 'sh -c "python 10.py"'
    sh -c 'sh -c "python 10.py"' >/dev/null

The second command shows that it doesn't write everything to stdout,
it still maintains stdout/stderr separation. The nested `-c' shows that
it works through any (I guess) level of indirection, and also if I use
different shells (`bash', `dash'), so it doesn't look like shell is special-
casing, it really can handle any commands.

Finally, even with `python -u' I get non-ideal behavior from Emacs:
stderr chunks come before the stdout one, even though the script writes
to stdout before stderr.

Paul


On Tue, 8 Dec 2020 at 22:43, Bruno Barbier <brubar.cs@gmail.com> wrote:

>
> Hi Paul,
>
> Paul Pogonyshev <pogonyshev@gmail.com> writes:
>
> > So, I got to separating stdout and stderr, but I get them in a wrong
> > order now...  Here is an example Python program that I use to generate
> > output (store as `10.py'):
> ...
>
> > When I run the Emacs script, I receive all stderr output line-by-line
> > (which is fine), but all stdout output comes in one chunk at the very
> > end, which is not...
> >
>
> You may want to pass the "-u" option to Python, so that it uses
> unbuffered outputs.
>
> You're example is working for me, when using the "-u" option; i.e. I'm
> getting interleaved chunks of stderr/stdout.
>
> Bruno
>
> > I cannot really use `pty' connection type, because I want (in real
> > usecase, not in the script) to additionally keep stdout and stderr in
> > separate buffers, for possible later processing.  Also, with pty I
> > wouldn't be able to use `message' or `princ' depending on the stream
> > used by the child process.
> >
> > Is there any way to fix this?
> >
> > Paul
> >
> >
> > On Sun, 6 Dec 2020 at 22:40, Paul Pogonyshev <pogonyshev@gmail.com>
> wrote:
> >
> >> Thank you, this seems to work (haven't tried separating stderr from
> >> stdout yet). Is there a normal way to wait for asynchronous process to
> >> terminate? Currently I have managed to make it work by throwing from
> >> sentinel, but this feels a bit hackish, maybe there is something
> >> better?
> >>
> >> Currently I have something like this:
> >>
> >>     (let ((process (make-process ... :sentinel (lambda (_process
> >> _event) (throw 'done nil)))))
> >>       (catch 'done
> >>         (while (process-live-p process)
> >>           (accept-process-output nil 60)))
> >>       ...
> >>
> >> Paul
> >>
> >> On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote:
> >> >
> >> >
> >> > IMO, in batch mode, `message` writes to stderr, `princ` writes to
> >> > stdout. You can install a filter for childprocess, and run functions I
> >> > mentioned above to forward these outputs.
> >> >
> >> >
> >> > Paul Pogonyshev writes:
> >> >
> >> > > Hi,
> >> > >
> >> > > I'm using Emacs in batch mode. I need to invoke a child process that
> >> > > is a longish operation (a few minutes). During this time, it writes
> to
> >> > > its stdout, so user will see that it is working and what exactly is
> >> > > being done. However, if I invoke it from Emacs (e.g. using
> >> > > `call-process') I see no way of forwarding this output to the "real"
> >> > > stdout. So, for a user this looks like the process (or batch Emacs
> on
> >> > > top of it) is hung.
> >> > >
> >> > > Am I missing a way to forward output?
> >> > >
> >> > > Paul
> >> >
> >> >
> >> > --
> >> > Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp
> >> >
> >> > Zihao
> >>
>

[-- Attachment #2: Type: text/html, Size: 5823 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2020-12-09  1:14 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-12-06 13:30 Forwarding stdout/stderr of a child process in batch mode Paul Pogonyshev
2020-12-06 13:36 ` Zhu Zihao
2020-12-06 21:40   ` Paul Pogonyshev
2020-12-08 15:55     ` Paul Pogonyshev
2020-12-08 21:43       ` Bruno Barbier
2020-12-09  1:14         ` Paul Pogonyshev

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).